Half of both, shake, enjoy …
In the OSGi world, there is different ways to test “bundles” / “systems” / “applications”.Spring Integration Testing fit very well with Spring applications, but in all the others cases it doesn’t work very well.
BND Testing Harness is a testing framework integrated with BND. Despite very powerful to generate bundle on the fly, it sounds not very adequate when test must be integrated inside a build process expect Ant and Eclipse (such as Maven).
Junit4osgi is a very simple test framework, running junit 3 tests on OSGi. The goal was to provide a simple way to write tests checking services … junit4osgi also provides high-level method simplifying the OSGi test writing.
Finally, pax:exam is a test framework for OSGi allowing to finely configure the test framework.
Recently, I discuss with the pax:exam developers about providing THE test framework for OSGi. In fact, I was considering moving the iPOJO test suites and the application test suites that I developed to pax:exam. After trying pax:exam, I was finally able to understand what pax:exam provides and what would be great…
Requirements
First what are the requirements of a cool test framework for OSGi.
To begin, it has to reuse existing “test engine” such as junit or testNG. Java developers have already developed tests with those frameworks (I hope so…), and so this should ease the OSGi test writing.
Then, as you know, testing is never a very exiting task. So, the test framework should provide high-level methods to ease the OSGi integration. Methods such as “getService”, “waitForService” … should be provided and so the developer can focus on testing its service and not how to get the service.
Third important point: the configuration. The application/service have to be deployed and run is a “real” OSGi platform. This platform has to be configured. This configuration contains obviously the bundles to deploy but can also contain other properties (extra packages, system configuration…). Supporting several OSGi implementation and several dependent bundle package is also a good property.
Another important aspect related to the configuration is the dispatching test / platform. This should allow configuring which tests have to be deployed and execute on the SAME frameworks. Despite this can create side effects; sometimes we are looking for those side effects.
The test framework must provide a “seamless” integration with build process and development tools such as Ant, Maven, Eclipse… Tests should be executed when the project is built, and developer should be able to execute quickly their tests from their favorite IDE.
OSGi defines Execution-Environment. The minimum environment is J2ME Foundation Profile 1.1. The test framework should provide a mode allowing executing test on this environment. Indeed, sometimes we want to check that everything works on a production-like environment or on different Java Virtual Machine.
Finally, the test framework should provide a way to execute the test manually on a “regular” platform. This way allows debugging what does not work. The developer recreates the environment or uses the production environment, deploys its tests and executes them manually.
Junit4osgi / pax:exam : The comparison
Test Engine
- Junit4OSGi: junit3 only
- Pax:Exam: sub-set of junit4 (no afterClass and beforeClass), possibility to support others test engines.
- Junit4OSGi: Helper pattern (osgi (bundle, service, package admin), ipojo, config admin, event admin…)
- Pax:Exam: just the bundle context
- Junit4OSGi: all maven dependencies that are bundles + system properties. Use Felix only
- Pax:Exam: @Configure method allowing to finely configure that framework (provisioning, Maven helper…). Allow running test on several configuration (OSGi implementation…).
- Junit4OSGi: one framework per Maven project
- Pax:Exam: One framework per @Test method
- Junit4OSGi: Maven integration, possibility to manually launch the tests
- Pax:Exam: Maven, any IDE (at least Eclipse)
- Junit4OSGi: J2ME Foundation Profile 1.1
- Pax:Exam: Java 5
- Junit4OSGi: Yes, all the tests are inside a bundle
- Pax:Exam: No, must use the regular test runner
I recently tried pax:exam and developed a couple of tests with it. It is really a promising framework, and the integration of the junit4osgi features is now on the roadmap. However, I have a couple of complaints and solutions:
- pax:exam fits very well to test OSGi applications, but to test framework we need a way to create on the fly bundles. Pax:exam does not provide natively such kind of feature. However, Tony Menzel developed tinybundle, a small and simple API to create bundle on the fly. I extended it a little bit to support iPOJO bundles. So, I’m able to create iPOJO bundles on the fly inside my tests! Definitely great.
- The pax:exam development model relies on the bundle context. As a junit4osgi power user, I want high-level method. I just externalized junit4OSGi helpers, and so was able to reuse all the junit4osgi methods.
- Pax:Exam is definitely slow… As it creates one framework per @Test method, it’s really inefficient when your test suite contains thousands of tests. I have 2000 tests with junit4osgi. Executing all of them takes almost 5 minutes (I have very expensive test with asynchronous event handling…). I also have a test suite developed with pax:exam of 20 tests. Such suite is executed in 3 minutes. So, there is definitely a performance / time issue. However, the pax:exam team is developing a Felix container as the junit4osgi one. This will improve performance.
- A way to execute junit3 tests (working on Foundation Profile 1.1) and TestNG tests would be great.
- A way to create test bundles “deployable” manually on a manually configured platform
So, in conclusion, pax:exam is a promising framework. I’m looking forward the next improvements and hope that junit4OSGi features will be provided soon.
Resources
Junit4osgi: http://felix.apache.org/site/apache-felix-ipojo-junit4osgi.html
Pax:exam: http://wiki.ops4j.org/display/paxexam/Pax+Exam
BND Testing Harness: http://felix.apache.org/site/bnd-testing-harness.html
Spring Integration Testing: http://springosgi.googlepages.com/ch04.html
Junit 3: http://junit.sourceforge.net/junit3.8.1/index.html
Junit 4: http://www.junit.org
TestNG: http://testng.org
Helper class to ease test writing: http://svn.apache.org/viewvc/felix/trunk/ipojo/tests/api/src/test/java/org/apache/felix/ipojo/tests/api/
Tinybundle: https://scm.ops4j.org/repos/ops4j/projects/pax/swissbox/pax-swissbox-tinybundles/
iPOJO extension to tinybundle: http://svn.apache.org/viewvc/felix/sandbox/clement/BundleAsiPOJO/
Posts
3 comments:
Have you looked at the DA-Testing Framework? Sounds like it has similar goals.
//Odd
Hi,
I just discovered it. Sounds interesting. I like the precondition idea (checking services before executing test...). It is a pretty useful feature.
IFAIK, this kind of preconditions are not supported in pax:exam.
I see that you use sleep (# sleep(50, MILLISECONDS);). It generally not a good smell, because time is definitely dependent of the platform. Can't you do such kind of things with a "waitForService" ?
Clement
Very nice post.
I like Junit4Osgi, combined with EclEmma to code coverage is the only way to solve my problem with OSGi application integration tests.
They are solving my needs.
But, I never used both with Maven, but it's my next step, so, I'll get a continuous integration test to my bundles, what is my next challenging.
ps: I would like to follow your blog with Google Friend Connect.
Post a Comment