Monday, November 24, 2008

22.7 Mb!

Today, all began perfectly.
I moved the junit4osgi plug-in to the iPOJO Trunk and decided with junit4osgi users to move the junit4osgi projects as ‘iPOJO top level’ project. And, icing on the cake, I finished my PHD Defense slides.

Moreover, I was pretty proud of my new script checking iPOJO current trunk:

mkdir tmp
cd tmp
svn co https://svn.apache.org/repos/asf/felix/trunk/ipojo
cd ipojo
mvn clean install -Pexamples,tests
mvn rat:check
mvn checkstyle:check
cd tests
cd integration-tests
mvn clean integration-test
mvn org.apache.maven.plugins:maven-surefire-report-plugin:report
/usr/bin/osascript -e 'tell application "Safari" to activate open location "file:/Users/clement/tmp/ipojo/tests/integration-tests/target/site/surefire-report.html"'

However, I executed it from scratch (empty maven repository), and here arrives the issue …
22.7 Mb is the size of my maven repository after the execution of the script ! It just downloads and compiles iPOJO trunk (not the whole Felix project) and executes tests.

After a quick analysis, some artifacts were downloaded in multiple versions. So, it’s time for me to track some of these artifacts, and to try to reduce this size. I’m pretty sure, that I can’t drastically reduce this mess, anyway, I’ll try.

Here are the duplicated artifacts:

commons-collections-2.1.jar
commons-collections-3.2.jar
commons-logging-1.0.3.jar
commons-logging-1.0.4.jar
commons-validator-1.1.4.jar
commons-validator-1.2.0.jar
org.osgi.compendium-1.0.0.jar
org.osgi.compendium-1.2.0.jar
org.osgi.core-1.0.0.jar
org.osgi.core-1.0.1.jar
org.osgi.core-1.2.0.jar
org.osgi.foundation-1.0.0.jar
org.osgi.foundation-1.2.0.jar
doxia-core-1.0-alpha-10.jar
doxia-core-1.0-alpha-8.jar
doxia-decoration-model-1.0-alpha-10.jar
doxia-decoration-model-1.0-alpha-11.jar
doxia-decoration-model-1.0-alpha-8.jar
doxia-site-renderer-1.0-alpha-10.jar
doxia-site-renderer-1.0-alpha-8.jar
maven-archiver-2.0.jar
maven-archiver-2.2.jar
maven-archiver-2.3.jar
maven-artifact-2.0.jar
maven-artifact-2.0.7.jar
maven-artifact-manager-2.0.jar
maven-artifact-manager-2.0.7.jar
maven-model-2.0.jar
maven-model-2.0.7.jar
maven-plugin-api-2.0.jar
maven-plugin-api-2.0.7.jar
maven-profile-2.0.jar
maven-profile-2.0.7.jar
maven-project-2.0.jar
maven-project-2.0.7.jar
maven-repository-metadata-2.0.jar
maven-repository-metadata-2.0.7.jar
maven-jar-plugin-2.1.jar
maven-jar-plugin-2.2.jar
maven-plugin-plugin-2.3.jar
maven-plugin-plugin-2.4.1.jar
maven-surefire-plugin-2.3.jar
maven-surefire-plugin-2.4.2.jar
maven-reporting-impl-2.0.jar
maven-reporting-impl-2.0.4.jar
maven-reporting-impl-2.0.4.1.jar
surefire-api-2.3.jar
surefire-api-2.4.2.jar
surefire-booter-2.3.jar
surefire-booter-2.4.2.jar
wagon-provider-api-1.0-alpha-5.jar
wagon-provider-api-1.0-beta-2.jar
plexus-archiver-1.0-alpha-3.jar
plexus-archiver-1.0-alpha-7.jar
plexus-archiver-1.0-alpha-9.jar
plexus-container-default-1.0-alpha-8.jar
plexus-container-default-1.0-alpha-9-stable-1.jar
plexus-i18n-1.0-beta-6.jar
plexus-i18n-1.0-beta-7.jar
plexus-utils-1.0.4.jar
plexus-utils-1.1.jar
plexus-utils-1.4.jar
plexus-utils-1.4.1.jar
plexus-utils-1.4.4.jar
plexus-utils-1.4.5.jar
plexus-utils-1.4.7.jar
plexus-utils-1.4.9.jar
plexus-utils-1.5.1.jar
plexus-velocity-1.1.2.jar
plexus-velocity-1.1.3.jar
plexus-velocity-1.1.7.jar
oro-2.0.7.jar
oro-2.0.8.jar

Tuesday, November 11, 2008

maven-junit4osgi-plugin

Update: the junit4osgi artifact has recently changed. The groupID became org.apache.felix. Moreover, the plugin is now in the iPOJO trunk.

In my last post, I explain the advantages of integrating integration test in the build process. However, such tools was inexistent until … now ☺

As I explained, the junit4osgi framework allows testing an OSGi applications with a "junit++" framework (i.e. an adapted junit distribution providing useful methods to handle easily OSGi specific features such as services).

But, the junit4osgi framework is a “standalone” framework and is not integrated in a build tools such as Ant or Maven. This post describes a maven plug-in, maven-junit4osgi-plugin, executing tests in a maven-based build process.

Constraints of such tools
Providing such front-end is useful but should meet some requirements:
Tests are executed “in container”. So, tested bundles are really deployed on an OSGi framework (I choose Apache Felix). So tests are executing in an execution environment close to the final (production) execution environment. The goal of integration tests is to test the application in a production-like environment.
  • The project under construction is not a functional bundle, but can either be empty or contain integration tests. Integration tests are not placed in the same project as the project under test. Integration tests are packaged inside others bundles deployed on the same framework as the application under test..
  • To recreate the “final” execution environment, several bundles can be required (technical services, the application under test…). So the plug-in must support the deployment of required bundles.
  • Test results must be reported to the user. Maven provides an infrastructure to create web site. Moreover, Surefire (the Maven “regular” test plug-in) provides a plug-in generating a web page with test result. The provided plug-in should provide the same features and should reuse the same format as Surefire.
The good news is that the provided plug-in, maven-junit4osgi-plugin, match these requirements!

What does the maven-junit4osgi-plugin provide?
  • Allows testing OSGi applications
  • Integrated in a Maven-based build process
  • Provides the same output as Surefire
  • Supports Maven site generation
Using the plug-in

Download and building the plug-in
The plug-in sources are available in the iPOJO trunk.
However the junit4osgi and iPOJO runtime are also required. So, download the source of iPOJO:
svn co http://svn.apache.org/repos/asf/felix/trunk/ipojo/
To compile it, run the following commands:
cd ipojo
mvn clean install –Pexamples
(the –Pexamples allows compiling iPOJO examples and so the junit4osgi framework ans the plug-in). Now you can use the plug-in in your project.

Simple configuration
So, first the project using the plug-in is not the project under test. It’s another project containing either only integration-test packaged in a bundle, or is empty (and so depends on other bundles containing integration tests).
Tests contained in the project are developed with junit4osgi, and are packaged in a bundle with the maven-bundle-plugin.
In the pom file, add the following plugin configuration to use the maven-junit4osgi-plugin:

<plugin>
<groupid>org.apache.felix</groupid>
<artifactid>maven-junit4osgi-plugin</artifactid>
<executions>
<execution>
<goals>
<goal>test</goal>
</goals>
<configuration>
<deployprojectartifact>true</deployprojectartifact>
</configuration>
</execution>
</executions>
</plugin>

Plugin parameter
The plug-in has only one parameter. The 'deployProjectArtifact' parameter enables or disables the current artifact deployment. If the current project contains tests, the plug-in can deploy the built artifact (as illustrated in this pom). Otherwise, the current project artifact is not deployed. This can be useful if the project just depends on other test bundles and sets the test configuration (as this pom).

Configuring the set of bundles to deploy

There is two different ways to configure the plug-in to deploy other bundles. If the bundle to deploy is a maven artifact, just add this artifact as a maven project dependency and set the dependency scope to ‘test’. Here is an example:

<dependency>
<artifactid>tests.manipulation.metadata</artifactid>
<groupid>ipojo.tests</groupid>
<version>1.1.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>

If your bundle is not a maven artifact, you can configure the plugin with the bundle URL (from where the bundle will be deployed)

<configuration>
<deployprojectartifact>true</deployprojectartifact>
<bundles>
<param>file:/Users/clement/bundles/test-metadata.jar</param>
</bundles>
</configuration>


Set bundles are installed and started. You can depend on bundle that does not contain test as well as bundle containing tests.

Executing the plug-in

To execute test, just launch the ‘mvn clean integration-test’ command.

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building iPOJO Primitive Manipulation Test Suite
[INFO] task-segment: [integration-test]
[INFO] ------------------------------------------------------------------------
[INFO] [resources:resources]
[INFO] Using default encoding to copy filtered resources.
[INFO] [compiler:compile]
[INFO] Nothing to compile - all classes are up to date
[INFO] [resources:testResources]
[INFO] Using default encoding to copy filtered resources.
[INFO] [compiler:testCompile]
[INFO] No sources to compile
[INFO] [surefire:test]
[INFO] No tests to run.
[INFO] [bundle:bundle]
[INFO] [ipojo:ipojo-bundle {execution: default}]
[INFO] Start bundle manipulation
[INFO] Metadata file : /Users/clement/Documents/workspaces/felix-trunk/ipojo/tests/manipulator/primitives/target/classes/metadata.xml
[INFO] Input Bundle File : /Users/clement/Documents/workspaces/felix-trunk/ipojo/tests/manipulator/primitives/target/tests.manipulation.primitives-1.1.0-SNAPSHOT.jar
[INFO] Bundle manipulation - SUCCESS
[INFO] [junit4osgi:test {execution: default}]
Analyzing org.apache.felix.ipojo - compile
Analyzing org.apache.felix.ipojo.metadata - compile
Analyzing org.osgi.core - compile
Analyzing junit - compile
Analyzing org.apache.felix.ipojo.junit4osgi - compile
Analyzing tests.manipulation.metadata - test

-------------------------------------------------------
T E S T S
-------------------------------------------------------
Deploy : /Users/clement/Documents/workspaces/felix-trunk/ipojo/tests/manipulator/primitives/target/tests.manipulation.primitives-1.1.0-SNAPSHOT.jar
Loading org.apache.felix.ipojo.test.scenarios.manipulation.ManipulationTestSuite
Loading org.apache.felix.ipojo.test.scenarios.manipulation.ManipulationTestSuite
Junit Extender starting ...
Running Manipulation Metadata Test Suite
Tests run: 16, Failures: 0, Errors: 0, Time elapsed: 0 sec
Running Primitive Manipulation Test Suite
Tests run: 17, Failures: 0, Errors: 0, Time elapsed: 0 sec

Results :

Tests run: 33, Failures: 0, Errors:0

Unload test suites [class org.apache.felix.ipojo.test.scenarios.manipulation.ManipulationTestSuite]
Unload test suites [class org.apache.felix.ipojo.test.scenarios.manipulation.ManipulationTestSuite]
Cleaning test suites ...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6 seconds
[INFO] Finished at: Mon Nov 10 21:30:21 CET 2008
[INFO] Final Memory: 9M/18M
[INFO] ------------------------------------------------------------------------

Failures and errors are reported in the plugin output.

Generating the report web page
When test are executed, the plug-in generates XML reports (int the target/junit4osgi-reports directory) using the same convention as Surefire. So, it is possible to configure Surefire to generate the web page with test results.
To do this, add the following report configuration to the project executing tests:

<reporting>
<plugins>
<plugin>
<groupid>org.apache.maven.plugins</groupid>
<artifactid>maven-surefire-report-plugin</artifactid>
<version>2.4.3</version>
<configuration>
<showsuccess>true</showsuccess>
<reportsdirectories>
<param>target/junit4osgi-reports</param>
</reportsdirectories>
</configuration>
</plugin>
</plugins>
</reporting>
This snippet configures the maven-surefire-report-plugin to collect results from the ‘target/junit4osgi-reports’ directory.
Then execute the plugin with the following command:
mvn org.apache.maven.plugins:maven-surefire-report-plugin:2.4.3:report

This command generates the web page with test results in ‘target/site’. This page shows an example of page generated with this command.

Plug-in design
The plug-in is quiet simple, it just starts an embedded Felix with a special activator installing and starting the junit4osgi framework and specified bundles.

Then, before executing test, the plug-in waits for “stability”. Indeed, as bundle activation can be asynchronous, the plug-in need to wait that the configuration is stable. Stability is obtained when all bundles are activated, and no new services appear or disappear on a 500 ms period. If after several second the stability cannot be reached, the plug-in stops.

Once the stability is reached, the junit4ogsi runner service is used to execute tests. Then results are collected and reports are generated.

Conclusion
This post has presented a front-end automating the execution of junit4osgi tests. Now it is possible to integrate OSGi application tests in a build process. The presented maven plugin provides following features:
  • An easy integration in a Maven-based build process
  • A good flexibility allowing reproducing production execution environments to test the application
  • Test result output is the same as surefire
  • Is able to generate Surefire-like reports

Sunday, November 2, 2008

Lessons learned from iPOJO testing process

Recently, we ask me several times if iPOJO is tied to the Felix runtime (i.e. works only on Felix). So, the answer is simple:

iPOJO relies only on the OSGi 4.1 specification, and so can work on any compliant implementation.

This post just explains how iPOJO is currently tested, and reflexion on OSGi application testing.

Test, Test and Test...

So, why testing?
Just because I'm very clumsy, and I can't guaranty that a patch don't have side effects broking a features. So, my tests aims to check that features still works and that I don't broke everything... That's why I developed junit4osgi at the same times as iPOJO. Before this improvement, testing iPOJO takes something like a working day! (now, it's close to 3 minutes...)

The iPOJO testing process weekly executed on three different OSGi implementations:
"Tested" means that the test suite is executed successfully on each implementation. This is done thanks to the junit4OSGi framework compatible with those implementations.
The test suite is also executed on different VMs such as Mika, JamVM and JRockit. Those tests are twofold:
  1. Checks that the iPOJO framework can be executed on different VMs.
  2. Checks that the iPOJO manipulation process generates consistent bytecode.
The second point is important, as the VMs are more or less tolerant to some mistakes ☺. Specially, Sun VMs are very tolerant!

So, let’s back to the testing process. The iPOJO test suite checks the most part of the iPOJO’s features. The iPOJO trunk contains a lot of test cases executed with the junit4osgi framework. Executing the test suite is quite simple: launch the OSGi implementation, launch the junit4osgi framework, deploy tests and finally use a front end (command line, GUI) to run tests.

Simple, isn’t it? However, during the development of the test suite, several issues appears:
- Handling asynchronous interactions
- Integrating test in the building process

Issues to test OSGi Applications

Testing services and their behavior is really easy with junit4OSGi. However, it becomes trickier when services realize asynchronous actions (i.e. actions are executed in a different thread and so not sequentially).

OSGi proposes an execution platform where a lot of actions are made asynchronously. Testing this kind of interaction is difficult, and generally requires magic waiting time.

For example, imagine a test pushing a configuration inside the configuration admin. The test creates a new configuration, pushes it to the configuration to the configuration admin, and checks that the configuration is correctly applied. However, the configuration admin applies the configuration in another thread. So the test must wait before checking that the configuration is correctly applied. Waiting is admitted, but how much time? Choosing a default time is rarely is good choice because it greatly depends on your configuration… Setting a long duration implies a long executing time…

So, you can argue that the Configuration Admin provide mechanisms notifying when configurations are correctly processed. That’s great, and the OSGi specification defines similar mechanisms for the most part of asynchronous interactions. So it should be possible to wait for this notification (or eventually throws a timeout).


However, if the ManagedService or the ManagedServiceFactory to update is not available, the notification is fired immediately, and is no more fired once the service arrives (at least on the Felix Configuration Admin). So, in this case you have to wait for an arbitrary time ☹. I’m looking for a better solution, but right now it still obscures to me how to handle those interactions cleanly.

The second issue with junit4osgi is very different. Executing tests are easy but I’m very (very) lazy. So, integrating those tests inside my build process could be great. An Ant task or a Maven plug-in could execute those tests automatically and react according to the results (if a failure or an error is detected, the build process fails). This automation can be easily implemented. It just requires starting an OSGi implementation, deploying the junit4osgi framework, starting required bundles as well as bundles containing tests and launching the test execution. It exist testing framework doing this, but they are generally usable only inside the build process and so lacks of flexibility. It must be possible to execute test manually on non-standard configurations. I’m going to implement a simple Ant task doing this. I’ll keep you posted…