development testing

Sling Testing Tools

While this information is still valid, we recommend using the tools and techniques described in the newer "Junit Server-Side Tests Support" page instead, see link below.

As mentioned above, this is now replaced by the Junit Server-Side Tests Support page.

Sling provides a number of testing tools to support the following use cases:

Run JUnit tests contributed by OSGi bundles in an OSGi system. This does not require Sling and should work in other OSGi environments.
Run scriptable tests in a Sling instance, using any supported scripting language.
Run integration tests via HTTP against a Sling instance that is started during the Maven build cycle, or independently.

This page describes those tools, and points to the bundles that implement them.

The main Sling integration tests at launchpad/integration-tests were created before this testing framework, and do not use it yet (as of March 2011). The new testing tools are simpler to use, but the "old" tests (all 400 of them as I write this) fulfill their validation role for testing Sling itself, there's no real need to modify them to use the new tools.

See also Testing Sling-based applications which discusses testing in general.

Required bundles

These tools require a number of bundles on top of the standard Sling launchpad. See the for instance Sling Models IT pom.xml for an up-to-date list. Look for sling.additional.bundle.* entries in that pom for the bundle artifact IDs, and see the dependencies section for their version numbers.

Server-side JUnit tests contributed by bundles

The services provided by the org.apache.sling.junit.core bundle allow bundles to register JUnit tests, which are executed server-side by the JUnitServlet registered by default at /system/sling/junit. This bundle is not dependent on Sling, it should work in other OSGi environments.

Note that the JUnitServlet does not require authentication, so it would allow any client to run tests. The servlet can be disabled by configuration if needed, but in general the `/system` path should not be accessible to website visitors anyway.

For tighter integration with Sling, the alternate `SlingJUnitServlet` is registered with the `sling/junit/testing` resource type and `.junit` selector, if the bundle is running in a Sling system. Using this servlet instead of the plain JUnitServlet also allows Sling authentication to be used for running the tests, and the standard Sling request processing is used, including servlet filters for example.

To try the JUnitServlet interactively, install a bundle that contains tests registered via the Sling-Test-Regexp=.*Test bundle header.

The JUnit core services use this regular expression to select which classes of the test bundle should be executed as JUnit tests.

To list the available tests, open http://localhost:8080/system/sling/junit/ . The servlet shows available tests, and allows you to execute them via a POST request.

Adding a path allows you to select a specific subset of tests, as in http://localhost:8080/system/sling/junit/org.apache.sling.junit.remote.html - the example integration tests described below use this to selectively execute server-side tests. The JUnitServlet provides various output formats, including in particular JSON, see http://localhost:8080/system/sling/junit/.json for example.

To supply tests from your own bundles, simply export the tests classes and add the Sling-Test-Regexp header to the bundle so that the Sling JUnit core services register them as tests.

Injection of OSGi services

The @TestReference annotation is used to inject OSGi services in tests that are executed server side.The BundleContext can also be injected in this way.

Curl examples

Here's an example executing a few tests using curl:

$ curl -X POST http://localhost:8080/system/sling/junit/org.apache.sling.testing.samples.sampletests.JUnit.json
[{
    "INFO_TYPE": "test",
    "description": "testPasses(org.apache.sling.testing.samples.sampletests.JUnit3Test)"
  },{
    "INFO_TYPE": "test",
    "description": "testPasses(org.apache.sling.testing.samples.sampletests.JUnit4Test)"
  },{
    "INFO_TYPE": "test",
    "description": "testRequiresBefore(org.apache.sling.testing.samples.sampletests.JUnit4Test)"
  }
]

And another example with a test that fails:

$ curl -X POST http://localhost:8080/system/sling/junit/org.apache.sling.testing.samples.failingtests.JUnit4FailingTest.json

Scriptable server-side tests

If the org.apache.sling.junit.scriptable bundle is active in a Sling system, (in addition to the org.apache.sling.junit.core bundle), scriptable tests can be executed by the JUnitServlet according to the following rules:

A node that has the sling:Test mixin is a scriptable test node.
For security reasons, scriptable test nodes are only executed as tests if they are found under /libs or /apps, or more precisely under a path that's part of Sling's ResourceResolver search path.
To execute a test, the scriptable tests provider makes an HTTP request to the test node's path, with a .test.txt selector and extension, and expects the output to contain only the string TEST_PASSED. Empty lines and comment lines starting with a hash sign (#) are ignored in the output, and other lines are reported as failures.

Here's a minimal example that sets up and executes a scriptable test:

$ curl -u admin:admin -Fjcr:primaryNodeType=sling:Folder -Fsling:resourceType=foo -Fjcr:mixinTypes=sling:Test http://localhost:8080/apps/foo
...
$ echo TEST_PASSED > /tmp/test.txt.esp ; curl -u admin:admin -T/tmp/test.txt.esp http://localhost:8080/apps/foo/test.txt.esp

At this point, foo.test.txt is what the scriptable test framework will request, and that outputs just TEST_PASSED:

$ curl -u admin:admin http://localhost:8080/apps/foo.test.txt
TEST_PASSED

And a POST to the JUnit servlet returns information on the test's execution:

curl -u admin:admin -XPOST http://localhost:8080/system/sling/junit/org.apache.sling.junit.scriptable.ScriptableTestsProvider.json
[{
    "INFO_TYPE": "test",
    "description": "verifyContent[0](org.apache.sling.junit.scriptable.TestAllPaths)",
    "test_metadata": {
      "test_execution_time_msec": 2
    }
  }
]

Test failures would be included in this JSON representation - you can test that by modifying the script to fail and making the same request again.

Integration tests example

The IT Samples modules run some simple integration tests against a Sling Launchpad instance that's setup from scratch before running the tests.

Either of those modules can be used as examples to setup your own integration testing modules for Sling-based apps - or for any other runnable jar that provides an http service.

Besides serving as examples, some of the tests in this module are used to validate the testing tools. They run as part of the full Sling continuous integration build, so they're guaranteed to be correct examples if that build is successful.

The information below is outdated and needs an update, please rather directly check the source files linked above.

The sample uses the testing/tools to make the test code simpler. See the [OsgiConsoleTest|https://svn.apache.org/repos/asf/sling/trunk/testing/samples/integration-tests/src/test/java/org/apache/sling/testing/samples/integrationtests/http/OsgiConsoleTest.java] class for an example of a test that's very readable and requires no test setup or boilerplate code.

The following steps are executed in the integration-test phase of this module's Maven build:

A random port number for the Sling server is selected by the Maven build helper plugin, unless explicitely set (see pom.xml for such options).
Additional bundles, defined in the module's pom, are downloaded from the Maven repository in the target/sling/additional-bundles folder.
The first test that inherits from the SlingTestBase class causes the Sling runnable jar (defined as a dependency in the module's pom) to be started.
The SlingTestBase class waits for the Sling server to be ready, based on URLs and expected responses defined in the pom.
The SlingTestBase class installs and starts the bundles found in the target/sling/additional-bundles folder.
The test can now either test Sling directly via its http interface, or use the JUnitServlet to execute server-side tests contributed by bundles or scripts, as described above.
The Sling runnable jar is stopped when the test VM exits.
The test results are reported via the usual Maven mechanisms.

If -DkeepJarRunning is used on the Maven command line, the Sling runnable jar does not exit, to allow for running individual tests against this instance, for example when debugging the tests or the server code. See the pom for details.

Running tests against existing server

Instead of provisioning a completely new Sling server, the ITs can also be executed on an already existing server instance. For that the test-server-url system property has to point to the existing server url. Additional bundles can still be deployed by using the sling.additional.bundle.<num> system property.

Optionally, the additional bundles can be undeployed after the execution of the IT by setting additional.bundles.uninstall to true. (since Sling Testing Tools 1.0.12, SLING-4819)

Remote test execution

The testing tools support two types of remote test execution.

SlingRemoteTestRunner

The SlingRemoteTestRunner is used to run tests using the JUnitServlet described above. In this case, the client-side JUnit test only defines which tests to run and some optional assertions. Checking the number of tests executed, for example, can be useful to make sure all test bundles have been activated as expected, to avoid ignoring missing test bundles.

See the ServerSideSampleTest class for an example.

It's a good idea to check that the JUnit servlet is ready before running those tests, see the ServerSideTestsBase for an example of how to do that.

SlingRemoteExecutionRule

The SlingRemoteExecutionRule is a JUnit Rule that allows tests to be executed remotely in a Sling instance from an IDE, assuming the test is available on both sides.

The ExampleRemoteTest class demonstrates this. To run it from your IDE, set the sling.remote.test.url in the IDE to the URL of the JUnitServlet, like http://localhost:8080/system/sling/junit for example.

Debugging ITs

The JVM is usually forked twice during the execution of integration tests. The first time by the maven-surefire-plugin which executes the client-side (i.e. Maven-side) part of the tests. To debug this side the option -Dmaven.surefire.debug can be used which waits for a debugger to be attached on port 5005 before the (client-side) test is executed. More information is available in the documentation of the maven-surefire-plugin.

Then the JarExecutor is forking the VM a second time to start the server (this does not happen if connecting to an already running instance). The system environment variable jar.executor.vm.options can be used to start that VM with debug options. All debug options are described at the JPDA documentation. If running

mvn test -Djar.executor.vm.options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000"

the server start is interrupted until a debugger is connected on port 8000.