Sling Testing Tools

Sling provides a number of testing tools to support the following use cases:

• Run JUnit tests contributed by OSGi bundles in an OSGi system. This does not require Sling and should work in other OSGi environments.
• Run scriptable tests in a Sling instance, using any supported scripting language.
• Run integration tests via HTTP against a Sling instance that is started during the Maven build cycle, or independently.

This page describes those tools, and points to the bundles that implement them.

The testing/samples/integration-tests module demonstrates these tools, and is also meant as a sample project to show how to run integration tests for Sling-based applications.

The main Sling integration tests at launchpad/integration-tests were created before this testing framework, and do not use it yet (as of March 2011). The new testing tools are simpler to use, but the "old" tests (all 400 of them as I write this) fulfill their validation role for testing Sling itself, there's no real need to modify them to use the new tools.

See also Testing Sling-based applications which discusses testing in general.

Server-side JUnit tests contributed by bundles

The services provided by the org.apache.sling.junit.core bundle allow bundles to register JUnit tests, which are executed server-side by the JUnitServlet registered by default at /system/sling/junit. This bundle is not dependent on Sling, it should work in other OSGi environments.

To try the JUnitServlet interactively, install the org.apache.sling.testing.samples.sampletests bundle.

This bundle contains a number of test classes, which are registered with the org.apache.sling.junit.core services by way of the Sling-Test-Regexp=.*Test bundle header, defined in the bundle's pom.xml. The JUnit core services use this regular expression to select which classes of the test bundle should be executed as JUnit tests.

To list the available tests, open http://localhost:8080/system/sling/junit/ . The servlet shows available tests, and allows you to execute them via a POST request.

Adding a path allows you to select a specific subset of tests, as in http://localhost:8080/system/sling/junit/org.apache.sling.junit.remote.html - the example integration tests described below use this to selectively execute server-side tests. The JUnitServlet provides various output formats, including in particular JSON, see http://localhost:8080/system/sling/junit/.json for example.

To supply tests from your own bundles, simply export the tests classes and add the Sling-Test-Regexp header to the bundle so that the Sling JUnit core services register them as tests.

Injection of OSGi services

The @TestReference annotation is used to inject OSGi services in tests that are executed server side.The BundleContext can also be injected in this way, see the OsgiAwareTest for an example.

Curl examples

Here's an example executing a few tests using curl:

$ curl -X POST http://localhost:8080/system/sling/junit/org.apache.sling.testing.samples.sampletests.JUnit.json
[{
    "INFO_TYPE": "test",
    "description": "testPasses(org.apache.sling.testing.samples.sampletests.JUnit3Test)"
  },{
    "INFO_TYPE": "test",
    "description": "testPasses(org.apache.sling.testing.samples.sampletests.JUnit4Test)"
  },{
    "INFO_TYPE": "test",
    "description": "testRequiresBefore(org.apache.sling.testing.samples.sampletests.JUnit4Test)"
  }
]

And another example with a test that fails:

$ curl -X POST http://localhost:8080/system/sling/junit/org.apache.sling.testing.samples.failingtests.JUnit4FailingTest.json

Scriptable server-side tests

If the org.apache.sling.junit.scriptable bundle is active in a Sling system, (in addition to the org.apache.sling.junit.core bundle), scriptable tests can be executed by the JUnitServlet according to the following rules:

• A node that has the sling:Test mixin is a scriptable test node.
• For security reasons, scriptable test nodes are only executed as tests if they are found under /libs or /apps, or more precisely under a path that's part of Sling's ResourceResolver search path.
• To execute a test, the scriptable tests provider makes an HTTP requests to the test node's path, with a .test.txt selector and extension, and expects the output to contain only the string TEST_PASSED. Empty lines and comment lines starting with a hash sign (#) are ignored in the output, and other lines are reported as failures.

The ScriptableTestsTest class, from the integration test samples module described below, sets up such a test node and its accompanying script, and calls the JUnitServlet to execute the test. It can be used as a detailed example of how this works.

Integration tests example

The testing/samples/integration-tests module runs some simple integration tests against a Sling Launchpad instance that's setup from scratch before running the tests.

This module's pom and Java code can be used as examples to setup your own integration testing modules for Sling-based apps - or for any other runnable jar that provides an http service.

Besides serving as examples, some of the tests in this module are used to validate the testing tools. They run as part of the full Sling continuous integration build, so they're guaranteed to be correct examples if that build is successful.

The sample uses the testing/tools to make the test code simpler. See the [OsgiConsoleTest|https://svn.apache.org/repos/asf/sling/trunk/testing/samples/integration-tests/src/test/java/org/apache/sling/testing/samples/integrationtests/http/OsgiConsoleTest.java] class for an example of a test that's very readable and requires no test setup or boilerplate code.

The following steps are executed in the integration-test phase of this module's Maven build: 1. A random port number for the Sling server is selected by the Maven build helper plugin, unless explicitely set (see pom.xml for such options). 1. Additional bundles, defined in the module's pom, are downloaded from the Maven repository in the target/sling/additional-bundles folder. 1. The first test that inherits from the SlingTestBase class causes the Sling runnable jar (defined as a dependency in the module's pom) to be started. 1. # The SlingTestBase class waits for the Sling server to be ready, based on URLs and expected responses defined in the pom. 1. # The SlingTestBase class installs and starts the bundles found in the target/sling/additional-bundles folder. 1. The test can now either test Sling directly via its http interface, or use the JUnitServlet to execute server-side tests contributed by bundles or scripts, as described above. 1. The Sling runnable jar is stopped when the test VM exits. 1. The test results are reported via the usual Maven mechanisms.

If -DkeepJarRunning is used on the Maven command line, the Sling runnable jar does not exit, to allow for running individual tests against this instance, for example when debugging the tests or the server code. See the pom for details.

Remote test execution

The testing tools support two types of remote test execution.

SlingRemoteTestRunner

The SlingRemoteTestRunner is used to run tests using the JUnitServlet described above. In this case, the client-side JUnit test only defines which tests to run and some optional assertions. Checking the number of tests executed, for example, can be useful to make sure all test bundles have been activated as expected, to avoid ignoring missing test bundles.

See the ServerSideSampleTest class for an example.

It's a good idea to check that the JUnit servlet is ready before running those tests, see the ServerSideTestsBase for an example of how to do that.

SlingRemoteExecutionRule

The SlingRemoteExecutionRule is a JUnit Rule that allows tests to be executed remotely in a Sling instance from an IDE, assuming the test is available on both sides.

The ExampleRemoteTest class demonstrates this. To run it from your IDE, set the sling.remote.test.url in the IDE to the URL of the JUnitServlet, like http://localhost:8080/system/sling/junit for example.