Running tests sequentially seems to be the current status quo in the Java community, despite the number of CPU cores our computers have these days. On the other hand, executing all of them in parallel may look great on paper, but it’s often easier said than done, especially in the already existing projects.

With version 5.3, the JUnit framework has introduced experimental support for the parallel test execution, which can allow selective test parallelization driven by the code. Instead of an exhaustive overview of this feature (the official User Guide does a great job here), I’d like to propose a pragmatic solution, that should be applicable to many types of projects. You can think of it as a low-hanging fruit of test parallelization.

For the impatients (the plan)

The proposed approach consists of three steps:

1. Enable JUnit 5 parallel tests execution but run everything sequentially by default (status quo).
2. Create custom @ParallelizableTest annotation promoting class-level parallelization (all the tests methods inside will be executed in parallel).
3. Enable parallel execution for selected tests starting from unit tests (safe default).

Complete configuration (together with a few example tests cases) is available on GitHub.

Enabling parallel execution

First, let’s enable JUnit parallel execution by creating the junit-platform.properties file (under src/test/resources) with the following content:

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = same_thread

Apart from enabling the feature itself, it also specifies that both: test classes and their test methods, should be executed sequentially. This preserves the previous behavior by default, where tests are executed one by one by the same thread.

Alternatively, we can specify JUnit configuration via Maven Surefire within pom.xml:

<build>
  <plugins>
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.0.0-M5</version>
      <configuration>
        <properties>
          <configurationParameters>
            junit.jupiter.execution.parallel.enabled = true
            junit.jupiter.execution.parallel.mode.default = same_thread
            junit.jupiter.execution.parallel.mode.classes.default = same_thread
          </configurationParameters>
        </properties>
      </configuration>
    </plugin>
  </plugins>
</build>

In fact, I’d advise combining both approaches, by keeping complete configuration in junit-platform.properties but allowing to enable/disable parallel test execution via dedicated system property:

<project>
  <properties>
    <parallelTests>true</parallelTests>
  </properties>

  <build>
    <plugins>
      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M5</version>
        <configuration>
          <properties>
            <configurationParameters>
              junit.jupiter.execution.parallel.enabled = ${parallelTests}
            </configurationParameters>
          </properties>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

This way, selected tests will be run in parallel by default, but they can still be run sequentially on-demand with mvn clean verify -DparallelTests=false.

Note: with all the advanced/non-standard JUnit 5 features it’s worth switching the Surefire version to the 3.x branch, because of various compatibility improvements introduced there.

Going parallel with annotations

By annotating a test class or a test method with Junit 5 @Execution we can control its parallel execution. Let’s take a look at the small example:

@Execution(ExecutionMode.CONCURRENT) // note: propagates downstream!
class MyParallelTest { // runs in parallel with other test classes

    @Test
    @Execution(ExecutionMode.CONCURRENT)
    void shouldVerifySomethingImportant() {
        // runs in parallel with other test cases
        // (would behave the same without the annotation - inherited)
    }
    
    @Test
    @Execution(ExecutionMode.SAME_THREAD)
    void shouldVerifySomethingImportantSequentially() {
        // runs in the same thread as its parent (override)
    }
    
    // ...

}

Such an annotation applied at the class level will have an impact on all the non-annotated test cases it has inside. So, once we enable concurrent execution on the test class level, all its test cases would be executed in parallel as well. This means, such a technique should be used when test cases are completely independent of each other.

Luckily, JUnit 5 already improves the separation between test cases for us:

In order to allow individual test methods to be executed in isolation and to avoid unexpected side effects due to mutable test instance state, JUnit creates a new instance of each test class before executing each test method

This means that even if we have some shared non-static fields (e.g. for mocks), each test case will get its own instances. This makes enabling concurrent execution on the test class level safe enough for most of the use cases.

In order to promote such a class-level parallelization, we can create our own @ParallelizableTest annotation, which (unlike the JUnit one) can’t be used on the test case (method) level:

@Execution(ExecutionMode.CONCURRENT) // <- the original JUnit annotation
@TestInstance(TestInstance.Lifecycle.PER_METHOD)
// ^ makes the default "safe behavior" explicit
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE) // class-level only
public @interface ParallelizableTest {
}

Because of the sequential by default settings from junit-platform.properties, only the test classes annotated with @ParallelizableTest could be now run in parallel. This allows us to make test-by-test choices easily.

@ParallelizableTest
class MyParallelTest { // runs in parallel with other test classes
    // ...
}

In existing projects with a lot of test code inside, such a technique could be used to iteratively increase the number of parallel tests over time.

The proposed approach has two advantages compared to relying on built-in annotations. First, it prevents us from using them too often - for example by mixing class and test case level declarations with no good reason. Secondly, it explicitly enables “separate instance per method” semantics, so we’re no longer dependent on the overridable defaults.

Choosing what to parallelize

Finally, with all the machinery available, we have to decide what to parallelize first. The answer could be found in a good-old test pyramid.

It shouldn’t be a surprise, that the unit tests are the easiest ones to parallelize first. Usually, the only thing required to do so would be annotating them with @ParallelizableTest, as the rest should still work. Despite being the least beneficial in terms of reducing the total execution time, the low effort makes parallelizing them almost free. In fact, doing so emphasizes their inherent isolation from other tests.

Note: there seems to be a lot of controversy around what a “unit test” really means. To avoid confusion, I rely on the definition from the “Unit Testing: Principles, Practices, and Patterns” book by Vladimir Khorikov here:

A unit test verifies a single unit of behavior, does it quickly and does it in isolation from other tests.

As the next step, you may want to select other test classes and parallelize them too. While those may result in significant execution time reduction, the required effort could increase as well. For example, re-using the same DB instance may require proper data randomization in all parallelized tests in order to prevent cross-interference. In some use cases, more sophisticated synchronization options of Junit 5 could be helpful too.

Especially for some non-trivial tests, the costs of parallelization may even outweigh the profits due to the complexity introduced. This is why I perceive selective test parallelization so beneficial.

Putting it all together

For the purpose of this article, I’ve created a small example project consisting of 6 test classes with 3 test cases each (named A, B, and C). Half of them can be run in parallel using the configuration described above. Each test case prints its thread name, class & case name on start and end. Running mvn clean verify (and limiting the parallelism to 6 threads via configuration) seems to prove the correctness of the proposed setup:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.mikemybytes.junit.parallel.Parallel1Test
[INFO] Running com.mikemybytes.junit.parallel.Parallel2Test
[INFO] Running com.mikemybytes.junit.parallel.Parallel3Test
[INFO] Running com.mikemybytes.junit.sequential.Sequential3Test
[ForkJoinPool-1-worker-5] START: Parallel3Test#A
[ForkJoinPool-1-worker-6] START: Parallel3Test#B
[ForkJoinPool-1-worker-4] START: Parallel3Test#C
[ForkJoinPool-1-worker-3] START: Parallel2Test#C
[ForkJoinPool-1-worker-1] START: Sequential3Test#A
[ForkJoinPool-1-worker-2] START: Parallel1Test#C
[ForkJoinPool-1-worker-6]   END: Parallel3Test#B
[ForkJoinPool-1-worker-6] START: Parallel2Test#A
[ForkJoinPool-1-worker-3]   END: Parallel2Test#C
[ForkJoinPool-1-worker-4]   END: Parallel3Test#C
[ForkJoinPool-1-worker-3] START: Parallel2Test#B
[ForkJoinPool-1-worker-7] START: Parallel1Test#A
[ForkJoinPool-1-worker-5]   END: Parallel3Test#A
[ForkJoinPool-1-worker-1]   END: Sequential3Test#A
[ForkJoinPool-1-worker-2]   END: Parallel1Test#C
[ForkJoinPool-1-worker-1] START: Sequential3Test#B
[ForkJoinPool-1-worker-5] START: Parallel1Test#B
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.784 s - in com.mikemybytes.junit.parallel.Parallel3Test
[ForkJoinPool-1-worker-6]   END: Parallel2Test#A
[ForkJoinPool-1-worker-1]   END: Sequential3Test#B
[ForkJoinPool-1-worker-1] START: Sequential3Test#C
[ForkJoinPool-1-worker-3]   END: Parallel2Test#B
[ForkJoinPool-1-worker-7]   END: Parallel1Test#A
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.284 s - in com.mikemybytes.junit.parallel.Parallel2Test
[ForkJoinPool-1-worker-5]   END: Parallel1Test#B
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.289 s - in com.mikemybytes.junit.parallel.Parallel1Test
[ForkJoinPool-1-worker-1]   END: Sequential3Test#C
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.547 s - in com.mikemybytes.junit.sequential.Sequential3Test
[INFO] Running com.mikemybytes.junit.sequential.Sequential2Test
[ForkJoinPool-1-worker-1] START: Sequential2Test#A
[ForkJoinPool-1-worker-1]   END: Sequential2Test#A
[ForkJoinPool-1-worker-1] START: Sequential2Test#B
[ForkJoinPool-1-worker-1]   END: Sequential2Test#B
[ForkJoinPool-1-worker-1] START: Sequential2Test#C
[ForkJoinPool-1-worker-1]   END: Sequential2Test#C
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.775 s - in com.mikemybytes.junit.sequential.Sequential2Test
[INFO] Running com.mikemybytes.junit.sequential.Sequential1Test
[ForkJoinPool-1-worker-1] START: Sequential1Test#A
[ForkJoinPool-1-worker-1]   END: Sequential1Test#A
[ForkJoinPool-1-worker-1] START: Sequential1Test#B
[ForkJoinPool-1-worker-1]   END: Sequential1Test#B
[ForkJoinPool-1-worker-1] START: Sequential1Test#C
[ForkJoinPool-1-worker-1]   END: Sequential1Test#C
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.024 s - in com.mikemybytes.junit.sequential.Sequential1Test
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0

The tests marked as parallelizable run in parallel with each other (as expected) but also in parallel with the sequential ones. This may look a bit suspicious at first. However, as our parallelizable tests claim to be independent of the others, they should make no harm even to those running one by one. Additionally, all the sequential tests were executed on the same thread (ForkJoinPool-1-worker-1).

Limitations

For the time of writing, the main limitation of the proposed approach is related to the accuracy of the Maven Surefire plugin reports being generated for tests execution. In the example project, there were 3 test classes being executed in parallel with 3 test cases each. This means we should expect 9 tests to be reported in total. However, the reports available at target/surefire-reports seem to indicate something different.

-------------------------------------------------------------------------------
Test set: com.mikemybytes.junit.parallel.Parallel1Test
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.289 s - in com.mikemybytes.junit.parallel.Parallel1Test

-------------------------------------------------------------------------------
Test set: com.mikemybytes.junit.parallel.Parallel2Test
-------------------------------------------------------------------------------
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.284 s - in com.mikemybytes.junit.parallel.Parallel2Test

-------------------------------------------------------------------------------
Test set: com.mikemybytes.junit.parallel.Parallel3Test
-------------------------------------------------------------------------------
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.784 s - in com.mikemybytes.junit.parallel.Parallel3Test

This is a known Surefire limitation (see SUREFIRE-1643 and SUREFIRE-1795 tickets) not related to JUnit 5 in particular - the reporting part supports only a sequence of test events.

That’s why allowing on demand sequential execution as described before is so important. If anything goes wrong (or a certain tool relies on the generated report), we can still run the tests one-by-one with -DparallelTests=false.

Summary

Parallel test execution introduced in JUnit 5 is a simple but yet powerful tool for utilizing our hardware resources better in order to shorten the feedback loop. Running only selected tests in parallel gives us full control over test execution and the “speed vs effort” trade-offs. Thanks to that, we can gradually increase the parallelism of our tests instead of changing everything at once. Despite some limitations, parallel test execution is already a useful addition to our development toolbox.