Pragmatic tests parallelization with JUnit 5

testing Nov 24, 2021

Running tests sequentially seems to be the current status quo in the Java community, despite the number of CPU cores our computers have these days. On the other hand, executing all of them in parallel may look great on paper, but it's often easier said than done, especially in the already existing projects.

With version 5.3, the JUnit framework has introduced experimental support for the parallel test execution, which - with a bit of configuration - could allow selective test parallelization driven by the code. Instead of an exhaustive overview of this feature (the official User Guide does a great job here), I'd like to propose a pragmatic solution, that should be applicable to many types of projects. You can think of it as a low-hanging fruit of test parallelization.

For the impatients (the plan)

The proposed approach consists of three steps:

  1. Enable JUnit 5 parallel tests execution but run everything sequentially by default (status quo).
  2. Create custom @ParallelizableTest annotation promoting class-level parallelization (run tests classes in parallel, but all their methods sequentially within each class).
  3. Enable parallel execution for selected tests starting from unit tests (safe default).

Complete configuration (together with a few example tests cases) is available on GitHub.

Enabling parallel execution

First, let's enable JUnit parallel execution by creating the junit-platform.properties file (under src/test/resources) with the following content:

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = same_thread

Apart from enabling the feature itself, it also specifies that both: test classes and their test methods, should be executed sequentially. This preserves the previous behavior by default, where tests are executed one by one by the same thread.

Alternatively, we can specify JUnit configuration via Maven Surefire within pom.xml:

<build>
  <plugins>
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.0.0-M5</version>
      <configuration>
        <properties>
          <configurationParameters>
            junit.jupiter.execution.parallel.enabled = true
            junit.jupiter.execution.parallel.mode.default = same_thread
            junit.jupiter.execution.parallel.mode.classes.default = same_thread
          </configurationParameters>
        </properties>
      </configuration>
    </plugin>
  </plugins>
</build>

In fact, I'd advise combining both approaches, by keeping complete configuration in junit-platform.properties but allowing to enable/disable parallel test execution via dedicated system property:

<project>
  <properties>
    <parallelTests>true</parallelTests>
  </properties>

  <build>
    <plugins>
      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M5</version>
        <configuration>
          <properties>
            <configurationParameters>
              junit.jupiter.execution.parallel.enabled = ${parallelTests}
            </configurationParameters>
          </properties>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

This way, selected tests will be run in parallel by default, but they can still be run sequentially on-demand with mvn clean verify -DparallelTests=false.


Note: with all the advanced/non-standard JUnit 5 features it's worth switching the Surefire version to the 3.x branch, because of various compatibility improvements introduced there.


Going parallel with annotations

By annotating a test class or a test method with Junit 5 @Execution we can control its parallel execution. Because of the "sequential by default" settings provided via junit-platform.properties before, the only variant we care about is the one with ExecutionMode.CONCURRENT:

@Execution(ExecutionMode.CONCURRENT)
class MyParallelTest { // runs in parallel with other test classes

    @Test
    @Execution(ExecutionMode.CONCURRENT)
    void shouldVerifySomethingImportant() {
        // runs in parallel with other test cases within this class
    }
    
    // ...

}

As a starting point, I'd advise enabling parallelization only on the test classes-level leaving their test cases sequentially, as the proper isolation is usually easier to achieve this way. For example, we can still define mocks only once for a test class instead of instantiating them for each test case separately (to make sure they don't interfere with each other).

In order to promote class-level parallelization, we can create our own @ParallelizableTest annotation, which (unlike the JUnit one) can't be used on the test case (method) level:

@Execution(ExecutionMode.CONCURRENT) // the original JUnit annotation
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
public @interface ParallelizableTest {
}

Now, only the test classes annotated with @ParallelizableTest could be run in parallel, which means we can easily make test-by-test choices.

@ParallelizableTest
class MyParallelTest { // runs in parallel with other test classes
    // ...
}

In existing projects with a lot of test code inside, such a technique could be used to iteratively increase the number of parallel tests over time.

Choosing what to parallelize

Finally, with all the machinery available, we have to decide what to parallelize first. The answer could be found in a good-old test pyramid.

The unofficial test parallelization pyramid

It shouldn't be a surprise, that the unit tests are the easiest ones to parallelize first. Usually, the only thing required to do so would be annotating them with @ParallelizableTest, as the rest should still work. Despite being the least beneficial in terms of reducing the total execution time, the low effort makes parallelizing them almost free. In fact, doing so emphasizes their inherent isolation from other tests.


Note: there seems to be a lot of controversy around what a "unit test" really means. To avoid confusion, I rely on the definition from the "Unit Testing: Principles, Practices, and Patterns" book by Vladimir Khorikov here:

A unit test verifies a single unit of behavior, does it quickly and does it in isolation from other tests.


As the next step, you may want to select other test classes and parallelize them too. While those may result in significant execution time reduction, the required effort could increase as well. For example, re-using the same DB instance may require proper data randomization in all parallelized tests in order to prevent cross-interference. In some use cases, more sophisticated synchronization options of Junit 5 could be helpful too.

Especially for some non-trivial tests, the costs of parallelization may even outweigh the profits due to the complexity introduced. This is why I perceive selective test parallelization so beneficial.

Putting it all together

For the purpose of this article, I've created a small example project consisting of 6 test classes with 3 test cases each (named A, B, and C). Half of them can be run in parallel using the configuration described above. Each test case prints its thread name, class & case name on start and end. Running mvn clean verify seems to prove the correctness of the proposed setup:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.mikemybytes.junit.sequential.Sequential2Test
[INFO] Running com.mikemybytes.junit.parallel.Parallel2Test
[INFO] Running com.mikemybytes.junit.parallel.Parallel1Test
[INFO] Running com.mikemybytes.junit.parallel.Parallel3Test
[INFO] [stdout] [ForkJoinPool-1-worker-1] START: Sequential2Test#A
[INFO] [stdout] [ForkJoinPool-1-worker-3] START: Parallel2Test#C
[INFO] [stdout] [ForkJoinPool-1-worker-4] START: Parallel3Test#C
[INFO] [stdout] [ForkJoinPool-1-worker-2] START: Parallel1Test#C
[INFO] [stdout] [ForkJoinPool-1-worker-1]   END: Sequential2Test#A
[INFO] [stdout] [ForkJoinPool-1-worker-4]   END: Parallel3Test#C
[INFO] [stdout] [ForkJoinPool-1-worker-3]   END: Parallel2Test#C
[INFO] [stdout] [ForkJoinPool-1-worker-4] START: Parallel3Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-3] START: Parallel2Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-1] START: Sequential2Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-2]   END: Parallel1Test#C
[INFO] [stdout] [ForkJoinPool-1-worker-2] START: Parallel1Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-4]   END: Parallel3Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-4] START: Parallel3Test#A
[INFO] [stdout] [ForkJoinPool-1-worker-2]   END: Parallel1Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-2] START: Parallel1Test#A
[INFO] [stdout] [ForkJoinPool-1-worker-3]   END: Parallel2Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-3] START: Parallel2Test#A
[INFO] [stdout] [ForkJoinPool-1-worker-1]   END: Sequential2Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-1] START: Sequential2Test#C
[INFO] [stdout] [ForkJoinPool-1-worker-4]   END: Parallel3Test#A
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.601 s - in com.mikemybytes.junit.parallel.Parallel3Test
[INFO] [stdout] [ForkJoinPool-1-worker-3]   END: Parallel2Test#A
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.836 s - in com.mikemybytes.junit.parallel.Parallel2Test
[INFO] [stdout] [ForkJoinPool-1-worker-1]   END: Sequential2Test#C
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.852 s - in com.mikemybytes.junit.sequential.Sequential2Test
[INFO] Running com.mikemybytes.junit.sequential.Sequential3Test
[INFO] [stdout] [ForkJoinPool-1-worker-1] START: Sequential3Test#A
[INFO] [stdout] [ForkJoinPool-1-worker-2]   END: Parallel1Test#A
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.073 s - in com.mikemybytes.junit.parallel.Parallel1Test
[INFO] [stdout] [ForkJoinPool-1-worker-1]   END: Sequential3Test#A
[INFO] [stdout] [ForkJoinPool-1-worker-1] START: Sequential3Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-1]   END: Sequential3Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-1] START: Sequential3Test#C
[INFO] [stdout] [ForkJoinPool-1-worker-1]   END: Sequential3Test#C
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.517 s - in com.mikemybytes.junit.sequential.Sequential3Test
[INFO] Running com.mikemybytes.junit.sequential.Sequential1Test
[INFO] [stdout] [ForkJoinPool-1-worker-1] START: Sequential1Test#A
[INFO] [stdout] [ForkJoinPool-1-worker-1]   END: Sequential1Test#A
[INFO] [stdout] [ForkJoinPool-1-worker-1] START: Sequential1Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-1]   END: Sequential1Test#B
[INFO] [stdout] [ForkJoinPool-1-worker-1] START: Sequential1Test#C
[INFO] [stdout] [ForkJoinPool-1-worker-1]   END: Sequential1Test#C
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.017 s - in com.mikemybytes.junit.sequential.Sequential1Test
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0

The tests marked as parallelizable run in parallel with each other (as expected) but also in parallel with the sequential ones (in this case Sequential2Test). This could look a bit suspicious at first. However, as our parallelizable tests claim to be independent of the others, they should make no harm even to those running one by one.

Limitations

For the time of writing, the main limitation of the proposed approach is related to the accuracy of the Maven Surefire plugin reports being generated for tests execution. In the example project, there were 3 test classes being executed in parallel with 3 test cases each. This means we should expect 9 tests to be reported in total. However, the reports available at target/surefire-reports seem to indicate something different.

-------------------------------------------------------------------------------
Test set: com.mikemybytes.junit.parallel.Parallel1Test
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.088 s - in com.mikemybytes.junit.parallel.Parallel1Test
-------------------------------------------------------------------------------
Test set: com.mikemybytes.junit.parallel.Parallel2Test
-------------------------------------------------------------------------------
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.847 s - in com.mikemybytes.junit.parallel.Parallel2Test
-------------------------------------------------------------------------------
Test set: com.mikemybytes.junit.parallel.Parallel3Test
-------------------------------------------------------------------------------
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.604 s - in com.mikemybytes.junit.parallel.Parallel3Test

This is a known Surefire limitation (see SUREFIRE-1643 and SUREFIRE-1795 tickets) not related to JUnit 5 in particular - the reporting part supports only a sequence of test events.

That's why allowing on demand sequential execution as described before is so important. If anything goes wrong (or a certain tool relies on the generated report), we can still run the tests one-by-one with -DparallelTests=false.

Summary

Parallel test execution introduced in JUnit 5 is a simple but yet powerful tool for utilizing our hardware resources better in order to shorten the feedback loop. Running only selected tests in parallel gives us full control over test execution and the "speed vs effort" trade-offs. Thanks to that, we can gradually increase the parallelism of our tests instead of changing everything at once. Despite some limitations, parallel test execution is already a useful addition to our development toolbox.

Tags

Mike Kowalski

Software engineer & consultant believing in craftsmanship and the power of fresh espresso. Writing in & about Java, distributed systems, and beyond. Mikes his own opinions and bytes.