Reporting Code Coverage using Maven and JaCoCo plugin

1. OVERVIEW

Code coverage is a metric indicating which percentage of lines of code are executed when running automated tests. Unit and integration tests for instance.

It’s known that having automated tests as part of your build process improves the software quality and reduces the number of bugs.

Do you know if you need more unit tests? Or if your tests cover all possible branches of an if or switch statements? Or if your code coverage is decreasing over time? Especially after you join a team to work on an on-going project.

JaCoCo Code Coverage - jacoco-maven-plugin

Code coverage helps to answer these questions. This post covers reporting code coverage using Maven’s jacoco-maven-plugin, a library that adds minimal overhead with normal build.

Tip: Read on to learn how to upload JaCoCo code coverage reports to SonarQube.

2. REQUIREMENTS

Java 7+.
Maven 3.2+.
Overview of my previous post, Splitting Unit and Integration Tests using Maven and Surefire plugin because this post uses the same source code.

3. SAMPLE APPLICATION

The example application has two unit test classes, DemoControllerTest, DefaultSomeBusinessServiceTest, and two integration tests classes, DemoControllerIT, ApplicationTests; similar to those discussed in Splitting Unit and Integration Tests using Maven and Surefire plugin section.

4. CONFIGURING jacoco-maven-plugin AND COVERAGE THRESHOLD

Warning: According to the JaCoCo documentation, do not set forkCount to 0 or set forkMode to never as it would prevent executing the tests with the JaCoCo javaagent and no coverage would be recorded.

Let’s configure jacoco-maven-plugin in pom.xml:

...
<plugin>
  <groupId>org.jacoco</groupId>
  <artifactId>jacoco-maven-plugin</artifactId>
  <version>0.8.3</version>
  <executions>
    <execution>
      <id>coverage-initialize</id>
      <goals>
        <goal>prepare-agent</goal>
      </goals>
    </execution>
    <execution>
      <id>coverage-report</id>
      <phase>post-integration-test</phase>
      <goals>
        <goal>report</goal>
      </goals>
    </execution>
    <!-- Threshold -->
    <execution>
      <id>coverage-check</id>
      <goals>
        <goal>check</goal>
      </goals>
      <configuration>
        <rules>
          <rule>
            <element>CLASS</element>
            <excludes>
              <exclude>com.asimio.demo.Application</exclude>
            </excludes>
            <limits>
              <limit>
                <counter>LINE</counter>
                <value>COVEREDRATIO</value>
                <minimum>80%</minimum>
              </limit>
            </limits>
          </rule>
        </rules>
      </configuration>
    </execution>
  </executions>
</plugin>
...

The prepare-agent goal sets up the property argLine (for most packaging types) pointing to the JaCoCo runtime agent. You can also pass argLine as a VM argument. maven-surefire-plugin uses argLine to set the JVM options to run the tests.

If you are explicitly setting argLine, make sure it allows late replacements like:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <argLine>@{argLine} -more -arguments</argLine>
...
  </configuration>
</plugin>

so that maven-surefire-plugin picks up changes made by other Maven plugins such as jacoco-maven-plugin.

The JaCoCo Java agent will collect coverage information when maven-surefire-plugin runs the tests. It’ll write it to destFile property value if set, or target/jacoco.exec by default. Read more at https://www.eclemma.org/jacoco/trunk/doc/prepare-agent-mojo.html.

The report goal creates code coverage reports for tests in HTML, XML, CSV formats. This goal reads the dataFile property value if set, or target/jacoco.exec. And writes the resulting reports to outputDirectory property value or target/site/jacoco. Read more at https://www.eclemma.org/jacoco/trunk/doc/report-mojo.html.

Tip: Read on to learn how to upload JaCoCo code coverage reports to SonarQube.

The check goal validates the coverage rules (discussed later) are met. In case they are not, it interrupts and fails the build unless haltOnFailure property is set to false. Read more at https://www.eclemma.org/jacoco/trunk/doc/check-mojo.html.

5. RUNNING THE TESTS AND CREATING THE COVERAGE REPORTS

Let’s build the application and analyze the Maven command output:

mvn clean verify
...
[INFO] --- jacoco-maven-plugin:0.8.3:prepare-agent (coverage-initialize) @ unit-integration-tests-jacoco-coverage ---
[INFO] argLine set to -javaagent:/Users/ootero/.m2/repository/org/jacoco/org.jacoco.agent/0.8.3/org.jacoco.agent-0.8.3-runtime.jar=destfile=/Users/ootero/Projects/bitbucket.org/unit-integration-tests-jacoco-coverage/target/jacoco.exec
...
[INFO] --- maven-surefire-plugin:2.22.1:test (unit-tests) @ unit-integration-tests-jacoco-coverage ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
...
[INFO] --- maven-surefire-plugin:2.22.1:test (integration-tests) @ springboot2-split-unit-integration-tests ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
...
[INFO] --- jacoco-maven-plugin:0.8.3:report (coverage-report) @ unit-integration-tests-jacoco-coverage ---
[INFO] Loading execution data file /Users/ootero/Projects/bitbucket.org/unit-integration-tests-jacoco-coverage/target/jacoco.exec
[INFO] Analyzed bundle 'unit-integration-tests-jacoco-coverage' with 3 classes
...
[INFO] --- jacoco-maven-plugin:0.8.3:check (coverage-check) @ unit-integration-tests-jacoco-coverage ---
[INFO] Loading execution data file /Users/ootero/Projects/bitbucket.org/unit-integration-tests-jacoco-coverage/target/jacoco.exec
[INFO] Analyzed bundle 'unit-integration-tests-jacoco-coverage' with 3 classes
[INFO] All coverage checks have been met.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
...

Right after the clean phase completes, jacoco-maven-plugin’s prepare-agent goal (bound to the Maven’s Build Default Lifecycle’s initialize phase) sets the argLine property pointing to the JaCoCo Java agent.

Unit and Integration tests ran separately as covered in a previous post.

Next, not included in this log output, the Maven artifact is built and repackaged.

After that, jacoco-maven-plugin’s coverage-report goal (bound to the Maven’s Build Default Lifecycle’s post-integration-test phase) generates HTML, XML and CSV reports.

$ ls -1a target/site/jacoco/
.
..
com.asimio.demo
com.asimio.demo.rest
com.asimio.demo.service
index.html
jacoco.csv
jacoco.xml
jacoco-resources
jacoco-sessions.html

Opening the HTML report at target/site/jacoco/index.html results in:

JaCoCo Code Coverage - jacoco-maven-plugin Code coverage report for a successful build

Lastly, jacoco-maven-plugin’s check goal (bound to the Maven’s Build Default Lifecycle’s verify phase) checks the code coverage metrics are met.

6. JACOCO RULES

Let’s take a closer look at the jacoco-maven-plugin’s coverage-check rules configuration in pom.xml:

<rules>
  <rule>
    <element>CLASS</element>
    <excludes>
      <exclude>com.asimio.demo.Application</exclude>
    </excludes>
    <limits>
      <limit>
        <counter>LINE</counter>
        <value>COVEREDRATIO</value>
        <minimum>80%</minimum>
      </limit>
    </limits>
  </rule>
</rules>

Setting the rule element to CLASS means every Java class from the application would need to meet each counter limit for the build to pass.

In this example there is only one limit, a LINE counter that needs a coverage of at least 80%. I’ll cover JaCoCo Counters later.

Other JaCoCo rules you could used are:

BUNDLE	The set of counter limits would have to be met at the application as a whole
PACKAGE	The set of counter limits would have to be met for all packages (eg `com.asimio.demo`, `com.asimio.demo.rest`, etc.)
CLASS	The set of counter limits would have to be met for every Java class
SOURCEFILE
METHOD	The set of counter limits would have to be met for every class method

Notice that has you move down in the JaCoCo rules table, the check goal becomes more constraining.

As an example, if you remove com.asimio.demo.Application from the excludes sections, the build fails because the LINE counter doesn’t reach 80% for said class:

-    <excludes>
-      <exclude>com.asimio.demo.Application</exclude>
-    </excludes>

mvn clean verify
...
[WARNING] Rule violated for class com.asimio.demo.Application: lines covered ratio is 0.33, but expected minimum is 0.80
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
...

7. JACOCO COUNTERS

Even though I only used the LINE counter when covering JaCoCo Rules, there are a handful of other counters you could include in the limits set in the jacoco-maven-plugin configuration.

INSTRUCTION	The amount of code that can be executed or missed
BRANCH	The total number of branches (if and switch statements) in a method that can be executed or missed. No coverage: No branches in the line has been executed (red diamond) Partial coverage: Only a part of the branches in the line have been executed (yellow diamond) Full coverage: All branches in the line have been executed (green diamond)
CYCLOMATIC COMPLEXITY	https://en.wikipedia.org/wiki/Cyclomatic_complexity
LINE	Executed when at least one instruction that is assigned to this line has been executed. No coverage: No instruction in the line has been executed (red background) Partial coverage: Only a part of the instruction in the line have been executed (yellow background) Full coverage: All instructions in the line have been executed (green background)
METHOD	Executed when at least one instruction has been executed
CLASS	Executed when at least one of its methods has been executed

Notice that has you move down in the JaCoCo counters table, the check goal becomes less constraining.

Examples of associating a counter to a rule are:

<limit>
  <counter>LINE</counter>
  <value>COVEREDRATIO</value>
  <minimum>80%</minimum>
</limit>
<limit>
  <counter>CLASS</counter>
  <value>MISSEDCOUNT</value>
  <maximum>0</maximum>
</limit>

A counter value is one of:

TOTALCOUNT

COVEREDCOUNT

MISSEDCOUNT

COVEREDRATIO

MISSEDRATIO

and a numeric minimum or maximum.

8. CONCLUSION

Although not a silver bullet, code coverage helps to measure what percentage of code is executed when running the test suites. And thus, helping to reduce the number of bugs and improving the software release quality.

Keeping a certain threshold might get difficult over time as a development team adds edge cases or implement defensive programming.

JaCoCo adds minimal overhead to the build process. jacoco-maven-plugin’s prepare-agent goal, bound to the initialize phase, sets the agent responsible for instrumenting the Java code before maven-surefire-plugin runs. coverage-report goal is bound to the post-integration-test phase. And coverage-report goal is bound to the verify phase. Read more at Maven’s Build Default Lifecycle. This means, unlike other libraries, JaCoCo doesn’t need to run the tests twice.

Tip: Read on to learn how to upload JaCoCo code coverage reports to SonarQube.

Thanks for reading and as always, feedback is very much appreciated. If you found this post helpful and would like to receive updates when content like this gets published, sign up to the newsletter.