Better magik coverage using sw5-jacoco-reporter
August 29, 2021 code coverage
Recently I wrote about Smallworld/Magik Code Coverage using Jacoco. This worked ok’ish, although it looks a bit weird with manged exemplar/method names. Furthermore, it only gave a percentage of the code that was run, seeing which lines were actually ‘hit’ was impossible. The sw5-jacoco-reporter is a report generator which is ended towards Smallworld 5 generating a better report with individual line coverage and proper method names.
JaCoCo is a tool to determine which code was run, during your unit tests for example. Although JaCoCo targets Java (hence the name Java Code Coverage Library,) it actually can be used for other languages which run on the JVM as well. The JaCoCo API is readily available with examples to use and extend JaCoCo to your own needs. As such, JaCoCo makes it easy to build your own report generator.
Given the way Smallworld 5 generates the Java bytecode, it is already fairly similar to the resulting Java bytecode generated by Java. This makes it easy for JaCoCo to already generate a report, albeit with some unresolved details such as original method names.
To generate a better coverage report, JaCoCo needs to be provided some additional information such as the source files. Furthermore, a transformation has to be done to use the original method names. Basically, this is what sw5-jacoco-reporter
does.
The result of using sw5-jacoco-reporter
is a report which provides the original method names and branch- and line-coverage. See the end of this article for examples.
Preliminary
Before starting, download the latest JaCoCo release from the JaCoCo website. Furthermore, you’ll need a running Smallworld 5 installation. This guide uses Smallworld 5.2.5 and JaCoCo 0.8.6.
Download the sw5-jacoco-reporter binary, or find the latest release at the releases page in the Github repository.
Steps to determine code coverage
To determine the code coverage in Smallworld 5 we need to do several steps:
- Compile the magik sources into jars
- Load the jars and instrument on the fly
- Run the tests
- Generate the report using
sw5-jacoco-reporter
Note that these steps are mostly the same as the original article, except for step 4 which uses the new reporter. Steps 1, 2, and 3 are included here for clarity. For more information see the article Smallworld/Magik Code Coverage using Jacoco.
1. Compile the magik sources into jars
Step 1 includes loading the product and calling the method compile_all_modules()
on it. This saves the compiled magik to (a) jar(s) under the libs
directory.
Magik> smallworld_product.add_product(".../munit") # For module dependencies.
...
Magik> prd << smallworld_product.add_product(".../sw_xsd_loader") # Load the actual product under test.
...
Magik> prd.compile_all_modules() # Compile modules, save the libs/jars.
...
Magik> quit()
The quit is required as we need to load the libraries from disk, or JaCoCo won’t be able to instrument them.
2. Load the jars and instrument on the fly
Once the jars are saved to the libs
directory, these can be loaded: Step 2. Before we do this, we need to start our Smallworld session with a JaCoCO java-agent. Start your session with the additional parameter to runalias
: -j -javaagent:.../jacoco-0.8.6/lib/jacocoagent.jar
.
Another option is to edit your environment
file and add the following, before starting your session:
SET SW_LAUNCH_JAVA_ARGS=%SW_LAUNCH_JAVA_ARGS% -javaagent:...\jacoco-0.8.6\lib\jacocoagent.jar
Your session will start normally, but a file jacoco.exec
will be created. Then, in this session load your product again, load the module containing your test(s) and run the test:
Magik> smallworld_product.add_product(".../munit") # For module dependencies.
...
Magik> smallworld_product.add_product(".../sw_xsd_loader") # Load the actual product under test.
...
3. Run the tests
Step 3 is running the tests themselves. First we load the tests-module, then we execute one test suite:
Magik> sw_module_manager.load_module(:xsd_loader_tests) # Load the tests for the product.
...
Magik> test_runner.run_in_foreground(char16_vector_test.suite()) # Run a single test suite.
...
Magik> quit()
JaCoCo now has recorded the code that lines of Magik code that were executed and stored in the file jacoco.exec
.
4. Generate the report using sw5-jacoco-reporter
Now that we have the file jacoco.exec
, we can generate a report from it. You can do this using the jacococli.jar
file, like so:
java -jar .../sw5-jacoco-reporter-1.0.0.jar --jacoco-file .../jacoco.exec --product-dir .../sw_xsd_loader --html .../coverage-report-sw5
This will create a new directory called coverage-report-sw5
. In this directory you’ll find a index.html
file which you can open with your browser. Navigate to the source file called magik.sw_xsd_loader.xsd_loader.char16_vector_31
and you’ll see the following:
Clicking on the method will show the individual branch- and line-coverage:
Conclusion
Earlier JaCoCo already provided some useful information about the code coverage our unit tests. Now, with the sw5-jacoco-reporter
, better insights are provided given the individual branch- and line-coverage and proper method names.