August 29, 2021 code coverage
Recently I wrote about Smallworld/Magik Code Coverage using Jacoco. This worked ok’ish, although it looks a bit weird with manged exemplar/method names. Furthermore, it only gave a percentage of the code that was run, seeing which lines were actually ‘hit’ was impossible. The sw5-jacoco-reporter is a report generator which is ended towards Smallworld 5 generating a better report with individual line coverage and proper method names.
JaCoCo is a tool to determine which code was run, during your unit tests for example. Although JaCoCo targets Java (hence the name Java Code Coverage Library,) it actually can be used for other languages which run on the JVM as well. The JaCoCo API is readily available with examples to use and extend JaCoCo to your own needs. As such, JaCoCo makes it easy to build your own report generator.
Given the way Smallworld 5 generates the Java bytecode, it is already fairly similar to the resulting Java bytecode generated by Java. This makes it easy for JaCoCo to already generate a report, albeit with some unresolved details such as original method names.
To generate a better coverage report, JaCoCo needs to be provided some additional information such as the source files. Furthermore, a transformation has to be done to use the original method names. Basically, this is what
The result of using
sw5-jacoco-reporter is a report which provides the original method names and branch- and line-coverage. See the end of this article for examples.
Before starting, download the latest JaCoCo release from the JaCoCo website. Furthermore, you’ll need a running Smallworld 5 installation. This guide uses Smallworld 5.2.5 and JaCoCo 0.8.6.
Steps to determine code coverage
To determine the code coverage in Smallworld 5 we need to do several steps:
- Compile the magik sources into jars
- Load the jars and instrument on the fly
- Run the tests
- Generate the report using
Note that these steps are mostly the same as the original article, except for step 4 which uses the new reporter. Steps 1, 2, and 3 are included here for clarity. For more information see the article Smallworld/Magik Code Coverage using Jacoco.
1. Compile the magik sources into jars
Step 1 includes loading the product and calling the method
compile_all_modules() on it. This saves the compiled magik to (a) jar(s) under the
Magik> smallworld_product.add_product(".../munit") # For module dependencies. ... Magik> prd << smallworld_product.add_product(".../sw_xsd_loader") # Load the actual product under test. ... Magik> prd.compile_all_modules() # Compile modules, save the libs/jars. ... Magik> quit()
The quit is required as we need to load the libraries from disk, or JaCoCo won’t be able to instrument them.
2. Load the jars and instrument on the fly
Once the jars are saved to the
libs directory, these can be loaded: Step 2. Before we do this, we need to start our Smallworld session with a JaCoCO java-agent. Start your session with the additional parameter to
Another option is to edit your
environment file and add the following, before starting your session:
SET SW_LAUNCH_JAVA_ARGS=%SW_LAUNCH_JAVA_ARGS% -javaagent:...\jacoco-0.8.6\lib\jacocoagent.jar
Your session will start normally, but a file
jacoco.exec will be created. Then, in this session load your product again, load the module containing your test(s) and run the test:
Magik> smallworld_product.add_product(".../munit") # For module dependencies. ... Magik> smallworld_product.add_product(".../sw_xsd_loader") # Load the actual product under test. ...
3. Run the tests
Step 3 is running the tests themselves. First we load the tests-module, then we execute one test suite:
Magik> sw_module_manager.load_module(:xsd_loader_tests) # Load the tests for the product. ... Magik> test_runner.run_in_foreground(char16_vector_test.suite()) # Run a single test suite. ... Magik> quit()
JaCoCo now has recorded the code that lines of Magik code that were executed and stored in the file
4. Generate the report using
Now that we have the file
jacoco.exec, we can generate a report from it. You can do this using the
jacococli.jar file, like so:
java -jar .../sw5-jacoco-reporter-1.0.0.jar --jacoco-file .../jacoco.exec --product-dir .../sw_xsd_loader --html .../coverage-report-sw5
This will create a new directory called
coverage-report-sw5. In this directory you’ll find a
index.html file which you can open with your browser. Navigate to the source file called
magik.sw_xsd_loader.xsd_loader.char16_vector_31 and you’ll see the following:
Earlier JaCoCo already provided some useful information about the code coverage our unit tests. Now, with the
sw5-jacoco-reporter, better insights are provided given the individual branch- and line-coverage and proper method names.