Ceedling enhancements: aborting test run if uncovered files are detected
Recently we've adopted Ceedling as our test automation framework for C. Ever since, we've been working on tailoring it for our needs. Below you will find details of the latest feature that we've added to it (pending PR merge).
The problem
One thing we noticed when using Ceedling was that when coding, if any of us added any new source file, and forgot to add a test suite for it (sometimes product of last-minute refactoring), the file without code coverage data would not affect the code coverage report in any way. This is pretty bad because it should at least count as 0% coverage.
Now, how did we solve this? We added an option to the project.yml file, that if enabled, any file is found that does not have code coverage data, the Ceedling command will fail so that our CI pipeline warns us pretty quick when we make this type of mistake.
However, there are some cases of files that just can't be tested with unit tests, or for whatever reason are not wanted to be tested, so we had to add an option to ignore certain source files as well.
Demo time
To show this functionality working, we're going to use the repo that we created in one of our previous blog posts:
Unit-testing C code with Ceedling and CMock
The repository with the code example can be found here:
https://github.com/zaleos/post-ceedling-demo.git
First, let's do some setting up:
# Clone the repository:
$ git clone --branch develop https://github.com/zaleos/post-ceedling-demo
$ cd post-ceedling-demo
# Build the docker image:
$ docker build --no-cache -t zaleos-ubuntu .
# Start a docker container with the generated image:
$ docker run --rm -ti -v $(pwd):/root/basic-ceedling-example zaleos-ubuntu
# Check that Ceedling is installed:
$ ceedling version
# (output should be similar to this):
Ceedling:: 0.30.0-zaleos
Unity:: 2.5.0
CMock:: 2.5.1
CException:: 1.3.2
(Note the -zaleos suffix in the version: at the moment, this feature is only available in our fork, until it is accepted upstream: see this pull request for details...)
To be able to run Ceedling with the gcov plugin, we will need to install it's dependency, gcovr, and also add -gcov to the enabled plugins in the project.yml file:
$ apt update && apt install python3-pip && pip3 install gcovr
$ vim project.yml
:plugins:
:load_paths:
- "#{Ceedling.load_path}"
:enabled:
- stdout_pretty_tests_report
- module_generator
- command_hooks
- gcov # *** NEW ***
Now, let's run Ceedling with the gcov plugin:
# First, we'll create the "external_lib" .so file:
$ cd external_libs/ && make && cd ..
# Generate html report:
$ ceedling gcov:all utils:gcov
# Check that it has been generated:
$ ls build/artifacts/gcov
GcovCoverageResults.html
If we open the generated HTML file from the last step, we should see something like this:
Now, on to show the new feature:
Lets edit the project.yml file, and add :abort_on_uncovered: true under the :gcov: key:
$ vim project.yml
...snip...
:gcov:
:abort_on_uncovered: true # *** NEW ***
:html_report: true # Enables the output of an HTML report for utils:gcov
...snip...
At this point, if we were to run the ceedling+gcov command again then, we would see no change, because currently there aren't any source code files without test coverage. Let's add a new source file, and not add any test cases that cover it:
# Create an empty source code file with no coverage:
$ touch src/uncovered.c
# Re-run the Ceedling command:
$ ceedling gcov:all utils:gcov
...snip...
Could not find coverage results for src/uncovered.c
There were files with no coverage results: aborting.
# Check exit status:
$ echo $?
255
As you can see, the ceedling command gave us an error message, and the command failed with a non-zero exit status. This comes in very handy for CI to detect these types of mistakes.
The other configuration option we added was an ignore list for the previous setting, to not trigger a failure. Let's see this in action:
$ vim project.yml
...snip...
:gcov:
:abort_on_uncovered: true
:uncovered_ignore_list: # *** NEW ***
- src/uncovered.c # *** NEW ***
...snip...
Now, let's run the command again:
$ ceedling gcov:all utils:gcov
Test 'test_a.c'
---------------
Running Hook pre_test_fixture_execute...
...snip...
vector_helper.c Lines executed:77.78% of 9
vector_helper.c Branches executed:100.00% of 4
vector_helper.c Taken at least once:50.00% of 4
vector_helper.c Calls executed:100.00% of 1
# Check exit status:
$ echo $?
0
As you can see, this time the file without coverage data is ignored and does not generate an error (the exit status is 0). Please note that it is preferable to add a test case that tests the code in these files than to use this: it should only be used as a last resort. Also, note that the ignore list is only for files that do not have coverage data: if a file is in the list, but also does have associated coverage data, then the coverage will be added to the report as normal, and this ignore list entry will have no effect.
Conclusions
Although this new feature we have contributed to Ceedling may be a small one, hopefully, it will come in useful to other developers, and we look forward to contributing more changes in the future.