Adding Tests¶

BLT has a built-in copy of the GoogleTest framework (gtest) for C and C++ unit tests and the Fortran Unit Test Framework (FRUIT) for Fortran unit tests.

Each GoogleTest or FRUIT file may contain multiple tests and is compiled into its own executable that can be run directly or as a make target.

In this section, we give a brief overview of GoogleTest and discuss how to add unit tests using the blt_add_test() macro.

Configuring Tests within BLT¶

Unit testing in BLT is controlled by the ENABLE_TESTS cmake option and is on by default.

For additional configuration granularity, BLT provides configuration options for the individual built-in unit testing libraries. The following additional options are available when ENABLE_TESTS is on:

ENABLE_GTEST: Option to enable gtest (default: ON).
ENABLE_GMOCK: Option to control gmock (default: OFF). Since gmock requires gtest, gtest is also enabled whenever ENABLE_GMOCK is true, regardless of the value of ENABLE_GTEST.
ENABLE_FRUIT: Option to control FRUIT (Default ON). It is only active when ENABLE_FORTRAN is enabled.

GoogleTest (C/C++ Tests)¶

The contents of a typical GoogleTest file look like this:

#include "gtest/gtest.h"

#include ...    // include headers needed to compile tests in file

// ...

TEST(<test_case_name>, <test_name_1>)
{
   // Test 1 code here...
   // ASSERT_EQ(...);
}

TEST(<test_case_name>, <test_name_2>)
{
   // Test 2 code here...
   // EXPECT_TRUE(...);
}

Each unit test is defined by the GoogleTest TEST() macro which accepts a test case name identifier, such as the name of the C++ class being tested, and a test name, which indicates the functionality being verified by the test. Within a test, failure of logical assertions (macros prefixed by ASSERT_) will cause the test to fail immediately, while failures of expected values (macros prefixed by EXPECT_) will cause the test to fail, but will continue running code within the test.

Note that the GoogleTest framework will generate a main() routine for each test file if it is not explicitly provided. However, sometimes it is necessary to provide a main() routine that contains operation to run before or after the unit tests in a file; e.g., initialization code or pre-/post-processing operations. A main() routine provided in a test file should be placed at the end of the file in which it resides.

Note that GoogleTest is initialized before MPI_Init() is called.

Other GoogleTest features, such as fixtures and mock objects (gmock) may be used as well.

See the GoogleTest Primer for a discussion of GoogleTest concepts, how to use them, and a listing of available assertion macros, etc.

FRUIT (Fortran Tests)¶

Fortran unit tests using the FRUIT framework are similar in structure to the GoogleTest tests for C and C++ described above.

The contents of a typical FRUIT test file look like this:

module <test_case_name>
  use iso_c_binding
  use fruit
  use <your_code_module_name>
  implicit none

contains

subroutine test_name_1
!  Test 1 code here...
!  call assert_equals(...)
end subroutine test_name_1

subroutine test_name_2
!  Test 2 code here...
!  call assert_true(...)
end subroutine test_name_2

The tests in a FRUIT test file are placed in a Fortran module named for the test case name, such as the name of the C++ class whose Fortran interface is being tested. Each unit test is in its own Fortran subroutine named for the test name, which indicates the functionality being verified by the unit test. Within each unit test, logical assertions are defined using FRUIT methods. Failure of expected values will cause the test to fail, but other tests will continue to run.

Note that each FRUIT test file defines an executable Fortran program. The program is defined at the end of the test file and is organized as follows:

program fortran_test
  use fruit
  use <your_component_unit_name>
  implicit none
  logical ok

  ! initialize fruit
  call init_fruit

  ! run tests
  call test_name_1
  call test_name_2

  ! compile summary and finalize fruit
  call fruit_summary
  call fruit_finalize

  call is_all_successful(ok)
  if (.not. ok) then
    call exit(1)
  endif
end program fortran_test

Please refer to the FRUIT documentation for more information.

Adding a BLT unit test¶

After writing a unit test, we add it to the project’s build system by first generating an executable for the test using the blt_add_executable() macro. We then register the test using the blt_add_test() macro.

blt_add_test

This macro generates a named unit test from an existing executable and allows users to pass in command line arguments.

Returning to our running example (see Creating Targets), let’s add a simple test for the calc_pi library, which has a single function with signature:

double calc_pi(int num_intervals);

We add a simple unit test that invokes calc_pi() and compares the result to $\pi$ , within a given tolerance (1e-6). Here is the test code:

#include <gtest/gtest.h>

#include "calc_pi.hpp"

TEST(calc_pi, serial_example)
{
    double PI_REF = 3.141592653589793238462643;
    ASSERT_NEAR(calc_pi(1000),PI_REF,1e-6);
}

To add this test to the build system, we first generate a test executable:

    blt_add_executable( NAME       test_1
                        SOURCES    test_1.cpp 
                        DEPENDS_ON calc_pi gtest)

Note that this test executable depends on two targets: calc_pi and gtest.

We then register this executable as a test:

    blt_add_test( NAME    test_1 
                COMMAND test_1)

Note

The COMMAND parameter can be used to pass arguments into a test executable.

Note

The NAME of the test can be different from the test executable, which is passed in through the COMMAND parameter.

Running Tests and Examples¶

To run the tests, type the following command in the build directory:

$ make test

This will run all tests through cmake’s ctest tool and report a summary of passes and failures. Detailed output on individual tests is suppressed.

If a test fails, you can invoke its executable directly to see the detailed output of which checks passed or failed. This is especially useful when you are modifying or adding code and need to understand how unit test details are working, for example.

Note

You can pass arguments to ctest via the ARGS parameter, e.g. make test ARGS="..." Useful arguments include:

-R Regular expression filtering of tests. E.g. -R foo only runs tests whose names contain foo -j Run tests in parallel, E.g. -j 16 will run tests using 16 processors -VV (Very verbose) Dump test output to stdout

Converting CTest XML to JUnit¶

It is often useful to convert CTest’s XML output to JUnit for various reporting tools such as CI. This is a two step process.

First run your test suite with one of the following commands to output with CTest’s XML and to turn off compressed output:

make CTEST_OUTPUT_ON_FAILURE=1 test ARGS="--no-compress-output -T Test -VV -j8"
ctest -DCTEST_OUTPUT_ON_FAILURE=1 --no-compress-output -T Test -VV -j8

Then convert the CTest XML file to JUnit’s format with the XSL file included in BLT. This can be done in many ways, but most Linux or Unix machines come with a program called xsltproc

cd build-dir
xsltproc -o junit.xml path/to/blt/tests/ctest-to-junit.xsl Testing/*/Test.xml

Then point the reporting tool to the outputted junit.xml file.