Corrade::TestSuite::Tester class

Basic testing workflow

A test starts with deriving the Tester class. The test cases are parameter-less void member functions that are added using addTests() in the constructor and the main() function is created using CORRADE_TEST_MAIN(). The goal is to have as little boilerplate as possible, thus the test usually consists of only one *.cpp file with no header and the derived type is a struct to avoid having to write public keywords.

struct MyTest: TestSuite::Tester {
    explicit MyTest();

    void addTwo();
    void subtractThree();
};

MyTest::MyTest() {
    addTests({&MyTest::addTwo,
              &MyTest::subtractThree});
}

void MyTest::addTwo() {
    int a = 3;
    CORRADE_COMPARE(a + 2, 5);
}

void MyTest::subtractThree() {
    int b = 5;
    CORRADE_COMPARE(b - 3, 2);
}

CORRADE_TEST_MAIN(MyTest)

The above gives the following output:

Starting MyTest with 2 test cases...
    OK [1] addTwo()
    OK [2] subtractThree()
Finished MyTest with 0 errors out of 2 checks.

Actual testing is done via various CORRADE_VERIFY(), CORRADE_COMPARE(), CORRADE_COMPARE_AS() and CORRADE_COMPARE_WITH() macros. If some comparison in given test case fails, a FAIL with concrete file, line and additional diagnostic is printed to the output and the test case is exited without executing the remaining statements. Otherwise, if all comparisons in given test case pass, an OK is printed. The main difference between these macros is the kind of diagnostic output they print when comparison fails — for example a simple expression failure reported by CORRADE_VERIFY() is enough when checking for non- nullptr value, but for comparing a 1000-element array you might want to use CORRADE_COMPARE_AS() with Compare::Container instead.

Additionally there are CORRADE_SKIP(), CORRADE_EXPECT_FAIL() and CORRADE_EXPECT_FAIL_IF() control flow helpers that allow you to say for example that a particular test was skipped due to missing functionality on given platform (printing a SKIP on output and exiting the test case right after the statement) or documenting that some algorithm produces incorrect result due to a bug, printing an XFAIL. Passing a test while failure is expected is treated as an error (XPASS), which can be helpful to ensure the assumptions in the tests don't get stale. Expected failures can be also disabled globally via a command-line option --no-xfail or via environment variable, see below.

The only reason why those are macros and not member functions is the ability to gather class/function/file/line/expression information via the preprocessor for printing the test output and exact location of possible test failure. If none of these macros is encountered when running the test case, the test case is reported as invalid, with ? on output.

The test cases are numbered on the output and those numbers can be used on the command-line to whitelist/blacklist the test cases with --only/--skip, randomly reorder them using --shuffle and more, see below for details. In total, when all test cases pass, the executable exits with 0 return code, in case of failure or invalid test case it exits with 1 to make it possible to run the tests in batch (such as CMake CTest). By default, after a failure, the testing continues with the other test cases, you can abort after first failure using --abort-on-fail command-line option.

Useful but not immediately obvious is the possibility to use templated member functions as test cases, for example when testing a certain algorithm on different data types:

struct PiTest: TestSuite::Tester {
    explicit PiTest();

    template<class T> void calculate();
};

PiTest::PiTest() {
    addTests<PiTest>({
        &PiTest::calculate<float>,
        &PiTest::calculate<double>});
}

template<class T> void PiTest::calculate() {
    setTestCaseName(std::is_same<T, float>::value ? "calculate<float>" : "calculate<double>");

    CORRADE_COMPARE(calculatePi<T>(), T(3.141592653589793));
}

CORRADE_TEST_MAIN(PiTest)

And the corresponding output:

Starting PiTest with 2 test cases...
    OK [1] calculate<float>()
    OK [2] calculate<double>()
Finished PiTest with 0 errors out of 2 checks.

This works with all add*() functions though please note that current C++11 compilers (at least GCC and Clang) are not able to properly detect class type when passing only templated functions to it, so you may need to specify the type explicitly. Also, there is no easy and portable way to get function name with template parameters so by default it will be just the function name, but you can call setTestCaseName() to specify a full name.

Instanced tests

Often you have an algorithm which you need to test on a variety of inputs or corner cases. One solution is to use a for-cycle inside the test case to test on all inputs, but then the diagnostic on error will not report which input is to blame. Another solution is to duplicate the test case multiple times for all the different inputs, but that becomes a maintenance nightmare pretty quickly. Making the function with a non-type template parameter is also a solution, but that's not possible to do for all types and with huge input sizes it is often not worth the increased compilation times. Fortunately, there is an addInstancedTests() that comes for the rescue:

struct RoundTest: TestSuite::Tester {
    explicit RoundTest();

    void test();
};

namespace {
    enum: std::size_t { RoundDataCount = 5 };

    constexpr const struct {
        const char* name;
        float input;
        float expected;
    } RoundData[RoundDataCount] {
        {"positive down", 3.3f, 3.0f},
        {"positive up", 3.5f, 4.0f},
        {"zero", 0.0f, 0.0f},
        {"negative down", -3.5f, -4.0f},
        {"negative up", -3.3f, -3.0f}
    };
}

RoundTest::RoundTest() {
    addInstancedTests({&RoundTest::test}, RoundDataCount);
}

void RoundTest::test() {
    setTestCaseDescription(RoundData[testCaseInstanceId()].name);

    CORRADE_COMPARE(round(RoundData[testCaseInstanceId()].input),
        RoundData[testCaseInstanceId()].expected);
}

CORRADE_TEST_MAIN(RoundTest)

Corresponding output:

Starting RoundTest with 5 test cases...
    OK [1] test(positive down)
    OK [2] test(positive up)
    OK [3] test(zero)
    OK [4] test(negative down)
    OK [5] test(negative up)
Finished RoundTest with 0 errors out of 5 checks.

The tester class just gives you an instance index via testCaseInstanceId() and it's up to you whether you use it as an offset to some data array or generate an input using it, the above example is just a hint how one might use it. Each instance is printed to the output separately and if one instance fails, it doesn't stop the other instances from being executed. Similarly to the templated tests, setTestCaseDescription() allows you to set a human-readable description of given instance. If not called, the instances are just numbered in the output.

Repeated tests

A complementary feature to instanced tests are repeated tests using addRepeatedTests(), useful for example to repeatedly call one function 10000 times to increase probability of potential race conditions. The difference from instanced tests is that all repeats are treated as executing the same code and thus only the overall result is reported in the output. Also unlike instanced tests, if a particular repeat fails, no further repeats are executed. The test output contains number of executed repeats after the test case name, prefixed by @. Example of testing race conditions with multiple threads accessing the same variable:

struct RaceTest: TestSuite::Tester {
    explicit RaceTest();

    template<class T> void threadedIncrement();
};

RaceTest::RaceTest() {
    addRepeatedTests<RaceTest>({
        &RaceTest::threadedIncrement<int>,
        &RaceTest::threadedIncrement<std::atomic_int>}, 10000);
}

template<class T> void RaceTest::threadedIncrement() {
    setTestCaseName(std::is_same<T, int>::value ?
        "threadedIncrement<int>" : "threadedIncrement<std::atomic_int>");

    T x{0};
    int y = 1;
    auto fun = [&x, &y] {
        for(std::size_t i = 0; i != 500; ++i) x += y;
    };
    std::thread a{fun}, b{fun}, c{fun};

    a.join();
    b.join();
    c.join();

    CORRADE_COMPARE(x, 1500);
}

CORRADE_TEST_MAIN(RaceTest)

Depending on various factors, here is one possible output:

Starting RaceTest with 2 test cases...
  FAIL [1] threadedIncrement<int>()@167 at …/RaceTest.cpp on line 60
        Values x and 1500 are not the same, actual is
        1000
        but expected
        1500
    OK [2] threadedIncrement<std::atomic_int>()@10000
Finished RaceTest with 1 errors out of 10167 checks.

Similarly to testCaseInstanceId() there is testCaseRepeatId() which gives repeat index. Use with care, however, as the repeated tests are assumed to execute the same code every time. On the command-line it is possible to increase repeat count via --repeat-every. In addition there is --repeat-all which behaves as like all add*() functions in the constructor were called multiple times in a loop. Combined with --shuffle this can be used to run the test cases multiple times in a random order to uncover potential unwanted interactions and order-dependent bugs.

It's also possible to combine instanced and repeated tests using addRepeatedInstancedTests().

Benchmarks

Besides verifying code correctness, it's possible to measure code performance. Unlike correctness tests, the benchmark results are hard to reason about using only automated means, so there are no macros for verifying benchmark results and instead the measured values are just printed to the output for users to see. Benchmarks can be added using addBenchmarks(), the actual benchmark loop is marked by CORRADE_BENCHMARK() and the results are printed to output with BENCH identifier. Example benchmark comparing performance of inverse square root implementations:

struct InvSqrtBenchmark: TestSuite::Tester {
    explicit InvSqrtBenchmark();

    void naive();
    void fast();
};

InvSqrtBenchmark::InvSqrtBenchmark() {
    for(auto fn: {&InvSqrtBenchmark::naive, &InvSqrtBenchmark::fast}) {
        addBenchmarks({fn}, 500, BenchmarkType::WallTime);
        addBenchmarks({fn}, 500, BenchmarkType::CpuTime);
    }
}

void InvSqrtBenchmark::naive() {
    volatile float a; /* to avoid optimizers removing the benchmark code */
    CORRADE_BENCHMARK(1000000)
        a = 1.0f/std::sqrt(float(testCaseRepeatId()));
    CORRADE_VERIFY(a);
}

void InvSqrtBenchmark::fast() {
    volatile float a; /* to avoid optimizers removing the benchmark code */
    CORRADE_BENCHMARK(1000000)
        a = fastinvsqrt(float(testCaseRepeatId()));
    CORRADE_VERIFY(a);
}

CORRADE_TEST_MAIN(InvSqrtBenchmark)

Note that it's not an error to add one test/benchmark multiple times — here it is used to have the same code benchmarked with different timers. Possible output:

Starting InvSqrtBenchmark with 4 test cases...
 BENCH [1]   8.24 ± 0.19   ns naive()@499x1000000 (wall time)
 BENCH [2]   8.27 ± 0.19   ns naive()@499x1000000 (CPU time)
 BENCH [3]   0.31 ± 0.01   ns fast()@499x1000000 (wall time)
 BENCH [4]   0.31 ± 0.01   ns fast()@499x1000000 (CPU time)
Finished InvSqrtBenchmark with 0 errors out of 0 checks.

The number passed to addBenchmarks() is equivalent to repeat count passed to addRepeatedTests() and specifies measurement sample count. The number passed to CORRADE_BENCHMARK() is number of iterations of the inner loop in one sample measurement to amortize the overhead and error caused by clock precision — the faster the measured code is the more iterations it needs. The measured value is then divided by that number to represent cost of a single iteration. The testCaseRepeatId() returns current sample index and can be used to give some input variation to the test. By default the benchmarks measure wall clock time, see BenchmarkType for other types of builtin benchmarks. The default benchmark type can be also overriden on the command-line via --benchmark.

It's possible to use all CORRADE_VERIFY(), CORRADE_COMPARE() etc. verification macros inside the benchmark to check pre/post-conditions. If one of them fails, the benchmark is treated on output just like failing test, with no benchmark results being printed out. Keep in mind, however, that those macros have some overhead, so try to not use them inside the benchmark loop.

The benchmark output is calculated from all samples except the first discarded samples. By default that's one sample, --benchmark-discard and --repeat-every command-line options can be used to override how many samples are taken and how many of them are discarded at first. In the output, the used sample count and sample size is printed after test case name, prefixed with @. The output contains mean value and a sample standard deviation, calculated as:

Different benchmark type have different units, time values are displayed in ns, µs, ms and s, dimensionless count is suffixed by k, M or G indicating thousands, millions and billions, instructions with I, kI, MI and GI, cycles with C, kC, MC and GC and memory in B, kB, MB and GB. In case of memory the prefixes are multiples of 1024 instead of 1000. For easier visual recognition of the values, by default the sample standard deviation is colored yellow if it is larger than 5% of the absolute value of the mean and red if it is larger than 25% of the absolute value of the mean. This can be overriden on the command-line via --benchmark-yellow and --benchmark-red.

It's possible to have instanced benchmarks as well, see addInstancedBenchmarks().

Custom benchmarks

It's possible to specify a custom pair of functions for intiating the benchmark and returning the result using addCustomBenchmarks(). The benchmark end function returns an unsigned 64-bit integer indicating measured amount in units given by BenchmarkUnits. To further describe the value being measured you can call setBenchmarkName() in the benchmark begin function. Contrived example of benchmarking number of copies when using std::vector::push_back():

struct VectorBenchmark: TestSuite::Tester {
    explicit VectorBenchmark();

    void insert();

    void copyCountBegin();
    std::uint64_t copyCountEnd();
};

namespace {
    std::uint64_t count = 0;

    struct CopyCounter {
        CopyCounter() = default;
        CopyCounter(const CopyCounter&) {
            ++count;
        }
    };

    enum: std::size_t { InsertDataCount = 3 };

    constexpr const struct {
        const char* name;
        std::size_t count;
    } InsertData[InsertDataCount]{
        {"100", 100},
        {"1k", 1000},
        {"10k", 10000}
    };
}

VectorBenchmark::VectorBenchmark() {
    addCustomInstancedBenchmarks({&VectorBenchmark::insert}, 1, InsertDataCount,
        &VectorBenchmark::copyCountBegin,
        &VectorBenchmark::copyCountEnd,
        BenchmarkUnits::Count);
}

void VectorBenchmark::insert() {
    setTestCaseDescription(InsertData[testCaseInstanceId()].name);

    std::vector<CopyCounter> data;
    CORRADE_BENCHMARK(1)
        for(std::size_t i = 0, end = InsertData[testCaseInstanceId()].count; i != end; ++i)
            data.push_back({});
}

void VectorBenchmark::copyCountBegin() {
    setBenchmarkName("copy count");
    count = 0;
}

std::uint64_t VectorBenchmark::copyCountEnd() {
    return count;
}

CORRADE_TEST_MAIN(VectorBenchmark)

Running the benchmark shows that calling push_back() for 10 thousand elements actually causes the copy constructor to be called 26 thousand times:

Starting VectorBenchmark with 3 test cases...
 BENCH [1] 227.00             insert(100)@1x1 (copy count)
 BENCH [2]   2.02          k  insert(1k)@1x1 (copy count)
 BENCH [3]  26.38          k  insert(10k)@1x1 (copy count)
Finished VectorBenchmark with 0 errors out of 0 checks.

Specifying setup/teardown routines

While the common practice in C++ is to use RAII for resource lifetime management, sometimes you may need to execute arbitrary code at the beginning and end of each test case. For this, all addTests(), addInstancedTests(), addRepeatedTests(), addRepeatedInstancedTests(), addBenchmarks(), addInstancedBenchmarks(), addCustomBenchmarks() and addCustomInstancedBenchmarks() have an overload that is additionally taking a pair of parameter-less void functions for setup and teardown. Both functions are called before and after each test case run, independently on whether the test case passed or failed.

Command-line options

Command-line options that make sense to be set globally for multiple test cases are also configurable via environment variables for greater flexibility when for example running the tests in a batch via ctest.

Usage:

./my-test [-h|--help] [-c|--color on|off|auto] [--skip "N1 N2..."]
    [--skip-tests] [--skip-benchmarks] [--only "N1 N2..."] [--shuffle]
    [--repeat-every N] [--repeat-all N] [--abort-on-fail] [--no-xfail]
    [--benchmark TYPE] [--benchmark-discard N] [--benchmark-yellow N]
    [--benchmark-red N]

Arguments:

• -h, --help — display this help message and exit
• -c, --color on|off|auto — colored output (environment: CORRADE_TEST_COLOR, default: auto). The auto option enables color output in case an interactive terminal is detected. Note that on Windows it is possible to output colors only directly to an interactive terminal unless CORRADE_UTILITY_USE_ANSI_COLORS is defined.
• --skip "N1 N2..." — skip test cases with given numbers
• --skip-tests — skip all tests (environment: CORRADE_TEST_SKIP_TESTS=ON|OFF)
• --skip-benchmarks — skip all benchmarks (environment: CORRADE_TEST_SKIP_BENCHMARKS=ON|OFF)
• --only "N1 N2..." — run only test cases with given numbers
• --shuffle — randomly shuffle test case order (environment: CORRADE_TEST_SHUFFLE=ON|OFF)
• --repeat-every N — repeat every test case N times (environment: CORRADE_TEST_REPEAT_EVERY, default: 1)
• --repeat-all N — repeat all test cases N times (environment: CORRADE_TEST_REPEAT_ALL, default: 1)
• --abort-on-fail — abort after first failure (environment: CORRADE_TEST_ABORT_ON_FAIL=ON|OFF)
• --no-xfail — disallow expected failures (environment: CORRADE_TEST_NO_XFAIL=ON|OFF)
• --benchmark TYPE — default benchmark type (environment: CORRADE_BENCHMARK). Supported benchmark types:
  • wall-time — wall time spent
  • cpu-time — CPU time spent
  • cpu-cycles — CPU cycles spent (x86 only, gives zero result elsewhere)
• --benchmark-discard N — discard first N measurements of each benchmark (environment: CORRADE_BENCHMARK_DISCARD, default: 1)
• --benchmark-yellow N — deviation threshold for marking benchmark yellow (environment: CORRADE_BENCHMARK_YELLOW, default: 0.05)
• --benchmark-red N — deviation threshold for marking benchmark red (environment: CORRADE_BENCHMARK_RED, default: 0.25)

Compiling and running tests

In general, just compiling the executable and linking it to the TestSuite library is enough, no further setup is needed. When running, the test produces output to standard output / standard error and exits with non-zero code in case of a test failure.

if(CORRADE_TARGET_EMSCRIPTEN OR CORRADE_TARGET_ANDROID)
    set(JPEG_TEST_DIR ".")
else()
    set(JPEG_TEST_DIR ${CMAKE_CURRENT_SOURCE_DIR})
endif()

# Contains just
#  #define JPEG_TEST_DIR "${JPEG_TEST_DIR}"
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/configure.h.cmake
               ${CMAKE_CURRENT_BINARY_DIR}/configure.h)

corrade_add_test(JpegTest JpegTest.cpp
    LIBRARIES ${JPEG_LIBRARIES}
    FILES rgb.jpg rgba.jpg grayscale.jpg)
target_include_directories(JpegTest ${CMAKE_CURRENT_BINARY_DIR})

adb push <path-to-the-test-build>/MyTest /data/local/tmp
adb shell /data/local/tmp/MyTest

cd <test-build-directory>
node MyTest.js

cd <test-build-directory>
python -m http.server

Public types

class TesterConfiguration

Tester configuration.

enum class BenchmarkType { Default = 1, WallTime = 2, WallClock = int(WallTime) deprecated, CpuTime = 3, CpuCycles = 4 }

Benchmark type.

enum class BenchmarkUnits { Nanoseconds = 100, Time = int(Nanoseconds) deprecated, Cycles = 101, Instructions = 102, Bytes = 103, Memory = int(Bytes) deprecated, Count = 104 }

Custom benchmark units.

using Debug = Corrade::Utility::Debug

Alias for debug output.

using Warning = Corrade::Utility::Warning

Alias for warning output.

using Error = Corrade::Utility::Error

Alias for error output.

Constructors, destructors, conversion operators

Tester(TesterConfiguration configuration = TesterConfiguration{}) explicit

Constructor.

Public functions

auto arguments() -> std::pair<int&, char**>

Command-line arguments.

template<class Derived>

void addTests(std::initializer_list<void(Derived::*)()> tests)

Add test cases.

template<class Derived>

void addRepeatedTests(std::initializer_list<void(Derived::*)()> tests, std::size_t repeatCount)

Add repeated test cases.

template<class Derived>

void addTests(std::initializer_list<void(Derived::*)()> tests, void(Derived::*)() setup, void(Derived::*)() teardown)

Add test cases with explicit setup and teardown functions.

template<class Derived>

void addRepeatedTests(std::initializer_list<void(Derived::*)()> tests, std::size_t repeatCount, void(Derived::*)() setup, void(Derived::*)() teardown)

Add repeated test cases with explicit setup and teardown functions.

template<class Derived>

void addInstancedTests(std::initializer_list<void(Derived::*)()> tests, std::size_t instanceCount)

Add instanced test cases.

template<class Derived>

void addRepeatedInstancedTests(std::initializer_list<void(Derived::*)()> tests, std::size_t repeatCount, std::size_t instanceCount)

Add repeated instanced test cases.

template<class Derived>

void addInstancedTests(std::initializer_list<void(Derived::*)()> tests, std::size_t instanceCount, void(Derived::*)() setup, void(Derived::*)() teardown)

Add instanced test cases with explicit setup and teardown functions.

template<class Derived>

void addRepeatedInstancedTests(std::initializer_list<void(Derived::*)()> tests, std::size_t repeatCount, std::size_t instanceCount, void(Derived::*)() setup, void(Derived::*)() teardown)

Add repeated instanced test cases with explicit setup and teardown functions.

template<class Derived>

void addBenchmarks(std::initializer_list<void(Derived::*)()> benchmarks, std::size_t batchCount, BenchmarkType benchmarkType = BenchmarkType::Default)

Add benchmarks.

template<class Derived>

void addBenchmarks(std::initializer_list<void(Derived::*)()> benchmarks, std::size_t batchCount, void(Derived::*)() setup, void(Derived::*)() teardown, BenchmarkType benchmarkType = BenchmarkType::Default)

Add benchmarks with explicit setup and teardown functions.

template<class Derived>

void addCustomBenchmarks(std::initializer_list<void(Derived::*)()> benchmarks, std::size_t batchCount, void(Derived::*)() benchmarkBegin, std::uint64_t(Derived::*)() benchmarkEnd, BenchmarkUnits benchmarkUnits)

Add custom benchmarks.

template<class Derived>

void addCustomBenchmarks(std::initializer_list<void(Derived::*)()> benchmarks, std::size_t batchCount, void(Derived::*)() setup, void(Derived::*)() teardown, void(Derived::*)() benchmarkBegin, std::uint64_t(Derived::*)() benchmarkEnd, BenchmarkUnits benchmarkUnits)

Add custom benchmarks with explicit setup and teardown functions.

template<class Derived>

void addInstancedBenchmarks(std::initializer_list<void(Derived::*)()> benchmarks, std::size_t batchCount, std::size_t instanceCount, BenchmarkType benchmarkType = BenchmarkType::Default)

Add instanced benchmarks.

template<class Derived>

void addInstancedBenchmarks(std::initializer_list<void(Derived::*)()> benchmarks, std::size_t batchCount, std::size_t instanceCount, void(Derived::*)() setup, void(Derived::*)() teardown, BenchmarkType benchmarkType = BenchmarkType::Default)

Add instanced benchmarks with explicit setup and teardown functions.

template<class Derived>

void addCustomInstancedBenchmarks(std::initializer_list<void(Derived::*)()> benchmarks, std::size_t batchCount, std::size_t instanceCount, void(Derived::*)() benchmarkBegin, std::uint64_t(Derived::*)() benchmarkEnd, BenchmarkUnits benchmarkUnits)

Add custom instanced benchmarks.

template<class Derived>

void addCustomInstancedBenchmarks(std::initializer_list<void(Derived::*)()> benchmarks, std::size_t batchCount, std::size_t instanceCount, void(Derived::*)() setup, void(Derived::*)() teardown, void(Derived::*)() benchmarkBegin, std::uint64_t(Derived::*)() benchmarkEnd, BenchmarkUnits benchmarkUnits)

Add custom instanced benchmarks with explicit setup and teardown functions.

auto testCaseId() const -> std::size_t

Test case ID.

auto testCaseInstanceId() const -> std::size_t

Test case instance ID.

auto testCaseRepeatId() const -> std::size_t

Test case repeat ID.

void setTestName(const std::string& name)

Set custom test name.

void setTestName(std::string&& name)

void setTestCaseName(const std::string& name)

Set custom test case name.

void setTestCaseName(std::string&& name)

void setTestCaseDescription(const std::string& description)

Set test case description.

void setTestCaseDescription(std::string&& description)

void setBenchmarkName(const std::string& name)

Set benchmark name.

void setBenchmarkName(std::string&& name)

Enum documentation

Typedef documentation

void myTestCase() {
    int a = 4;
    Debug() << a;
    CORRADE_COMPARE(a + a, 8);
}

Function documentation

setTestCaseName(__func__);