diff options
Diffstat (limited to 'contrib/catch2/docs/limitations.md')
| -rw-r--r-- | contrib/catch2/docs/limitations.md | 191 |
1 files changed, 191 insertions, 0 deletions
diff --git a/contrib/catch2/docs/limitations.md b/contrib/catch2/docs/limitations.md new file mode 100644 index 0000000..f5f60ba --- /dev/null +++ b/contrib/catch2/docs/limitations.md @@ -0,0 +1,191 @@ +<a id="top"></a> +# Known limitations + +Over time, some limitations of Catch2 emerged. Some of these are due +to implementation details that cannot be easily changed, some of these +are due to lack of development resources on our part, and some of these +are due to plain old 3rd party bugs. + + +## Implementation limits +### Sections nested in loops + +If you are using `SECTION`s inside loops, you have to create them with +different name per loop's iteration. The recommended way to do so is to +incorporate the loop's counter into section's name, like so: + +```cpp +TEST_CASE( "Looped section" ) { + for (char i = '0'; i < '5'; ++i) { + SECTION(std::string("Looped section ") + i) { + SUCCEED( "Everything is OK" ); + } + } +} +``` + +or with a `DYNAMIC_SECTION` macro (that was made for exactly this purpose): + +```cpp +TEST_CASE( "Looped section" ) { + for (char i = '0'; i < '5'; ++i) { + DYNAMIC_SECTION( "Looped section " << i) { + SUCCEED( "Everything is OK" ); + } + } +} +``` + +### Tests might be run again if last section fails + +If the last section in a test fails, it might be run again. This is because +Catch2 discovers `SECTION`s dynamically, as they are about to run, and +if the last section in test case is aborted during execution (e.g. via +the `REQUIRE` family of macros), Catch2 does not know that there are no +more sections in that test case and must run the test case again. + + +### MinGW/CygWin compilation (linking) is extremely slow + +Compiling Catch2 with MinGW can be exceedingly slow, especially during +the linking step. As far as we can tell, this is caused by deficiencies +in its default linker. If you can tell MinGW to instead use lld, via +`-fuse-ld=lld`, the link time should drop down to reasonable length +again. + + +## Features +This section outlines some missing features, what is their status and their possible workarounds. + +### Thread safe assertions +Catch2's assertion macros are not thread safe. This does not mean that +you cannot use threads inside Catch's test, but that only single thread +can interact with Catch's assertions and other macros. + +This means that this is ok +```cpp + std::vector<std::thread> threads; + std::atomic<int> cnt{ 0 }; + for (int i = 0; i < 4; ++i) { + threads.emplace_back([&]() { + ++cnt; ++cnt; ++cnt; ++cnt; + }); + } + for (auto& t : threads) { t.join(); } + REQUIRE(cnt == 16); +``` +because only one thread passes the `REQUIRE` macro and this is not +```cpp + std::vector<std::thread> threads; + std::atomic<int> cnt{ 0 }; + for (int i = 0; i < 4; ++i) { + threads.emplace_back([&]() { + ++cnt; ++cnt; ++cnt; ++cnt; + CHECK(cnt == 16); + }); + } + for (auto& t : threads) { t.join(); } + REQUIRE(cnt == 16); +``` + +We currently do not plan to support thread-safe assertions. + + +### Process isolation in a test +Catch does not support running tests in isolated (forked) processes. While this might in the future, the fact that Windows does not support forking and only allows full-on process creation and the desire to keep code as similar as possible across platforms, mean that this is likely to take significant development time, that is not currently available. + + +### Running multiple tests in parallel + +Catch2 keeps test execution in one process strictly serial, and there +are no plans to change this. If you find yourself with a test suite +that takes too long to run and you want to make it parallel, you have +to run multiple processes side by side. + +There are 2 basic ways to do that, +* you can split your tests into multiple binaries, and run those binaries + in parallel +* you can run the same test binary multiple times, but run a different + subset of the tests in each process + +There are multiple ways to achieve the latter, the easiest way is to use +[test sharding](command-line.md#test-sharding). + + +## 3rd party bugs + +This section outlines known bugs in 3rd party components (this means compilers, standard libraries, standard runtimes). + + +### Visual Studio 2017 -- raw string literal in assert fails to compile + +There is a known bug in Visual Studio 2017 (VC 15), that causes compilation +error when preprocessor attempts to stringize a raw string literal +(`#` preprocessor directive is applied to it). This snippet is sufficient +to trigger the compilation error: + +```cpp +#include <catch2/catch_test_macros.hpp> + +TEST_CASE("test") { + CHECK(std::string(R"("\)") == "\"\\"); +} +``` + +Catch2 provides a workaround, by letting the user disable stringification +of the original expression by defining `CATCH_CONFIG_DISABLE_STRINGIFICATION`, +like so: +```cpp +#define CATCH_CONFIG_DISABLE_STRINGIFICATION +#include <catch2/catch_test_macros.hpp> + +TEST_CASE("test") { + CHECK(std::string(R"("\)") == "\"\\"); +} +``` + +_Do note that this changes the output:_ +``` +catchwork\test1.cpp(6): +PASSED: + CHECK( Disabled by CATCH_CONFIG_DISABLE_STRINGIFICATION ) +with expansion: + ""\" == ""\" +``` + + +### Clang/G++ -- skipping leaf sections after an exception +Some versions of `libc++` and `libstdc++` (or their runtimes) have a bug with `std::uncaught_exception()` getting stuck returning `true` after rethrow, even if there are no active exceptions. One such case is this snippet, which skipped the sections "a" and "b", when compiled against `libcxxrt` from the master branch +```cpp +#include <catch2/catch_test_macros.hpp> + +TEST_CASE("a") { + CHECK_THROWS(throw 3); +} + +TEST_CASE("b") { + int i = 0; + SECTION("a") { i = 1; } + SECTION("b") { i = 2; } + CHECK(i > 0); +} +``` + +If you are seeing a problem like this, i.e. weird test paths that trigger only under Clang with `libc++`, or only under very specific version of `libstdc++`, it is very likely you are seeing this. The only known workaround is to use a fixed version of your standard library. + + +### Visual Studio 2022 -- can't compile assertion with the spaceship operator + +[The C++ standard requires that `std::foo_ordering` is only comparable with +a literal 0](https://eel.is/c++draft/cmp#categories.pre-3). There are +multiple strategies a stdlib implementation can take to achieve this, and +MSVC's STL has changed the strategy they use between two releases of VS 2022. + +With the new strategy, `REQUIRE((a <=> b) == 0)` no longer compiles under +MSVC. Note that Catch2 can compile code using MSVC STL's new strategy, +but only when compiled with a C++20 conforming compiler. MSVC is currently +not conformant enough, but `clang-cl` will compile the assertion above +using MSVC STL without problem. + +This change got in with MSVC v19.37](https://godbolt.org/z/KG9obzdvE). + |
