196 lines
7.9 KiB
Markdown
196 lines
7.9 KiB
Markdown
# Benchmarking
|
|
|
|
Benchmarking new code against old code is extremely important whenever making
|
|
large changes to how something works. If you are attempting to make something
|
|
faster, and you end up slowing it down, you'll never know if you don't
|
|
benchmark! We have benchmarks in the `toolsrc/src/vcpkg-test` directory, just
|
|
like the tests -- they're treated as a special kind of test.
|
|
|
|
## Running Benchmarks
|
|
|
|
Unlike normal tests, benchmarks are hidden behind a special define -- `CATCH_CONFIG_ENABLE_BENCHMARKING` -- so that you never try to run benchmarks
|
|
unless you specifically want to. This is because benchmarks actually take quite
|
|
a long time! However, if you want to run benchmarks (and I recommend running
|
|
only specific benchmarks at a time), you can do so by passing the
|
|
`VCPKG_ENABLE_BENCHMARKING` option at cmake configure time.
|
|
|
|
```sh
|
|
$ cmake -B toolsrc/out -S toolsrc -G Ninja \
|
|
-DCMAKE_BUILD_TYPE=Release \
|
|
-DVCPKG_BUILD_BENCHMARKING=On
|
|
|
|
-- The C compiler identification is MSVC 19.22.27905.0
|
|
-- The CXX compiler identification is MSVC 19.22.27905.0
|
|
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe
|
|
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe -- works
|
|
-- Detecting C compiler ABI info
|
|
-- Detecting C compiler ABI info - done
|
|
-- Detecting C compile features
|
|
-- Detecting C compile features - done
|
|
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe
|
|
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe -- works
|
|
-- Detecting CXX compiler ABI info
|
|
-- Detecting CXX compiler ABI info - done
|
|
-- Detecting CXX compile features
|
|
-- Detecting CXX compile features - done
|
|
-- Looking for pthread.h
|
|
-- Looking for pthread.h - not found
|
|
-- Found Threads: TRUE
|
|
-- Configuring done
|
|
-- Generating done
|
|
-- Build files have been written to: C:/Users/t-nimaz/src/vcpkg/toolsrc/out
|
|
|
|
$ cmake --build toolsrc/out
|
|
|
|
[0/2] Re-checking globbed directories...
|
|
[80/80] Linking CXX executable vcpkg-test.exe
|
|
```
|
|
|
|
You can then run benchmarks easily with the following command (which run the
|
|
files benchmarks):
|
|
|
|
```sh
|
|
$ ./toolsrc/out/vcpkg-test [!benchmark][file]
|
|
```
|
|
|
|
You can switch out `[file]` for a different set -- `[hash]`, for example.
|
|
|
|
## Writing Benchmarks
|
|
|
|
First, before anything else, I recommend reading the
|
|
[benchmarking documentation] at Catch2's repository.
|
|
|
|
Now, after that, let's say that you wanted to benchmark, say, our ASCII
|
|
case-insensitive string compare against your new implementation. We place
|
|
benchmarks for code in the same file as their tests, so open
|
|
`vcpkg-test/strings.cpp`, and add the following at the bottom:
|
|
|
|
```cpp
|
|
#if defined(CATCH_CONFIG_ENABLE_BENCHMARKING)
|
|
TEST_CASE ("case insensitive ascii equals: benchmark", "[strings][!benchmark]")
|
|
{
|
|
BENCHMARK("qwertyuiop") {
|
|
return vcpkg::Strings::case_insensitive_ascii_equals("qwertyuiop", "QWERTYUIOP");
|
|
};
|
|
}
|
|
#endif
|
|
```
|
|
|
|
Remember the `;` at the end of the benchmark -- it's not required for
|
|
`TEST_CASE`s, but is for `BENCHMARK`s.
|
|
|
|
Now, let's rebuild and run:
|
|
|
|
```sh
|
|
$ cmake --build toolsrc/out
|
|
[0/2] Re-checking globbed directories...
|
|
[2/2] Linking CXX executable vcpkg-test.exe
|
|
$ ./toolsrc/out/vcpkg-test [strings][!benchmark]
|
|
Filters: [strings][!benchmark]
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
vcpkg-test.exe is a Catch v2.9.1 host application.
|
|
Run with -? for options
|
|
|
|
-------------------------------------------------------------------------------
|
|
case insensitive ascii equals: benchmark
|
|
-------------------------------------------------------------------------------
|
|
C:\Users\t-nimaz\src\vcpkg\toolsrc\src\vcpkg-test\strings.cpp(36)
|
|
...............................................................................
|
|
|
|
benchmark name samples iterations estimated
|
|
mean low mean high mean
|
|
std dev low std dev high std dev
|
|
-------------------------------------------------------------------------------
|
|
qwertyuiop 100 2088 3.9672 ms
|
|
25 ns 24 ns 26 ns
|
|
6 ns 5 ns 8 ns
|
|
|
|
|
|
===============================================================================
|
|
test cases: 1 | 1 passed
|
|
assertions: - none -
|
|
```
|
|
|
|
You've now written your first benchmark!
|
|
|
|
But wait. This seems kind of silly. Benchmarking the comparison of literal
|
|
strings is great and all, but could we make it a little more realistic?
|
|
|
|
This is where `BENCHMARK_ADVANCED` comes in. `BENCHMARK_ADVANCED` allows one to
|
|
write a benchmark that has a little setup to it without screwing up the numbers.
|
|
Let's try it now:
|
|
|
|
```cpp
|
|
TEST_CASE ("case insensitive ascii equals: benchmark", "[strings][!benchmark]")
|
|
{
|
|
BENCHMARK_ADVANCED("equal strings")(Catch::Benchmark::Chronometer meter)
|
|
{
|
|
std::vector<std::string> strings;
|
|
strings.resize(meter.runs());
|
|
std::mt19937_64 urbg;
|
|
std::uniform_int_distribution<std::uint64_t> data_generator;
|
|
|
|
std::generate(strings.begin(), strings.end(), [&] {
|
|
std::string result;
|
|
for (std::size_t i = 0; i < 1000; ++i)
|
|
{
|
|
result += vcpkg::Strings::b32_encode(data_generator(urbg));
|
|
}
|
|
|
|
return result;
|
|
});
|
|
|
|
meter.measure(
|
|
[&](int run) { return vcpkg::Strings::case_insensitive_ascii_equals(strings[run], strings[run]); });
|
|
};
|
|
}
|
|
```
|
|
|
|
Then, run it again!
|
|
|
|
```sh
|
|
$ cmake --build toolsrc/out
|
|
[0/2] Re-checking globbed directories...
|
|
[2/2] Linking CXX executable vcpkg-test.exe
|
|
$ toolsrc/out/vcpkg-test [strings][!benchmark]
|
|
Filters: [strings][!benchmark]
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
vcpkg-test.exe is a Catch v2.9.1 host application.
|
|
Run with -? for options
|
|
|
|
-------------------------------------------------------------------------------
|
|
case insensitive ascii equals: benchmark
|
|
-------------------------------------------------------------------------------
|
|
C:\Users\t-nimaz\src\vcpkg\toolsrc\src\vcpkg-test\strings.cpp(36)
|
|
...............................................................................
|
|
|
|
benchmark name samples iterations estimated
|
|
mean low mean high mean
|
|
std dev low std dev high std dev
|
|
-------------------------------------------------------------------------------
|
|
equal strings 100 2 5.4806 ms
|
|
22.098 us 21.569 us 23.295 us
|
|
3.842 us 2.115 us 7.41 us
|
|
|
|
|
|
===============================================================================
|
|
test cases: 1 | 1 passed
|
|
assertions: - none -
|
|
```
|
|
|
|
And now you have a working benchmark to test the speed of the existing code, and
|
|
of new code!
|
|
|
|
If you're writing a lot of benchmarks that follow the same sort of pattern, with
|
|
some differences in constants, look into `vcpkg-test/files.cpp`'s benchmarks --
|
|
there are a lot of things one can do to make writing new benchmarks really easy.
|
|
|
|
If you wish to add a benchmark for a piece of code that has not yet been tested,
|
|
please read the [testing documentation], and please write some unit tests.
|
|
The speed of your code isn't very important if it doesn't work at all!
|
|
|
|
[benchmarking documentation]: https://github.com/catchorg/Catch2/blob/master/docs/benchmarks.md#top
|
|
[testing documentation]: ./testing.md#adding-new-test-files
|