5.7 KiB
CI Bench
This crate is meant for CI benchmarking. It measures CPU instructions using cachegrind
, outputs
the results in CSV format and allows comparing results from multiple runs.
Usage
You can get detailed usage information through cargo run --release -- --help
. Below are the most
important bits.
Running all benchmarks
Note: this step requires having valgrind
in your path.
Use cargo run --release -- run-all --output-dir foo
to generate the results inside the foo
directory. Within that directory, you will find an icounts.csv
file with the instruction counts
for the different scenarios we support. It should look like the following:
handshake_no_resume_1.2_rsa_aes_server,11327015
handshake_no_resume_1.2_rsa_aes_client,4314952
handshake_session_id_1.2_rsa_aes_server,11342136
handshake_session_id_1.2_rsa_aes_client,4327564
handshake_tickets_1.2_rsa_aes_server,11347746
handshake_tickets_1.2_rsa_aes_client,4331424
transfer_no_resume_1.2_rsa_aes_server,8775780
transfer_no_resume_1.2_rsa_aes_client,8818847
handshake_no_resume_1.3_rsa_aes_server,11517007
handshake_no_resume_1.3_rsa_aes_client,4212770
...
... rest omitted for brevity
...
In the cachegrind
subdirectory you will find output files emitted by the cachegrind
tool, which
are useful to report detailed instruction count differences when comparing two benchmark runs. This
subdirectory also contains log information from cachegrind itself (in .log
files), which can be
used to diagnose unexpected cachegrind crashes.
Comparing results
Use cargo run --release -- compare foo bar
. It will output a report using GitHub-flavored markdown
(used by the CI itself to give feedback about PRs). We currently consider differences of 0.2% to be
significant, but might tweak it in the future after we gain experience with the benchmarking setup.
Supported scenarios
We benchmark the following scenarios:
- Handshake without resumption (
handshake_no_resume
) - Handshake with ticket resumption (
handshake_tickets
) - Handshake with session id resumption (
handshake_session_id
) - Encrypt, transfer and decrypt 1MB of data sent by the server
(
transfer_no_resume
)
The scenarios are benchmarked with different TLS versions, certificate key types and cipher suites.
CPU counts are measured independently for the server side and for the client side. Hence, we end up
with names like transfer_no_resume_1.3_rsa_aes_client
.
Internals
We have made an effort to heavily document the source code of the benchmarks. In addition to that, here are some high-level considerations that can help you hack on the crate.
Architecture
An important goal of this benchmarking setup is that it should run with minimum noise on
standard GitHub Actions runners. We achieve that by measuring CPU instructions using cachegrind
,
which runs fine on the cloud (contrary to hardware instruction counters). This is the same
approach used by the iai benchmarking crate, but we needed more
flexibility and have therefore rolled our own setup.
Using cachegrind
has some architectural consequences because it operates at the process level
(i.e. it can count CPU instructions for a whole process, but not for a single function). The
most important consequences are:
- Since we want to measure server and client instruction counts separately, the benchmark runner spawns two child processes for each benchmark (one for the client, one for the server) and pipes their stdio to each other for communication (i.e. stdio acts as the transport layer).
- There is a no-op "benchmark" that measures the overhead of starting up the child process, so we can subtract it from the instruction count of the real benchmarks and reduce noise.
- Since we want to measure individual portions of code (e.g. data transfer after the handshake), there is a mechanism to subtract the instructions that are part of a benchmark's setup. Specifically, a benchmark can be configured to have another benchmark's instruction count subtracted from it. We are currently using this to subtract the handshake instructions from the data transfer benchmark.
Debugging
If you need to debug the crate, here are a few tricks that might help:
- For printf debugging, you should use
eprintln!
, because child processes use stdio as the transport for the TLS connection (i.e. if you print something to stdout, you won't even see it and the other side of the connection will choke on it). - When using a proper debugger, remember that each side of the connection runs as a child process.
If necessary, you can tweak the code to ensure both sides of the connection run on the parent
process (e.g. by starting each side on its own thread and having them communicate through TCP).
This should require little effort, because the TLS transport layer is encapsulated and generic
over
Read
andWrite
.
Why measure CPU instructions
This technique has been successfully used in tracking the Rust compiler's performance, and is known to work well when comparing two versions of the same code. It has incredibly low noise, and therefore makes for a very good metric for automatic PR checking (i.e. the automatic check will reliably identify significant performance changes).
It is not possible to deduce the exact change in runtime based on the instruction count difference (e.g. a 5% increase in instructions does not necessarily result in a 5% increase in runtime). However, if there is a significant change in instruction count, you can be fairly confident there is a significant change in runtime too. This is very useful information to have when reviewing a PR.
For more information, including the alternatives we considered, check out [this comment] (https://github.com/rustls/rustls/issues/1385#issuecomment-1668023152) in the issue tracker.