- Published on
A Benchmark Result Someone Else Can Check
- Authors
- Name
- Mehdi Akiki
Investigation · Part 20 of 20 · When the benchmark lies
For this last part, I did what I would like a reader to do with any part of this series: I ran two of them again, with the same script, and compared the new files with the published ones. The assembly that the articles show was identical. The times of part 1 were 10 to 14 percent shorter. And the conclusions of both parts stayed the same.
That is a good result, but only because the files had enough information to say why the times moved. This part is about that information.
What a number needs around it
A benchmark result is a number plus everything that can change it. In this series, each of these things changed a number at least once:
- The hardware and the core type. The same chain took 1.8 to 1.9 times longer on an efficiency core, and a busy sibling thread made a performance core as slow as an efficiency core (part 7).
- The frequency and its settings. The clock of one core moved between 3.3 and 4.2 GHz during one minute, without any change in my program (part 6).
- The load. With all 16 CPUs busy, the loop of part 1 was more than three times slower.
- The toolchain and the flags. A test was 24 to 39 times slower under
cargo testthan with--release, in two single runs (part 2). - The exact commands, so that the run can be repeated without guessing.
- The raw samples and the summary method. The mean of four million
pushcalls was 22.8 ns, and the slowest call took 315,684 ns (part 8). - The number of runs, and whether the runs are really independent (part 12).
- The interleaving and the A/A check. Run in blocks, a program was significantly faster or slower than an identical copy of itself in 3 of 5 experiments (part 13).
A machine reserved for benchmarks removes many of these effects at the source, and part 6 lists the usual steps. They need root, and this is a shared laptop, so I can not do that. The next best thing is to write them down next to every number.
The format of this series
The main results file of every part starts and ends with the same few lines, written by run.sh: the compiler version, the load average, the governor and the current frequency of the CPU, and how busy each CPU was during one second. For this part, run.sh also describes the machine itself:
# part 20: the machine and its settings
cpu: 12th Gen Intel(R) Core(TM) i7-1260P
microcode: 0x43b
kernel: Linux 6.6.126-1-MANJARO
...
cpu2: max 4700000 kHz, base 2100000 kHz, core shared with CPUs 2-3, governor powersave, energy_performance_preference power
...
cpu10: max 3400000 kHz, base 1500000 kHz, core shared with CPUs 10, governor powersave, energy_performance_preference power
...
intel_pstate: status active, no_turbo 0, min_perf_pct 10, max_perf_pct 100
transparent huge pages: enabled [always] madvise never, defrag always defer defer+madvise [madvise] never
clocksource: tsc
randomize_va_space: 2
rustc 1.98.0 (88d9e12ae 2026-08-18)
host: x86_64-unknown-linux-gnu
LLVM version: 22.1.8
...
Each line here explains a number somewhere in the series. energy_performance_preference power is the hint to save energy that part 6 found. [always] for transparent huge pages is why the slowest page faults of part 8 were so big. randomize_va_space: 2 means that the stack, the libraries and the heap start at new addresses in every process, and part 14 had to turn this off for its own processes with setarch -R. And the two kinds of CPU lines are the two kinds of cores of part 7.
The weakest point of my format is the raw samples. Most files keep one summary line per run, not every sample. The harness can write all of them, and for this part I used it, with a small Python script that computes the summary again from the file:
printed by the program: harness_sum min 276.237 median 280.596 p90 401.481 max 476.815 ns/call (n=31)
computed from the file: min 276.237 median 280.596 p90 401.481 max 476.815 (n=31)
The two lines agree, so the summary method can be checked, not only trusted. The file also shows what the median hides: 27 of the 31 samples are between 276 and 294 ns, and the other four between 401 and 477 ns. The median says 281 ns and nothing about these four.
Part 1 and part 17, again
sh run.sh 20 runs parts 1 and 17 again into another folder and compares the new files with the published ones. First the assembly of part 1:
== part 1: assembly, published against new
Rust, rustc -O: assembly of bench_unused identical (9 lines)
Rust, rustc -O: assembly of bench_printed_outlined identical (36 lines)
Rust, rustc -O: assembly of bench_black_box identical (74 lines)
C, gcc -O2: assembly of bench_unused identical (26 lines)
C, gcc -O2: assembly of bench_volatile identical (53 lines)
The four assembly listings of part 17 are identical too. The same compilers made the same code, line by line. This is important, because it means that the programs I timed again are exactly the programs of the articles. Then the times:
== part 1: ns per call in three runs, published, new, and the ratio of the medians
Rust, unused result 0.00005 0.00004 0.00005 0.00013 0.00004 0.00005 1.00
Rust, printed 401.0 321.1 325.4 381.1 292.5 283.5 0.90
Rust, printed, outlined 0.00052 0.00114 0.00048 0.00150 0.00051 0.00074 1.42
Rust, black_box 311.5 328.0 327.8 283.9 284.5 296.0 0.87
Rust, harness_sum 314.3 327.9 327.8 279.6 281.2 280.5 0.86
C, unused result 0.00003 0.00003 0.00003 0.00004 0.00004 0.00004 1.33
C, volatile sink 518.2 509.4 513.0 445.8 453.5 449.3 0.88
C, asm barrier 507.1 518.1 513.4 446.1 448.3 439.0 0.87
The impossible numbers are still impossible: the loops that the compiler removed, or reduced to one call, still take between 0.00003 and 0.0015 ns per call. For them the last column means nothing. These are totals of 32 to 1,502 ns for a million calls, so a few tens of nanoseconds more or less change the ratio a lot. The real loops take 10 to 14 percent less time than before: the black_box loop 284 to 296 ns instead of 311 to 328, and the C loops 439 to 454 ns instead of 507 to 518.
Why 10 to 14 percent? The first thing to look at is the conditions, at the start and at the end of each file:
published load average 3.20, 9.11, 8.37
published cpu2: governor powersave, scaling_cur_freq 3288160 kHz
published load average 3.53, 8.99, 8.34
published cpu2: governor powersave, scaling_cur_freq 3863518 kHz
new load average 1.41, 1.32, 1.60
new cpu2: governor powersave, scaling_cur_freq 1201693 kHz
new load average 1.38, 1.32, 1.60
new cpu2: governor powersave, scaling_cur_freq 4518712 kHz
When the published run started, the load average of the last five minutes was 9.11, so something had used many CPUs just before. For the new run it was 1.32. And the frequency read at the end of each file, right after the last C run, was 3,864 MHz then and 4,519 MHz now: 17 percent higher, close to the difference in time. One reading is a hint, not a proof. But part 6 showed that busy cores lower the clock of my core, and the two lines point in the same direction. Without them, a reader who gets 284 ns could only guess if the program, the method or the machine was different.
Something did not change at all: in both files, the first run of "printed" is the slowest, 401 and 381 ns. It is the first long measurement of the script, so I suspect the slow start of a CPU that was idle before, but I did not measure it. A re-run that repeats even the strange details tells me that they are not random.
Part 17 is about ratios, not about absolute times, and the ratios moved even less:
== part 17: ratio random/sorted in three runs, published, new, and the ratio of the medians
opt-level=3, sum_filter 1.11 1.00 1.00 1.05 1.01 1.01 1.01
opt-level=3, sum_if 1.10 0.93 1.00 1.00 1.00 1.00 1.00
opt-level=3, sum_if_effect 2.91 2.91 2.98 2.87 3.25 2.88 0.99
opt-level=1, sum_filter 0.94 0.99 0.96 1.00 1.00 1.00 1.04
opt-level=1, sum_if 1.02 0.93 1.00 1.00 1.00 0.97 1.00
opt-level=1, sum_if_effect 10.15 10.01 10.46 10.20 10.22 10.18 1.00
The branch that the compiler must keep still makes random data about 10 times slower at opt-level=1, and 2.9 to 3.3 times slower at opt-level=3. The functions without a branch stay between 0.93 and 1.11 in both files. The frequency at the end of part 17 was even lower in the new run, 2,953 MHz against 3,175, and the ratios did not care, because both sides of a ratio are measured one right after the other, in the same process.
I also ran the complete fixture once more, all twenty parts with the same run.sh, and its files are in results/second-run. During that run the laptop was quieter, and this changed more results than a change of code would: the slow start of the clock that part 6 saw once appeared in many short processes. It made most JVM runs of part 5 two to three times slower, and the comparison of the two JIT compilers in that part did not reproduce. Parts 4, 5, 6, 7, 12, 13 and 16 describe what changed in their second run, and parts 2, 3, 8, 14, 15, 18 and 19 mention some details of theirs. The main conclusions of all parts held, with different absolute numbers.
What my files do not carry
Writing this part showed me also what is missing in my own results:
- The frequency during each run. Most files read it only at the start and at the end. Parts 6, 7, 11, 12 and 18 read it during or right after each run, and in those parts I could explain much more.
- The raw samples, as above.
- The other programs. I can see that CPU 6 was busy, not what it was doing.
- An A/A check for every part. Only parts 13 and 15 have an explicit one.
In fact, the comparison above is a kind of A/A test over time: the same code on the same machine, at another moment. It moved the times of part 1 by 10 to 14 percent. So a difference of a few percent between two runs made at different moments, on this laptop, means nothing by itself.
Looking back at twenty parts
Most of the wrong numbers in this series were not wrong measurements. The clock measured correctly what the machine did. The compiler deleted a loop (part 1), computed a sum before the run (part 3), or removed a branch (part 17). The machine changed its clock (part 6), ran my program on another kind of core (part 7), or gave the same bytes a different speed at another address (part 15). The statistics hid the slow requests (parts 8 and 9), or found a difference between a program and its own copy (part 13). And the program around the benchmark changed the answer (parts 18 and 19). In each case the number was true, but it answered another question than the one I had.
So the most useful check of the series was the cheapest one, from part 1: is this number possible for this machine? The second one was the A/A test: measure a thing against itself before you believe a difference. And some effects did not appear at all, like the size of the environment in part 14. I think these negative results are as useful as the others, as long as they come with their files.
The same rules work for the pages of this site. The site performance lab separates the result, the experiment and the conclusion of each investigation, keeps the failed and excluded runs, and for the last experiments wrote the go and no-go gates before the runs. A result that someone else can check is more honest, and it is also the only kind of result that I can check myself, later.
Everything of this series is in the fixture: the programs, run.sh, and the results, including p20-machine.txt and p20-rerun.txt of this part.