Posts tagged "when the benchmark lies"

A small Java function took 66 ns per call in the first batch and 3.6 ns after 185 batches. I matched the speed changes with the JIT compile log, compared C1 and C2, and looked at why the steady state was not one number.

The first iteration of my loop took 145 microseconds and the steady state 12. I separated the causes I could show on my laptop: page faults, cold caches, a pause of 300 ms, lazy binding in C, and the start of the process itself.

On my laptop, summing the numbers below one thousand takes 0.81 ns, and below one billion also 0.81 ns. black_box hid the value of n, but LLVM replaced the loop with a formula. I show the assembly, the IR, the LLVM pass, and a benchmark that really grows with n.

A test called score_is_fast measured 44,646 ns per call under cargo test and 1,136 ns with --release, one run each. I compare four optimization levels in Rust and two in C, and I show in the assembly what a debug build really measures.

My first benchmark said that my laptop adds 4,096 numbers in 0.00005 nanoseconds. The compiler had deleted the loop. I show it in the assembly, fix it with black_box, make the same mistake in C, and explain the method of this series.