Posts tagged "when the benchmark lies"

I placed the same 11-instruction loop at all 64 positions of a cache line, two copies each, and measured them in shuffled order. At one position the loop was 16 to 21 percent slower, in three runs out of three. The same bytes, a different address, a different speed.

A bigger environment moves every variable on the stack, and a 2009 paper showed that small details of the setup can bias benchmarks. I scanned 256 environment sizes three times. The first scan found slow sizes; the shuffled scan showed that they came from the order of the runs.

I compared a program with an identical copy of itself, and in 3 of 5 experiments the copy was significantly faster or slower. Running the two versions in blocks measured the moment, not the version. Interleaved runs found no difference, and found a real 4% change every time.

I ran the same sort in 300 fresh processes, in three sets of 100. Each set gave a median with a narrow bootstrap interval, and the intervals did not agree. More runs make the interval smaller, but only if the runs are independent, and on my laptop they were not.

Timed one call at a time, a small hash function and an empty loop both take 15 ns: the clock measures itself. With more calls per measurement, the same function costs 0.79 ns per call, or 3.02 ns when each call waits for the previous one.