Mehdi Akiki

Posts tagged "benchmarking"

  • Published on

    Investigation · Part 14 of 20 · When the benchmark lies

    Can Environment Variables Change How Fast Your Code Runs?

    A bigger environment moves every variable on the stack, and a 2009 paper showed that small details of the setup can bias benchmarks. I scanned 256 environment sizes three times. The first scan found slow sizes; the shuffled scan showed that they came from the order of the runs.
  • Published on

    Investigation · Part 13 of 20 · When the benchmark lies

    Compare Two Versions by Interleaving the Runs

    I compared a program with an identical copy of itself, and in 3 of 5 experiments the copy was significantly faster or slower. Running the two versions in blocks measured the moment, not the version. Interleaved runs found no difference, and found a real 4% change every time.
  • Published on

    Investigation · Part 12 of 20 · When the benchmark lies

    How Many Benchmark Runs Are Enough

    I ran the same sort in 300 fresh processes, in three sets of 100. Each set gave a median with a narrow bootstrap interval, and the intervals did not agree. More runs make the interval smaller, but only if the runs are independent, and on my laptop they were not.
  • Published on
    Pattern count alone cannot predict fast string search. I show how rare bytes, short literals, prefixes, alphabet size, and candidate density shape SIMD-assisted Rust performance.