Mehdi Akiki

Posts tagged "statistics"

  • Published on
    Compare important workflows as well as the overall evaluation result. Report case counts and uncertainty so an aggregate gain cannot hide a smaller critical regression.
  • Published on

    Investigation · Part 20 of 20 · When the benchmark lies

    A Benchmark Result Someone Else Can Check

    For the last part, I ran two earlier parts again with the same script. The assembly was identical, part 1 ran 10 to 14 percent faster, and the conclusions of both parts held. This is what a result must carry so that someone else can make the same check.
  • Published on

    Investigation · Part 14 of 20 · When the benchmark lies

    Can Environment Variables Change How Fast Your Code Runs?

    A bigger environment moves every variable on the stack, and a 2009 paper showed that small details of the setup can bias benchmarks. I scanned 256 environment sizes three times. The first scan found slow sizes; the shuffled scan showed that they came from the order of the runs.
  • Published on

    Investigation · Part 13 of 20 · When the benchmark lies

    Compare Two Versions by Interleaving the Runs

    I compared a program with an identical copy of itself, and in 3 of 5 experiments the copy was significantly faster or slower. Running the two versions in blocks measured the moment, not the version. Interleaved runs found no difference, and found a real 4% change every time.
  • Published on

    Investigation · Part 12 of 20 · When the benchmark lies

    How Many Benchmark Runs Are Enough

    I ran the same sort in 300 fresh processes, in three sets of 100. Each set gave a median with a narrow bootstrap interval, and the intervals did not agree. More runs make the interval smaller, but only if the runs are independent, and on my laptop they were not.