Change one performance variable at a time

Article · Measurement

The new scanner is faster. It also uses fewer patterns, returns fewer matches, and no longer includes result formatting in the timed section.

There may be a good optimization in the change. The comparison does not tell us which one it is.

I would first make the two versions do the same work. Performance evidence is much easier to discuss when correctness and workload are held steady.

Write the question before the benchmark

Are you testing a different search implementation, a different pattern set, or a different reporting strategy? Those are separate questions.

Use one recorded corpus and pattern set for the implementation comparison. Preserve match semantics, output ordering where required, and the amount of output produced. Verify those results before timing anything.

The SIMD pattern-shape investigation shows why pattern content can influence the path an implementation takes. A change in input can therefore look like an implementation improvement even when the code did nothing useful.

A simple comparison plan is often enough:

The trials isolate questions. They do not prove that effects will add together when combined.

Keep setup outside the wrong measurement

If the product reuses a compiled matcher, measure construction separately from steady-state search. If it rebuilds a matcher for every request, construction belongs in that request's cost.

Rust's black_box documentation explains one tool for discouraging some compiler optimizations in experiments. It does not make a benchmark representative or prevent every measurement mistake.

Record toolchain, target, optimization profile, input identity, and the exact command used in the existing CI workflow. Keep enough raw evidence to distinguish a real change from run-to-run variation.

Keep the raw observations beside the comparison. If the experiment has not run yet, describe the expected result as a hypothesis. That makes it possible for a reviewer to distinguish the plan from the evidence.

Then test the interaction

Once the isolated changes are understood, combine the useful ones and measure again. An optimization can remove the bottleneck that made another optimization valuable. Two improvements can also compete for cache or memory bandwidth.

Changing one variable at a time is a starting point. When two changes may affect each other, compare all four combinations: neither change, each change alone, and both together.

Keep an unchanged control when runs share infrastructure. If both the control and candidate improve by the same amount, the environment may explain the movement.

Keep the rejected idea too

A measured loss can be useful evidence. Record why the appealing change was rejected and the conditions under which it might be worth revisiting.

For the scanner, the first deliverable is an equivalent result set and a comparison that isolates one cause. The faster-looking version may still win. This process gives that win a meaning a reviewer can trust, rather than a number attached to several unrelated changes.

Four combinations expose an interaction

The HTTP experiment provides a small second example of the same reasoning. It varies two controls: one or two server work permits, and two or eight client requests in flight. Each cell performs the same forty requests, with the same simulated twenty-millisecond work and eighty-millisecond client deadline.

With one permit and concurrency two, the first round completed all forty at about 46.7 successful requests per second. Raising concurrency alone to eight completed five and timed out thirty-five. Successful throughput dropped to about 12.3 per second. The extra queue was a loss under this deadline.

With two permits and concurrency two, all forty completed at about 90.8 per second. Combining two permits with concurrency eight completed fourteen and timed out twenty-six, at about 34.2 successful requests per second. Adding server capacity helped, but it did not make the larger client queue useful.

The second round reverses the order. Its two high-concurrency cells still had many timeouts: thirty-four with one permit and thirty with two. Exact numbers moved, which is another reason to keep raw samples rather than quote one percentage improvement. There are no timing assertions declaring a universal speed advantage.

This fixture measures a controlled loopback service, not a production database and not the scanner from the opening example. The server permits model a scarce work resource; they do not identify which real production resource is scarce. Use the same comparison structure with the workload and correctness checks that matter to your system.

For the scanner, that still means identical matches and reporting semantics before timing. For the HTTP example, it means accounting for every success, timeout, and transport error. A change that finishes less useful work cannot win merely because its successful responses look fast. Keep the interaction visible when writing the result, and let the measured loss rule out an appealing configuration.