An overall eval score can hide the workflow you broke

Article · Measurement

Version B has a better overall score. The release looks easy until someone separates ordinary lookups from the smaller workflow that changes records.

The common workflow improved. The write workflow got worse.

An aggregate answers a weighted question about the cases in the dataset. It does not promise that every important part of the product improved.

A small invented example

These counts describe a synthetic dataset. They are not measurements from a model comparison:

The overall result rises from 90% to about 93.6%. The sensitive-write result falls from nine passes to five.

That is enough to ask a release question. It is not enough to estimate the true production failure rate of the write workflow precisely. Ten cases are a small slice, and repeated model runs may vary.

The paired-comparison article explains how to compare versions on the same cases and account for run-to-run noise. Apply that reasoning within the workflows you care about, not only to a single total.

Choose slices before seeing the winner

Start from product use and failure consequences: lookup, extraction, external write, permission denial, unsupported input, and recovery after a tool failure may deserve separate views.

Do not invent dozens of slices after a run and promote whichever one supports the preferred conclusion. Keep the intended comparisons and important invariants visible before the release test.

Slices may overlap. A cross-tenant write case can belong to both write and authorization categories. Explain the membership and do not add overlapping counts as if they were separate independent observations.

The dataset's frequency mix may differ from production traffic. Report that limitation. A balanced diagnostic suite and a traffic-weighted estimate answer different questions and can both be useful.

Inspect the changed cases

For each important slice, list cases where A passed and B failed, and the reverse. Review the actual trajectory when a failure involves tools. A better final sentence cannot compensate for a forbidden side effect.

Some properties should be deterministic gates rather than average quality points. If a fixture performs an unauthorized write, averaging it with many successful summaries is not a meaningful security decision.

The shipping-evidence article connects failure classes to evidence. Keep those classes in the release report so the score remains attached to a product behaviour.

Report what the sample supports

Include the number of cases, repeated runs where used, the comparison method, and uncertainty appropriate to the data. Separate a reproducible invariant violation from an estimated quality-rate change.

If the write slice is too small to resolve the question, collect more targeted evidence. Do not hide it in a larger easy-case total and call the uncertainty solved.

A good release report can still lead to a simple decision: ship, hold, or restrict the changed workflow. The useful improvement is that the decision reflects the parts of the system users depend on, rather than whichever part contributed the most rows to one average.

Turn the table into an explicit release decision

The runnable gate uses the same synthetic dataset as the table. Candidate A passes ninety lookup cases and nine write cases: ninety-nine of 110 in total. Candidate B passes ninety-eight lookups and five writes: 103 of 110. The higher aggregate score is real within this invented example, and it still hides a serious loss in the write slice.

Before evaluating either candidate, the fixture sets minimum evidence counts of one hundred lookup cases and ten write cases. It also sets separate pass floors: eighty-five percent for lookup and eighty percent for writes. A passes those conditions. B fails the write condition despite its higher total.

The gate has another independent rule: any recorded forbidden effect holds the release. The synthetic B data includes one such effect. That rule is not averaged into a percentage where enough harmless lookup successes could cancel it out. A one-case sample also fails the minimum-count rule rather than reporting a perfect score as sufficient evidence.

The Rust gate and assertions are deterministic arithmetic over labelled cases. No live model comparison produced these numbers. The thresholds are an example policy chosen to demonstrate the decision, not statistically justified universal limits for AI applications.

A real comparison needs stable case definitions, a documented grading rule, and repeated runs where outputs vary. Inspect paired changes: which exact inputs improved, which regressed, and whether graders or prompts changed. If a critical slice contains too few cases, the right conclusion may be insufficient evidence rather than approval or rejection based on an unstable rate.

Keep cost, latency, refusal quality, and effect safety visible alongside answer correctness. Different workflows can justify different tradeoffs. The release record should explain those choices in terms a reviewer can inspect. A single score is useful for navigation through the results; it should not erase the failures that determine whether a user can trust the particular workflow they need.