Don't delete the eval case that made the release look bad

Article · Measurement

The new version fails a difficult evaluation case. Removing that case brings the score back above the release threshold.

Sometimes a case really is obsolete or wrong. Sometimes it is the only example that exposes a regression.

The decision needs a reason that survives the release discussion. A lower score is a signal to investigate, not a sufficient reason to edit the evidence.

Give every case a purpose

An evaluation case should describe the behaviour it protects, its origin, and the expected outcome. A compact record can be enough:

case: invoice-owner-lookup
requirement: do not return another tenant's invoice
origin: synthetic reproduction of a reviewed failure
expected: deny the cross-tenant tool call before execution
owner: support-assistant workflow

This is an example record, not a claim about a production incident. Use safe synthetic data or reviewed redacted material with appropriate provenance.

The record-and-replay article explains how to preserve a workflow's interactions. Keep the failure's relevant trajectory, not only the final answer that happened to expose it.

A case that checks a forbidden effect may need deterministic assertions on tool calls and state. A wording-quality case may need a rubric and several observations. Do not force both into a brittle exact-string comparison.

Separate three different changes

A product requirement can change. Update the case and record the new requirement, with review from the relevant owner.

A fixture can be invalid. Correct its input or expected result, preserve why the old case was wrong, and rerun both versions against the corrected dataset when comparing them.

A regression can be real. Keep the case, fix the behaviour, or document an explicit release decision with its scope and consequence. Deleting the case does not repair the workflow.

Version the dataset and rubric with the results. A score without a dataset identity is difficult to compare honestly.

Keep development cases and held-out evidence distinct

Repeatedly tuning against one small set can produce a system that handles those examples unusually well. Preserve useful regression cases, but also maintain fresh or held-out evidence appropriate to the workflow.

Do not casually move private production conversations into a repository to increase coverage. The input needs a data-handling decision as well as an evaluation purpose.

The evidence-before-shipping article discusses matching evidence to failure classes. One difficult case can expose a serious missing boundary without estimating how frequently that failure occurs in production.

Make removal reviewable

A case-removal change should say which requirement disappeared, what replaces its coverage if anything, and whether the comparison was rerun on a consistent set.

Keep a case that fails on the old behaviour and passes on the corrected behaviour. When feasible, verify that relationship through the recorded harness rather than trusting a comment saying “regression test.”

There is no need to keep every historical case active forever. Duplicate cases, obsolete features, and broken fixtures can be retired. The useful rule is that retirement explains a change in evidence. It should not be the quiet mechanism that makes a weak release look stronger.

Make the retained case fail a deliberately broken lookup

The regression fixture adds an invoice that exists only in account B. Alice's server-constructed context can access account A. A correct read through the gateway returns NotFound for that B-only record and exposes no private invoice content.

The test then runs a known-bad lookup variant that searches by record ID across every account. It finds B's private fixture value. The same privacy contract requires no returned value for this cross-account request, so the deliberately broken result triggers an assertion failure. The test catches that failure to prove that the retained requirement can detect the regression.

A second broken variant removes the write guard and records an effect directly. It fails the zero-forbidden-effect assertion. This is not a proposed production helper; it exists only under test configuration so the checker is exercised against a concrete violation.

The retained Rust cases use synthetic identifiers and a private-data canary. They do not retain a real customer's invoice or claim that this exact incident occurred in the author's deployment. The case captures a plausible failure mechanism and the product requirement that should prevent it.

These are deterministic application checks. They do not estimate how often a language model will propose the bad lookup. Model evaluations can vary prompts, documents, and outputs around the case, but the gateway contract should remain true for every proposal that reaches it.

Keep the requirement, origin, and reason beside the input. If the product later adds legitimate cross-account search for a different role, preserve this ordinary-user case and add an authorized control. Do not delete the refusal merely because a broader feature now exists. A retained counterexample is valuable when it continues to distinguish the intended new behaviour from the old mistake.