How to Review AI Generated Tests Against the Requirement

Article · Measurement

Review AI generated tests against an agreed input and output before trusting the green result. A test can agree perfectly with the implementation and still check the wrong product rule. The useful question is what happens at the boundary where a user would notice the mistake.

Imagine a trial feature with a fourteen-day duration. Someone asks an agent to check whether a trial is active. The proposed code accepts the exact expiry instant, and the proposed test expects that instant to be accepted too.

Everything passes. Then the reviewer asks whether “expires at noon” means noon is still allowed.

This is a constructed review example, not a transcript of a model run. It differs from the contact-import prompting example: here the problem is a time interval, and the test must settle what its endpoint means.

Write the rule without looking at the implementation

For this example, the product decision is an elapsed duration of fourteen periods of twenty-four hours. Access starts at activation and stops at the expiry instant. The interval includes its beginning and excludes its end. The trusted service supplies both timestamps as integer milliseconds.

That decision is not the only possible trial policy. A product could instead promise calendar days in a named timezone or access through the end of a billing date. Those policies need different examples, especially around clock changes. Do not copy an elapsed-milliseconds calculation into them because the function name sounds suitable.

Here is the agreed contract for a trial starting at noon UTC on October 1:

The reviewer should be able to discuss that table with the product owner before reading a comparison operator. It is small enough to expose the decision without hiding it inside test helpers.

Also settle where time comes from. A browser clock supplied by the user is not an authority for granting access. Passing a clock into a function makes testing convenient; the caller still needs to obtain the real decision time from a trusted service.

Two matching mistakes can produce a green run

The faulty example is short:

const durationMs = 14 * 24 * 60 * 60 * 1000;

function inclusiveTrial({ startedAtMs, nowMs }) {
  return nowMs >= startedAtMs && nowMs <= startedAtMs + durationMs;
}

A test written from this code might assert true at the expiry instant. That assertion describes what the function does. It does not establish that the function implements the agreed interval.

This is why asking the same agent to “add tests” after implementation can leave a gap. It may work backwards from the code's choices. A person can make the same mistake. Separate the expected policy from the implementation before either becomes the convenient source of truth.

The independently specified case is:

const startedAtMs = Date.parse("2026-10-01T12:00:00Z");
const expiresAtMs = Date.parse("2026-10-15T12:00:00Z");

assert.equal(canUseTrial({ startedAtMs, nowMs: expiresAtMs }), false);

The expected boolean comes from the contract table. It is not computed by calling the function under test or copying its operator into another helper. The Node assertion API checks that expected result; it cannot decide which policy the product should have.

Repair the comparison and keep the boundary cases

With the policy settled, the core repair is an exclusive upper bound:

return nowMs >= startedAtMs && nowMs < expiresAtMs;

The complete fixture also rejects missing, fractional, non-finite, and overflowing timestamps. Those checks keep malformed input from quietly participating in comparisons. They do not replace authentication, account state, or other product conditions required by the real endpoint.

The trial policy fixture contains the faulty comparison, the repaired function, and independently listed boundary expectations. Its negative comparison explicitly distinguishes the two functions at expiry. That is more useful than a coverage percentage that both versions could achieve.

A useful CI check must reject the faulty boundary while accepting the agreed cases in the repaired function. Before remote verification, those are intended assertions, not recorded passing results. Keep the run link with the source revision when verification is available.

The example has no timer or sleep. Each test supplies a fixed instant, so waiting for a real fortnight is unnecessary. The runtime's ordinary numeric comparison is not the uncertainty here; the chosen interval is.

Give the reviewer a decision they can inspect

In the review description, state the duration policy, its timezone assumptions, and the exact expiry example. Then show which assertion changes when the comparison is wrong. A reviewer can question the requirement before approving its implementation.

Do not turn this into a demand that every tiny change needs a large specification. A label correction needs much less. A time boundary deserves these few examples because a single character changes which requests are accepted.

If the requirement later changes to calendar days, update the contract and tests together, explaining the new examples. Quietly editing the expected boolean until CI passes would hide a product change as a test repair.

When a test fails only sometimes, the investigation is different: establish whether completion, shared state, or application behavior varies. The Playwright repair keeps behavioral assertions while fixing a waiting problem. Here, waiting longer would never resolve the disagreement. The team has to decide what expiry means.