Mehdi Akiki
Published on

Claude Code in CI: Tie Each Review to the Code It Checked

Authors
  • Mehdi Akiki avatar
    Name
    Mehdi Akiki
    Twitter

Reference

Claude finishes a CI review. The process exits successfully. The summary says the tests passed.

Then another commit arrives on the pull request.

The review belongs to an older revision. Before acting on it, a reviewer needs to see which code and test results it covers.

Claude Code can inspect changes without an interactive session. Getting it to run is one part of the job. The surrounding code must keep track of what was reviewed, which checks finished, and whether the result is complete enough to show a person.

This guide uses a small Rust example to make those rules explicit. It constructs a Claude command and tests how review results are accepted. It does not launch Claude or evaluate a model's ability to find bugs.

Keep the review separate from the test result

An agent might notice that a Rust handler holds a connection during a remote call. That is a useful finding to investigate. A database test can then check whether another request makes progress while that call is paused.

The two results do different work. The review directs attention. The test records a specific observation.

Do not turn a model-generated tests_passed: true field into a successful CI check. Get that result from the component that ran the tests, through a path the review cannot rewrite.

The skills guide shows how to package a repeatable review. Here, the question is how to handle its output.

Write down which tree each job checked

A pull request can have a head commit and a separate synthetic merge commit. A job may test either. Record the candidate, its base, and the revision actually verified. Describe their relationship instead of assuming every SHA means the same thing. The wrong-commit investigation follows that failure in more detail.

Also consider tests changed by the candidate itself. An implementation and its new test can agree on the wrong behavior. For important requirements, keep some checks in a separately maintained suite that the candidate cannot replace. The example calls this the protected requirement harness.

A harness is still only a collection of checks. It cannot stand in for every business rule. Its own changes need review.

A commit ID also misses uncommitted edits. The runner has to establish which tree ran, including generated inputs. Reporting HEAD after changing files does not prove that the committed tree passed.

Give the review job a small job to do

Start with reading a candidate and returning findings. Keep merge credentials and production access in other jobs.

The Rust source constructs this command:

let mut command = Command::new("claude");
command.args([
    "--bare",
    "--print",
    "Read ci-review/instructions.md and review the candidate files. Return findings for human review.",
    "--model",
    model,
    "--tools",
    "Read,Grep,Glob",
    "--allowedTools",
    "Read,Grep,Glob",
    "--disallowedTools",
    "mcp__*",
    "--max-turns",
    "8",
    "--output-format",
    "json",
    "--json-schema",
    schema,
]);

The runner chooses model. The full example defines schema for a summary and a list of findings. The instruction file would come from the trusted workflow, not from the unreviewed candidate. This code inspects command construction without spawning the process.

The flags have distinct jobs. --tools selects built-in tools. --allowedTools pre-approves their use. That built-in selection does not remove MCP tools, so the command also denies mcp__*. --bare skips automatic loading of the host's project customizations. Check the CLI reference for the version you pin.

Enforce access in the worker too: a candidate snapshot mounted read-only, limited credentials, and a separate output directory. Instructions inside the candidate are text to inspect, not permission to change the review job.

Keep the project's existing runner selection and execution rules. An added review step does not need to move its builds and tests elsewhere.

A zero exit code is not the whole result

A review can stop at a turn limit, lose provider access, hit a tool denial, or return malformed output. Preserve the reason instead of translating all these cases into an empty finding list.

When using a schema, check that the response envelope contains structured_output before parsing it. Anthropic's programmatic usage guide describes that envelope.

The Rust example receives a normalized AgentRun: whether the process succeeded, whether the review completed, and the proposal bytes. A real adapter must derive those fields from its pinned CLI or SDK protocol. That parsing is outside these tests.

Set an overall deadline and an output byte limit as well. Eight turns is not a time budget. A pipe that collects output without a limit can still use unbounded memory. Keep stderr separate, record why a run stopped, and clean up child work when cancelling.

On a retry, keep the candidate identity fixed. A new commit needs a new review. It should not inherit an old attempt's success.

Reject results that belong to another change

Before reading the proposal, the example checks the independently supplied run information:

if evidence.source_sha != candidate {
    return Err("verification belongs to a different source revision");
}
if evidence.harness_sha != protected_harness {
    return Err("verification used a different requirement harness");
}
for required in ["build", "tests", "contract-fixtures"] {
    let matching: Vec<_> = evidence
        .checks
        .iter()
        .filter(|(name, _)| *name == required)
        .collect();
    if matching.len() != 1 || matching[0].1 != CheckState::Passed {
        return Err("required verification is missing, ambiguous, or unsuccessful");
    }
}
if !agent.exit_success || !agent.completed {
    return Err("agent run did not complete successfully");
}

TrustedRun is a stand-in for data collected by a protected CI adapter. These tests do not authenticate GitHub artifacts or implement that adapter.

The three required checks are a policy for this example. Each must appear once and pass. Missing, duplicate, failed, or skipped results make the packet incomplete. Confident review text cannot fill that gap.

Only then does the code parse the proposal. It rejects unknown fields and bounds the summary, finding count, finding sizes, and relative file locations. A model cannot add its own trusted verification fields. These checks keep the result manageable; a valid JSON finding can still be wrong.

Try the failures that would otherwise look successful

The tests include a valid review with no findings and one with a valid finding. Without those positive cases, code that rejects everything could appear correct.

They also try the awkward inputs:

InputExpected behavior
Passing tests for an older commitReject
Tests from a different requirement suiteReject
Confident summary with failed, skipped, missing, or duplicate checksReject
Successful process exit without a completed reviewReject
Proposal containing its own verification fieldsReject
Truncated, oversized, or invalidly located findingsReject

All eight Rust checks passed in the recorded GitHub run. They use synthetic proposals and run information. No model or GitHub API is called.

To evaluate a live reviewer, use patches with known bugs, correct patches, and incomplete information. Record the prompt or skill revision, candidate, requirement suite, CLI and model identities, tool actions, and findings. Count missed bugs and invented findings separately. The local parsing tests cannot answer those questions.

Publish findings from a separate job

Posting a comment is a write operation. Give a separate publisher only the credentials it needs, and pass it the checked result.

Show the reviewed revision beside the findings. Rechecking the pull request head before posting can catch a change, but it cannot make the comment atomic with every future push. An old review should remain clearly tied to its old commit.

Invalid or unfinished output can stay available as an artifact without becoming a public clean review. The Rust function here only prepares an advisory packet. It does not approve or merge a pull request.

For an actual installation, follow the official GitHub Actions guide for your pinned action and runner setup. The example in this article is the result-handling policy, not a complete installation.

If the agent also edits code, identify the proposed tree and run the existing checks against that tree. Keep that editing workspace separate from the immutable candidate used for verification. A failed check can guide another patch; changing its reporting should not make it disappear.

The review is useful when a person can open it and answer three ordinary questions: which code was checked, what actually ran, and what still needs attention. Keep those answers visible even when the generated summary sounds certain.