GitHub Actions continue-on-error and Visible Failed Checks

Article · Measurement

GitHub Actions continue-on-error can let a job continue after a step fails. Its final success does not mean that step passed. Read the original outcome, make the exception visible, and record when normal blocking behavior must return.

Suppose a repository has an integration check that has been unreliable for months. A note says it is “usually infrastructure.” One pull request merged after a rerun. Another added continue-on-error while someone investigated.

An agent reads those notes and sees a familiar failure on its own patch. It reports that the implementation is complete and dismisses the remaining failure as known noise.

This time the application is broken.

The agent is following the repository's notes and examples. The failed run hasn't retrained the model. Those notes have simply made ignoring this check look like the normal way to finish a task.

Compare the failures before dismissing one

“Known failure” can mean a reproduced bug with an owner, a suspected outage, or an error somebody remembers from last week. A tracked bug gives you more to work with than a remembered error message.

An agent needs enough detail to tell them apart. Which assertion failed? On what commit? What caused it? What would make today's failure different?

A note saying “usually infrastructure” doesn't answer those questions. This matters especially when the patch changes the code the check covers. Last week's network outage doesn't establish why today's request timed out.

Compare the failure with the linked issue. The error may look familiar while the cause is different. If the comparison leaves doubt, report that doubt and keep investigating.

Nobody in the repository set out to create a permanent bypass. Each exception seemed reasonable at the time. Together they made bypassing the check look normal, and the agent copied that habit.

A clearer note would name the failing assertion, link the confirmed issue, and state when the exception applies. That gives the next developer or agent something specific to compare with their own run.

Read the failed step as well as the badge

Consider this workflow fragment:

- name: Check import preview
  id: preview
  continue-on-error: true
  run: node scripts/check-import-preview.mjs

The job can continue even when this step fails. GitHub's contexts reference distinguishes the step's outcome before continue-on-error from its conclusion afterward. A failed step can have a failure outcome and a success conclusion.

A summary that only reports the final badge can hide that failed check. If the step is intentionally allowed to fail, say so and explain what it checks. If it once protected required behavior, making it optional changed what the workflow enforces.

A later step can make the original result visible:

- name: Report preview result
  if: ${{ always() }}
  env:
    PREVIEW_OUTCOME: ${{ steps.preview.outcome }}
  run: |
    printf 'Import preview outcome: %s\n' "$PREVIEW_OUTCOME" >> "$GITHUB_STEP_SUMMARY"

This reports the result only. It doesn't restore a blocking check. Keep that enforcement decision separate from the summary text.

“Fixed CI” is a poor description of a patch that only stops a failure from blocking the job. The agent should name the behavior that has lost protection.

Some checks can reasonably be optional. An experimental diagnostic may help an investigation without needing to stop every merge. Its name and result should make that role clear, so a reviewer doesn't mistake it for a required test that passed.

Give temporary exceptions an end

If a dependency service is down and blocking unrelated work, a temporary exception may be reasonable. Name the affected check, record the gap in coverage, give someone responsibility for it, and say what will end the exception.

“Temporary” alone doesn't bring a check back. A date, linked issue, or clear restoration condition gives the owner something to act on.

A record might look like this. The issue number, owner, dates, and failure description are constructed examples, not an active exception for this site:

check: import-preview-service
issue: "example issue 123: fixture service rejects connections"
owner: "the import service maintainer"
expiresAt: "2026-11-06T17:00:00Z"
restorationCondition: "fixture service is healthy and the original assertion passes"
requiredEvidence:
  - "same service endpoint and confirmed connection-refused failure"
  - "current checked revision and failed step log"
excludedFailures:
  - "wrong duplicate rows after a successful response"

The excluded case matters. A familiar check name does not make every failure eligible. The record requires evidence for the specific outage, while a successful request with incorrect rows still needs an application repair.

This YAML is documentation, not automatic enforcement. If the team wants an expired exception to fail CI, its workflow needs to read and validate the record. Until that exists, the owner and review process must restore the check. A job summary does not enforce a date merely because it displays one.

For example, a service outage might justify an exception until Friday. On Friday, the owner checks the service and restores the test if the cause is fixed. If the outage continues, extending the exception needs another decision and a new review date. Leaving the original note untouched would hide how long the team has gone without that coverage.

Keep the underlying failure visible while the exception exists. Put it in the job summary and retain the trace or other diagnostic files. The next reviewer should be able to see that the exception is active and why.

The agent should also explain why today's failure falls under that exception. A previous merge is not enough. The symptoms and affected check need to match the issue well enough to justify the decision, with any remaining uncertainty stated plainly.

Make unresolved work easy to report

The team replaces the broad infrastructure note with a clearer instruction:

Report failed required checks as unresolved.
For an existing exception, link its issue and explain why it applies.
Do not weaken an assertion or change its blocking status as a repair
without describing the behavior that would lose protection.

That lets the agent report a real infrastructure problem without calling every optional warning a release blocker. It also makes changing an assertion's status a decision that someone can review.

If today's failure is a preview regression, repair the behavior and restore the check's blocking status. The next handoff should name what passed, what failed, and any exception still in use.

Replace the old “usually infrastructure” note with the linked issue and the conditions under which the exception applies. The next developer or agent can then compare a failure with something specific. Repairing the Playwright waiting problem is a different task. An exception can record the gap while that investigation continues; it cannot establish that the application works.