Don't repair an entire integration inside a GET request

Article · Interrupted execution

Opening the customer page discovers that a remote account is missing locally. The handler starts a full synchronization before returning the page.

The first request is slow. The second request discovers the same gap and starts another synchronization. A provider outage now turns ordinary page views into a repair storm.

Read-time repair can be useful, but it needs a small, explicit scope. A GET handler is a poor accidental owner for an entire integration rebuild.

Separate what the read knows from what it fixes

The page may be able to show a known stale value, a pending state, or a clear unavailable result. Pick the product contract rather than quietly blocking until every inconsistency is repaired.

A Rust response model can expose that distinction:

enum RecordView<T> {
    Current(T),
    Stale { value: T, repair_id: String },
    Pending { repair_id: String },
    Unavailable,
}

These states are application choices. A financial balance or access decision may not permit a stale value. An informational profile page may. Do not apply the same fallback to both because the handler shares a helper.

The reconciliation article explains how systems converge after drift. Here the question is who starts and owns that work when a read notices the problem.

Deduplicate the repair request

If a read can request repair, give that request a stable scope: tenant, provider account, resource or partition, and relevant repair kind. Coordinate it in shared storage when several service replicas can discover the same issue.

Do not use only an in-memory flag if the service runs in several processes. Do not permanently mark a repair active without an expiry or recovery policy either.

A worker should claim bounded work, record progress, and expose failure. Repeated page views can refer to the existing repair identity instead of creating new provider traffic.

Keep customer authorization on the read path and the repair command. A guessed external ID should not let a reader schedule work against another customer's integration.

Respect the meaning of GET

HTTP defines safe methods in terms of the client's requested semantics. Incidental caching or bounded maintenance is different from using a read route as the public command to perform a broad business mutation.

If the product needs an explicit resynchronization action, offer an authorized command endpoint with a clear acceptance and outcome contract. That is easier to operate than an expensive side effect hidden behind page refresh.

Test repeated discovery

Hold the provider unavailable and issue several reads for the same missing record. Assert a bounded number of repair jobs, a predictable response, and no unbounded wait in the request handler.

Then let the worker fail and restart. The repair must not remain stuck forever because an old lease survived its owner. Verify that the page eventually reflects the repaired state.

Some small caches deliberately refresh inline, and some API contracts require a current remote read. Keep those requirements. Add a deadline and limit the repair to the operation promised by the route. The practical mistake is letting one missing record turn an ordinary read into an unlimited reconciliation job.

Repeated reads leave the repair queue alone

The HTTP-router fixture calls GET three times while no usable record exists. Each response reports unavailability. The jobs table still contains zero rows. Discovery is not secretly a write operation in this interface.

An authorized POST requests the repair. Repeating that POST three times leaves one durable job because the tenant scope has a uniqueness rule. A request for the other tenant is forbidden. A later GET reports Pending with a repair reference, so the caller can distinguish known queued work from a record that is simply absent.

The worker then claims the job. A signal confirms the claim before the test stops that worker task. The test explicitly expires its lease in the database and starts a new worker through a fresh pool. The new claim gets a later epoch, which is a monotonically increasing ownership number. It completes the repair. An attempted finish using the old epoch changes zero rows.

The final GET returns the repaired value. This makes the whole recovery path observable: discovery, authorized request, durable pending work, replacement worker, and current data. The Rust source uses the shared lease implementation rather than inventing a detached background task in the handler.

The interruption is a task abort with controlled lease expiry. It is not a machine crash, a power-loss test, or a measurement of natural lease timing. The authentication layer is a fixture token registry. A production route still needs its real identity system and authorization policy wired into the same decision.

This design gives repairs a cost and a state the product can explain. It also leaves room for backpressure: a busy repair queue can reject or defer another request without turning an ordinary page read into a long reconciliation operation. Keep GET fast enough to describe the known state, and make the repair request carry its own outcome.