Choose one layer to own retries

Article · Interrupted execution

The worker allows three attempts. The SDK allows three attempts. The HTTP transport also retries.

During an outage, a single job can produce many more calls than the person configuring the worker expected. Each layer looks polite on its own. Together they keep asking the same failing service for more work.

The first repair is an inventory. Count actual network attempts, not the retry setting nearest the business code.

Find the hidden retry owners

Read the client's documented defaults. Check middleware, redirects where relevant, token refresh, connection recovery, and the job runner's redelivery policy.

Some transport retries happen before a request is known to have reached the server. Others repeat an application operation whose outcome is uncertain. They are not interchangeable.

For one logical operation, choose the layer that understands enough context to decide whether another attempt is safe. Disable or constrain redundant policies where the SDK allows it. If layers must retry independently, make their combined ceiling explicit.

The retry-budget article covers backoff and deadlines. The practical addition here is making every attempt owner visible in one trace.

Carry an operation deadline

A Rust budget can make the two basic limits obvious:

struct AttemptBudget {
    remaining: u32,
    deadline: std::time::Instant,
}

impl AttemptBudget {
    fn claim(&mut self, now: std::time::Instant) -> bool {
        if now >= self.deadline || self.remaining == 0 {
            return false;
        }
        self.remaining -= 1;
        true
    }
}

This single-owner helper is not a distributed ledger. A concurrent or durable workflow needs coordinated accounting and a deadline representation suitable for crossing processes. A monotonic Instant cannot be serialized as a universal timestamp.

Claim an attempt before starting it. Include the next wait and request timeout in the remaining time decision. A Retry-After value beyond the deadline is a reason to stop or defer durably, not to extend the customer's deadline silently.

Keep repeated writes recoverable

HTTP's idempotent-method discussion explains why automatic repetition needs care. For a non-idempotent application write, use the provider's documented duplicate and outcome contract.

A stable operation identity should survive attempts. Regenerating the key for each retry can turn recovery into several separate writes. A timeout after transmission may require lookup rather than another immediate call.

Count attempts in the test transport

Return a controlled sequence of failures and record every invocation. Run the old nested policy to show its effective ceiling, then the chosen policy to show the intended one.

Add cancellation, an exhausted budget, and a retry delay longer than the remaining deadline. Assert that no further call starts. Use controlled time for the policy and separate integration checks for real socket behaviour.

The error handling can stay in several functions. The workflow still needs one clear rule for trying again. Record the effective attempt limit so the next engineer does not have to multiply three settings by hand during an incident.

Nine HTTP attempts versus one three-attempt budget

The retry reproduction starts a real loopback HTTP provider backed by PostgreSQL. Its retry endpoint commits an idempotent order and then responds with 503. The response invites a retry even though the business effect may already exist.

The first client has two nested loops, each allowing three attempts. The provider counts nine HTTP calls. This is a controlled representation of two retry owners, not a measurement of any SDK's defaults. It makes the multiplication visible without guessing which library retried behind the handler.

The bounded client has one operation budget with three remaining claims and a deadline. It sends three calls. A budget already at its deadline sends zero. Both paths keep the same operation identifier for the write. Repeating the request cannot create another order merely because the client decided to try again.

Cancellation is tested at a precise boundary. A one-shot signal announces that the client reached backoff. The test stops the task there and checks that no additional provider attempt occurs. The fixture uses a small deterministic delay sequence so its scheduling is repeatable; it is not a random-jitter implementation or evidence of retry behaviour under fleet-wide contention.

The example's provider always returns the controlled retry response on this route. A production policy also needs to distinguish successful responses, permanent rejection, retryable transport failure, and an unresolved write. It should stop when the outcome is known, not spend every available attempt by default.

Count at the network boundary when auditing a real stack. An application counter before an SDK call can miss retries made inside that call. Keep the attempt budget, remaining time, and stable write identity together. That gives the operation one owner even when transport details change beneath it.