Send worker failures back to the caller

Article · Interrupted execution

The request sends a command to a worker and waits for a reply. The worker fails before replying. The handler catches the receive error and returns an empty list.

The user sees “no results.” The real outcome was “the worker did not provide a result.” That difference matters when someone tries to diagnose missing data.

A reply channel needs an error contract just as much as an HTTP endpoint does.

Put the business result inside the reply

Tokio's oneshot channel carries one message. The receiver also reports when the sender disappears without delivering one.

Those are separate layers. A Rust reply can make both visible:

enum JobError {
    InvalidInput,
    UpstreamUnavailable,
}

enum RequestError {
    WorkerGone,
    Job(JobError),
}

async fn receive_reply(
    reply: tokio::sync::oneshot::Receiver<Result<Vec<u8>, JobError>>,
) -> Result<Vec<u8>, RequestError> {
    reply.await
        .map_err(|_| RequestError::WorkerGone)?
        .map_err(RequestError::Job)
}

The worker can return a business failure without crashing. A closed channel means the reply path ended; by itself, it does not explain whether the worker panicked, was cancelled, or deliberately abandoned the command.

The existing oneshot implementation article explains the underlying mechanism. For the service, the useful question is how those states reach the caller and operations team.

Somebody must observe the worker

Keep a task owner or supervisor that joins the worker and records unexpected termination. Otherwise every caller may see WorkerGone while the only useful panic information vanishes with an ignored JoinHandle.

Use a command identity to connect a request failure to a worker failure without copying the entire private command into logs. Keep public responses stable and internal diagnostics appropriately restricted.

If the receiver goes away first, sending the reply can fail too. That does not automatically mean the work should roll back. A write may already be committed. Decide whether the worker stops before starting, completes a durable job, or records an outcome for later lookup.

A reply timeout also means only that no reply arrived in time. It is not proof that the worker performed no side effect. The recovery path depends on the operation.

Check four different endings

The small test set should include a successful reply, a typed job error, a sender dropped before replying, and an unexpectedly terminated worker. Check that no failure becomes a successful empty value.

Add a receiver-disconnect case when work can change state. Inspect whether the write happened and whether its outcome remains observable. This is often where a neat command/reply abstraction needs a durable operation identity.

Not every worker needs a supervisor framework. A task handle owned by a service lifecycle can be enough for a small process. What matters is that its failure has an owner.

An empty list is a valid answer only when the job completed and found nothing. Keeping that meaning precise makes client behaviour, retries, and incident investigation much easier to understand.

Lose the receiver after the write

The runtime database fixture adds the opposite half of the reply failure. The request receiver is dropped, the worker commits an operation row, and sending its result fails. An independent query must still find that committed operation.

A failed send returns the message to the sender. It says the receiver cannot accept that reply. It says nothing about whether PostgreSQL already accepted an earlier statement. The worker can use the returned value for diagnostics or a stored outcome, but it should not manufacture a rollback claim from a channel error.

The lab also checks a request-owned task cancelled after a committed write. Both rows remain visible. These are deliberate interruption points, coordinated by signals and stored-state observations. A sleep followed by a row count would not establish when the caller disappeared relative to the write.

The separate article-derived tests cover successful empty and nonempty replies, a typed InvalidInput error, a dropped sender, and a worker panic. The panic is observed through its task handle as well as the closed reply path. This keeps the public error categories distinct while preserving the information an operator needs.

The database demonstration uses a synthetic operation identity and a simple local write. A real worker that changes customer data needs authorization, input binding, durable outcome lookup, and a retry contract. Its supervisor also needs a shutdown policy. The small typed reply is one part of that design: it preserves the difference between an actual empty result and a result that never arrived.

A worker panic and an invalid business input need different operator responses. Keep the operation reference in both paths so a caller can check an uncertain result without creating another job.