A webhook response should follow durable acceptance

Article · Interrupted execution

The webhook endpoint puts an event into an in-memory channel and returns success. The process restarts before the worker receives it.

The provider believes delivery succeeded. Your service has no record of the event.

If the response promises acceptance that survives a restart, persist the event before sending that response. The exact acknowledgement contract belongs to the provider, so check it before choosing response codes or retry behaviour.

Store received work as pending

Verify the signature and relevant request limits first. Then persist a scoped event identity, bounded payload or protected payload reference, and a pending state.

This Rust SQLx excerpt stores the verified request body with the receipt. It assumes a payload bytea NOT NULL column and a unique key on the scoped event identity. PostgreSQL's bytea type preserves raw bytes:

sqlx::query(
    "INSERT INTO webhook_receipts
     (tenant_id, provider, event_id, payload_hash, payload, state)
     VALUES ($1, $2, $3, $4, $5, 'pending')
     ON CONFLICT (tenant_id, provider, event_id) DO NOTHING",
)
.bind(tenant_id)
.bind(provider)
.bind(event_id)
.bind(payload_hash)
.bind(payload)
.execute(&mut *tx)
.await?;

tx.commit().await?;

Here, payload is the size-checked, signature-verified body as a byte slice. A hash alone would let the worker recognize an event without giving it anything to process after a restart. Keep the bytes and receipt state in the same transaction. The handler must also compare hashes for an existing identity and reject or quarantine conflicting content before acknowledging it.

PostgreSQL's conflict-handling documentation explains the insert behaviour. A uniqueness check alone does not decide whether an earlier receipt was successfully processed.

Received is not applied

The worker must be able to find pending receipts after a restart. It can claim work using a bounded lease or a transactional strategy appropriate to its store.

When applying a local business effect, keep the applied-event claim and the business mutation in the same transaction. The inbox-pattern article explains that atomic boundary.

Do not reuse a receipt row as a permanent “already processed” marker before the business mutation commits. That would recreate the loss in another form: a durable record exists, but every retry skips the missing effect.

If processing calls another service, give that external effect its own durable intent and duplicate contract. A local receipt does not make several systems commit together.

Enumerate the interruption points

Stop before receipt commit: no success response should have been sent, and provider redelivery should remain useful.

Stop after receipt commit but before response: the provider may redeliver. The duplicate should resolve to the existing receipt without losing its pending work.

Stop after acknowledgement but before processing: the worker should recover the pending receipt from storage.

Stop after business commit but before the worker records its transport-level completion: the applied-event claim should prevent a second committed local effect.

Test these states against stored rows and actual worker recovery. An endpoint response test cannot establish them by itself.

Keep acceptance bounded

Durable storage can fill too. Apply payload, rate, and retention limits, and define what happens when the receipt store is unavailable. Acknowledge only the promise the service can keep.

Some providers expect synchronous processing or have unusual retry rules. Follow that documented contract. For an asynchronous receiver, a small durable acceptance step gives both sides a much clearer handoff than a successful send into process memory.

Interrupt the application transaction, then resume pending work

The larger reproduction validates a signature over the raw request bytes before parsing an event. It limits the request body to 256 bytes. Invalid signatures return 401. A deliberate rollback before durable acceptance returns 503 and leaves zero receipts.

Once acceptance commits, the handler returns 202. The test discards that response object without consuming it, opens a fresh pool, and finds the exact event bytes still marked pending. This models an unobserved acknowledgement at the application boundary; it is not a TCP-disconnect experiment.

A worker locks the pending receipt and updates the business counter within one transaction. A signal pauses it after the provisional changes but before commit. The test aborts that worker task. An independent read still sees counter zero. A new worker can apply the pending event, then replay it safely. The final counter is three, with one applied-event claim and an applied receipt.

A duplicate delivery with the same signed payload is accepted without another application. A changed payload under the same event identity returns 409. Keeping the original bytes and a payload hash makes that conflict observable rather than silently treating different content as the same event.

The complete Rust test demonstrates transactional recovery from task interruption. It does not kill PostgreSQL, restart the operating system, or test storage after power loss. Those stronger claims need a separate fault environment and evidence.

The counter effect is local to the same database transaction. An external mail or payment effect cannot join that transaction simply because the receipt is durable. For external work, retain an outbox or operation record and use the destination's documented replay contract. Durable acceptance is the start of responsibility for recovery, not proof that every downstream action already happened.