Synchronization, events, schemas, and recovery
Engineering patterns for integrations and distributed data flows that must stay understandable when APIs change, deliveries repeat, and parts of the system fail.
Connecting two systems is easy in the successful case. The durable design appears in the less comfortable cases: duplicate delivery, partial progress, conflicting changes, disappearing records, schema drift, and a replay that happens months later.
This series treats integration as a systems problem. It develops explicit models for identity, ownership, ordering, recovery, and observability, then shows how those models guide implementation.
01
A synchronization flow needs explicit answers about which records correspond and which system may change each field.
02
Retries, idempotency, checkpoints, tombstones, and reconciliation are normal operation, not exceptional add-ons.
03
Stable internal models contain provider-specific schemas and keep external change from spreading through the product.
Entry points into this subject, ordered as a reading path.
01
When you sync two systems with webhooks, the copy slowly becomes wrong: some events are lost, some are duplicated, some arrive in the wrong order. Reconciliation is the way to make the two systems agree again. Here is how to design an API for it.
02
A checkpoint is a claim about durable effects, not the last page fetched. Compare commit protocols and crash points so recovery repeats work instead of losing it.
03
What actually happens between bytes arriving on a socket and your gRPC handler running? This article walks one real request through every parsing layer — TCP, HTTP/2 frames, HPACK, gRPC envelope, protobuf — so the magic disappears.
04
A tutor-style walkthrough of what a distributed database control plane actually is, why metadata is the crown jewel, where Raft fits in, and how to follow one request end-to-end to remember the whole design.
Derived state
Concrete simulations of the records a timestamp cursor can miss, plus safer tuple cursors, overlap windows, change logs, and reconciliation.
Derived state
A state-machine walkthrough for joining a full snapshot with live changes without gaps, stale overwrites, or false deletions.
Derived state
An exact overlap-window synchronization algorithm with tuple ordering, idempotent application, atomic checkpoints, and repair for unbounded lateness.
Derived state
Mutation simulations showing how offset, keyset, and cursor pagination behave when records are inserted, deleted, or updated during a long API scan.
Derived state
A dual-read timeline for importing historical data while live updates continue, without gaps or stale snapshot rows overwriting newer events.
Derived state
A correction-first design for late data using event time, watermarks, affected-window replay, versioned aggregates, and explicit finality.
Derived state
A concrete schema and concurrency model for mapping external IDs to internal entities without duplicates, cross-tenant collisions, or unsafe retries.
Derived state
A practical comparison of conflict policies that preserve business intention when two synchronized systems edit the same entity.
Derived state
A practical field-ownership matrix for deciding which system may write each value before loops, silent overwrites, and deletion conflicts enter a bidirectional sync.
Derived state
A lossiness ledger and test strategy for keeping a useful canonical model without silently erasing provider-specific meaning.
Derived state
Schema validation catches only part of provider drift. Test the fields, behaviours, and semantic mappings your consumer actually depends on, then verify them safely.
Derived state
An adapter contract and test suite for translating provider pagination, identity, nulls, versions, errors, retries, deletion, and capabilities—not only field names.
Derived state
Third-party APIs should not become your product model. Learn how a stable internal representation contains provider changes and keeps integrations understandable.
Derived state
API schema changes are not only field changes. Separate source contracts from product meaning, preserve evidence, and evolve mappings without spreading breakage.
Interrupted execution
A paginated import should resume from a committed cursor and stable item identities, not from a human page number that may no longer describe the same data.
Interrupted execution
A concurrent refresh design using local single-flight work, database versions, conditional writes, and recovery rules for providers that rotate refresh tokens.
Interrupted execution
A crash-safe checkpoint design for incremental API synchronization, including atomic progress, cursor invalidation, poison records, leases, and full-sync recovery.
Interrupted execution
A deduplication TTL must cover every supported path by which an old message can return. Derive the window from retries, replays, outages, and redrives.
Interrupted execution
A database inbox turns at-least-once delivery into one committed business effect by claiming message identity in the same transaction as domain changes.
Interrupted execution
Large API batches reduce request overhead until one bad record makes good records fail with it. A measured way to find the useful batch size and retry unit.
Interrupted execution
A bounded API retry policy that combines error classification, Retry-After, full-jitter backoff, operation deadlines, idempotency, and a shared retry budget.
Interrupted execution
An SFTP folder is not a queue. Reliable imports need a ready-file contract, stable file identity, durable claims, idempotent processing, and reconciliation.
Interrupted execution
A decision and failure matrix for combining low-latency webhooks with polling and reconciliation so an integration can recover from missed, duplicate, or reordered events.
Interrupted execution
Webhook verification middleware and tests that authenticate the exact request bytes, reject stale signed deliveries, support key rotation, and deduplicate effects durably.
Interrupted execution
A synchronization run rarely fails as one clean unit. Define progress, safe retries, checkpoints, and recovery so partial success becomes an ordinary state.
Interrupted execution
A useful fake provider controls time, pagination, throttling, malformed data, and ambiguous writes. Test the integration protocol, not only JSON decoding.
Interrupted execution
A measured additive-increase, multiplicative-decrease controller for finding useful API concurrency while respecting throttling, latency, retries, and tenant isolation.
Interrupted execution
When all tenants consume one provider quota, local retries and per-worker limits are not enough. Use a global budget with fair tenant admission and feedback.
Older, introductory or narrower pieces on the same subject.
Engineering context
I design data and integration systems around explicit contracts, observable failure, safe recovery, and the realities of changing external APIs.