Mehdi Akiki
Published on

Tombstones: How to Delete Data Across Systems

Authors
  • Mehdi Akiki avatar
    Name
    Mehdi Akiki
    Twitter

Reference · Derived state

You delete a customer in system A. The row is gone. Clean.

But system B has a copy. How does B learn about the delete? If A sends an event and B misses it, the customer stays in B forever. Nobody notices, because nothing points to a row that is not there.

This is the quiet danger of a real delete: it leaves no trace, so it cannot be observed.

"Absent" means too many things

Think about what a missing row tells you. If B asks A for customer 42 and gets nothing, what does it mean?

  • the customer was deleted, or
  • the customer never existed, or
  • the customer exists but this read failed.

A hard delete throws away the one thing B needs: the fact that this id was deleted on purpose. After that, B cannot tell a real delete from a bug.

The fix: do not delete, mark as deleted

A tombstone is a row that stays, but is marked dead. Instead of removing the record, you keep its id and add a flag and a time:

UPDATE customers
SET deleted_at = now()
WHERE id = 42;

Now the delete is a normal change, like any update. It has an id and a timestamp. Other systems can see it, sync it, and agree on it. A delete becomes something you can observe, not something that silently disappears.

Reads in A must then hide the dead rows:

SELECT * FROM customers WHERE deleted_at IS NULL;

And a list built for syncing should include the tombstones, so the other side learns what to remove.

The tradeoff

Tombstones do not stay free forever. They keep dead rows in your table, so you need a retention rule: after some safe time, when you are sure every consumer has seen the delete, you remove them for real.

You also have to remember to filter them everywhere, or deleted customers will come back to life in your queries.

This is the price of a delete that other systems can trust. If you want to keep two systems in sync, a delete that nobody can see is exactly the thing that creates ghost rows that live forever.