Posts tagged "what survives a crash"

A log needs a checkpoint, and the naive one loses data: on a plain restart it dropped two acknowledged operations, and after a crash it counted one twice. Log segments, a durable snapshot and a strict order fix it, and my simulator checks every crash point.

I built a tiny key-value store in Rust with a write-ahead log: append, fdatasync, apply, and only then say ok. My crash simulator checks every crash point, and it finds the bug that kill -9 never shows: an ok printed one fdatasync too early.

A log of length, CRC-32 and payload, and a recovery that stops at the first bad record. I tested it with kill -9, with my crash simulator and with a fuzzer that cuts the file at every byte. One small format detail decided if 17 records came back as 17 or as 529.

kill -9 could not tell a save with fsync from a save without it. So I wrote a small Python simulator that replays the strace output of a program and lists what a machine crash could leave at every moment. It found the zero-length file and the missing directory fsync.

One write() call of 1 GiB, killed after 50 ms, left a file of 110 MB. I measured what a single call leaves after kill -9, what another process sees during it, and when appends from four processes mix. Power loss is worse, and I explain why from documentation.