Posts tagged "what survives a crash"

I tested my key-value store and four versions with one bug each, using kill -9, fault injection, crash simulation and damaged files. Each method found some bugs and missed others, and one bug was invisible to all three crash methods. The table, the real tools, and a look back.

Two files saved safely one after the other are not saved safely together: a kill left a new config with old data 85 times in 300. A version directory and one symlink rename, or renameat2 with RENAME_EXCHANGE, switched both files at once in every test. LevelDB does the same with CURRENT.

I flipped one bit in four files. My log's CRC caught it and then threw away nine good records, SQLite returned the changed value and its integrity check said ok, gzip reported a CRC error, and the text file changed silently. Where checksums belong, and what ext4 checks itself.

I changed 64 MiB of a file without one write call, through a shared memory mapping. The pages became dirty like in the first part, kill -9 kept them, msync or fsync made them durable, and a truncated file killed my program with SIGBUS. The kernel decides the rest.

With every writer syncing alone, my laptop made about 1,800 records per second durable, with 1 or 16 writers. With group commit, one fdatasync for all waiting records, 16 writers reached 15,822. A small delay before the sync helped 16 writers and hurt one, as PostgreSQL documents.