Measurement
This site as an engineering laboratory
Kept, rejected and upstream results from more than fifty experiments on this site.
Evidence: experiments/
The main body of this writing follows one system below its interface: a compiler, a runtime, a protocol, a build, a data pipeline. It is organized around four engineering questions.
Investigations run over several parts and keep the programs and experiments that produced their evidence. Articles stand alone. Shorter notes, beginner material and tool guides stay in the archive and in search.
Multi-part work backed by code, measurements or reproductions that can be run again.
Measurement
Kept, rejected and upstream results from more than fifty experiments on this site.
Evidence: experiments/
Through the layers · 10 parts
Ten parts on what a type is and where it goes: sets of values, layouts and padding, erasure and monomorphization, the bits the processor actually sees, and what the checker knows that the binary forgets. Small programs in Rust, C, TypeScript and Python, with one script that reproduces every output.
Evidence: experiments/rust-atlas/types-under-the-hood/
Derived state · 3 parts
A measured diagnostic tree for slow Rust builds: cargo --timings for the shape, the fingerprint log for unexpected rebuilds, -Ztime-passes for the phase inside one crate, and what splitting a workspace into more crates actually changes.
Evidence: experiments/rust-atlas/build-times/
Interrupted execution · 3 parts
Five async Rust problems reproduced in one small tokio program: an oversized future, a future that is not Send, a blocking call, a select! loop that loses data, and a shutdown that hangs. Then the cost of async fn through dyn Trait, and cleanup when Drop cannot await.
Evidence: experiments/rust-atlas/async-symptoms/
Through the layers
Rust failures that are difficult to name and easy to misdiagnose, organized from symptom to mechanism, with a failing and a repaired fixture for most cases.
Through the layers · 20 parts
Twenty parts that follow small C and Rust programs from the linker to the exit of the process: ELF files, relocations, static and dynamic linking, the loader, the PLT and the GOT, LD_PRELOAD, the code before main, the first stack, the memory map, page faults, fork, malloc, system calls, stack overflows and thread-local storage. One script reproduces every output.
Evidence: experiments/from-source-to-process/
Interrupted execution · 20 parts
Twenty parts on what really reaches the disk: write and the page cache, the cost of fsync, saving a file with rename, the directory fsync, torn writes, a crash simulator built from strace output, a small write-ahead log with checkpoints and recovery, SQLite journal and WAL modes, group commit, mmap, checksums and crash testing. Small programs in Rust, Python and C, with one script that reproduces every output.
Evidence: experiments/what-survives-a-crash/
Measurement · 20 parts
Twenty ways a performance number goes wrong, each reproduced on one laptop: a loop the compiler deleted, debug builds, warm-up and the JIT, CPU frequency and efficiency cores, averages and coordinated omission, clocks, run counts, blocks against interleaved runs, environment size, code alignment, caches, branch prediction, the allocator, and a microbenchmark that lost inside the real program. One script reproduces every output.
Evidence: experiments/when-the-benchmark-lies/
Through the layers · 10 parts
Ten parts that follow one TCP connection on loopback, from connect() to a connection pool: the handshake, the accept queue, Nagle and delayed ACK, the byte stream, full socket buffers, what close() sends, TIME_WAIT, getaddrinfo and the cost of TLS. Only what ss, the kernel counters, timings and system calls show, with one script that reproduces every output.
Evidence: experiments/following-a-tcp-connection/
Derived state · 10 parts
Ten parts on what an index really is, measured on a SQLite table of one million rows: a second sorted copy of the data, SCAN against SEARCH, the cost on every insert, column order, covering indexes, the cases where the planner ignores an index, statistics that go stale, partial and expression indexes, free pages and VACUUM, and a search index kept in sync.
Evidence: experiments/inside-a-database-index/
Standalone pieces, grouped by the question they answer.
Through the layers
Source, runtime, protocol, compiler, machine. Each piece follows one behaviour down to the layer that decides it.
Investigations: Types under the hood, Async Rust under the hood, Rust Failure Atlas, From source to a running process, Following a TCP connection
Review database permissions, tenant identity, query time and response size in a Rust MCP tool. Use a real restricted PostgreSQL role to check the boundaries.
A one-tenant fixture cannot show cross-tenant access bugs. Exercise reads, writes, and reused database connections with two independent customer scopes.
Give a reporting service narrow database permissions and test denied writes using its real account. Read-only access still needs data and resource limits.
Keep network waits outside a shared-state lock, then check that the state is still current before applying the result. Shorter locks need a correctness policy too.
A bounded worker queue needs an admission policy. Choose between waiting, rejecting, and durable acceptance, and tell callers which promise they received.
Derived state
What gets stored, what gets recomputed, what becomes stale, and how the system knows.
Investigations: Rust build times, Inside a database index
A run timestamp makes every dependency cache key different. Key reusable inputs deliberately and let Cargo validate artifacts without treating a cache hit as correctness evidence.
A compiler upgrade and a dependency update are two changes. Compare compatible toolchain and lockfile combinations before assigning a build regression to rustc.
A generator can appear correct because old files remain in OUT_DIR. Check fresh and reused builds, declare inputs, and own cleanup of obsolete generated files.
Evaluate a crate boundary around a generated Rust client using representative edits. Compare incremental reuse with clean-build and generic-code costs before splitting the workspace.
Translate provider states at the integration boundary. Give the product an explicit unknown state instead of scattering external status checks through its interface.
Interrupted execution
Crashes, retries, cancellation, partial progress, ownership, recovery and idempotency.
Investigations: Async Rust under the hood, What survives a crash
A slow shipping request should not hold stock locks. Commit a pending order and durable delivery intent, then recover the external step explicitly.
Two webhook handlers can both pass an existence check. Use a unique database claim, bind it to the payload, and commit it with the business change.
Separate schema ownership from normal application access. Test that the runtime account can do its work and cannot change the tables or security policies.
Follow a checkout result through the commit and HTTP response. Handle rollback and a lost commit response as different outcomes.
Keep an interactive SQLite importer resumable and bounded. Commit batches with their progress and give foreground writes an explicit busy policy.
Measurement
Treating an assumption as a hypothesis, building an experiment, following a surprising result into a lower layer, and rejecting a change when the evidence says it loses.
Investigations: This site as an engineering laboratory, Rust build times, When the benchmark lies
Your tenant has enough documents, but an HNSW query returns too few. Compare real plans, iterative scans, and an exact fallback without weakening access checks.
Evaluate a PostgreSQL status index against both its read benefit and update work. Use HOT statistics as evidence about the mechanism, not as a speed score.
Capture a small regression fixture before changing code or asking an agent for a fix. Prove that the test detects the original failure and preserves the real requirement.
An is_err assertion can pass because the database is unreachable. Check the intended error category and the stored outcome so unrelated failures cannot make a negative test green.
More concurrent calls can raise throughput while making slow requests worse. Compare queue time, errors, and latency distributions under the same offered workload.
Find the delay, preserve the test, and connect each CI result to the code it checked.
10 published articles
Small investigations into slow requests, runtime changes, browser work, and data contracts.
10 published articles
Follow the file, process, network, permission, and runtime boundaries around an agent.
8 published articles
Use PostgreSQL, MySQL, SQLite, MariaDB, SQL Server, Oracle, Db2 and DuckDB from Rust. Work through transactions, locks, types and query plans.
30 published articles
Follow failed requests, shutdowns, traces, migrations, sessions, uploads and job leases through small Rust examples.
8 published articles
Follow request waits, uncertain writes, row streams, memory budgets and deployment dependencies through reproducible Rust examples.
7 published articles
Follow duplicate requests, external work, commit failures and import recovery through complete Rust reproductions.
6 published articles
Check index write costs, tenant isolation, and database permissions.
3 published articles
Trace shared locks, queue admission, task ownership, retained memory, and cleanup.
9 published articles
Keep failing inputs, diagnose build inputs, and compare performance with observable controls.
10 published articles
Keep provider outcomes, pagination, caches, durable acceptance, and repairs explicit.
10 published articles
Carry identity, permissions, approval, and resource limits to the place effects happen.
12 published articles
All 554 published pieces, newest first, each labelled as an investigation, an article or reference. Reference covers beginner Rust, framework and tool guides, AI explainers, interview preparation and older short pieces. The 61 short engineering notes have their own page.