This site as an engineering laboratory
Kept, rejected and upstream results from more than fifty experiments on this site.
Evidence: experiments/
Treating an assumption as a hypothesis, building an experiment, following a surprising result into a lower layer, and rejecting a change when the evidence says it loses.
Kept, rejected and upstream results from more than fifty experiments on this site.
Evidence: experiments/
3 parts
A measured diagnostic tree for slow Rust builds: cargo --timings for the shape, the fingerprint log for unexpected rebuilds, -Ztime-passes for the phase inside one crate, and what splitting a workspace into more crates actually changes.
Evidence: experiments/rust-atlas/build-times/
20 parts
Twenty ways a performance number goes wrong, each reproduced on one laptop: a loop the compiler deleted, debug builds, warm-up and the JIT, CPU frequency and efficiency cores, averages and coordinated omission, clocks, run counts, blocks against interleaved runs, environment size, code alignment, caches, branch prediction, the allocator, and a microbenchmark that lost inside the real program. One script reproduces every output.
Evidence: experiments/when-the-benchmark-lies/
Your tenant has enough documents, but an HNSW query returns too few. Compare real plans, iterative scans, and an exact fallback without weakening access checks.
Evaluate a PostgreSQL status index against both its read benefit and update work. Use HOT statistics as evidence about the mechanism, not as a speed score.
Capture a small regression fixture before changing code or asking an agent for a fix. Prove that the test detects the original failure and preserves the real requirement.
An is_err assertion can pass because the database is unreachable. Check the intended error category and the stored outcome so unrelated failures cannot make a negative test green.
More concurrent calls can raise throughput while making slow requests worse. Compare queue time, errors, and latency distributions under the same offered workload.
An optimization comparison needs equivalent work. Hold inputs steady, separate implementation changes, and investigate interactions after establishing a trustworthy baseline.
Keep reproducible failures in a versioned evaluation set. Retire obsolete cases with a recorded reason, and separate requirement changes from attempts to improve a release score.
Compare important workflows as well as the overall evaluation result. Report case counts and uncertainty so an aggregate gain cannot hide a smaller critical regression.
Follow an Axum request into a spawned SQLx lookup. Capture the missing span parent, then carry it explicitly with tracing::Instrument.
Capture actual SQL Server plans from Rust with bound parameters, then separate parameter-sensitive behavior from a slow query that only looks identical.
Generate a multi-row-group Parquet file, query a narrow range from Rust, and separate a pushed-down filter from measured I/O savings.
Run a bounded Rust sorting query with explicit memory and scratch budgets, and report observed spill instead of assuming it.
Ship readiness for an AI feature is one artifact per failure class, each with a stated minimum bar and a stated way it can be invalidated. Built here with a runnable 300 item fixture.
A simulation of a real 5 point improvement showing how often a fixed eval threshold gives a random red build, and why a paired comparison detects the gain at a much smaller sample size.
A working record-and-replay harness for agent tests: a JSON cassette of decisions and tool results, deterministic replay with zero live calls, and a readable diff when the trajectory changes.
A trajectory grader for detecting unauthorized reads, duplicate effects, fabricated tool results, waste, and unsafe recovery even when an AI agent's final answer looks correct.
A practical agreement experiment for measuring an LLM judge against human labels, including ambiguous cases, position bias, confidence, and release thresholds.
A retrieval benchmark for separating missing evidence from generation failures using Recall@k, Precision@k, MRR, unanswerable questions, and controlled oracle context.
A versioned evaluation matrix for separating prompt, model, retrieval, tool, and grader changes while protecting safety, quality, latency, and cost.
A graded assertion approach for regression-testing variable AI outputs without comparing every generated word to one frozen answer.
A measurement architecture that joins repeatable offline evals with production outcomes, safety signals, canaries, and privacy-aware review.
A practical AI failure taxonomy that separates what happened, where it happened, its impact, and the suspected cause so teams can build useful metrics.
A purely random sample hides rare and expensive AI failures. Combine baseline, risk-triggered, and exploratory samples without corrupting the denominator.
A strong system prompt cannot secure retrieval, tools, authorization, retries, or side effects. Red-team the complete AI workflow and its recovery paths.
A practical pipeline for learning from production AI failures while minimizing personal data, preserving the failure signal, and keeping evaluation fixtures reviewable.
Use a recorded GitHub Actions run to separate setup, early feedback, and the full build before changing a slow CI pipeline.
A trial-expiry boundary shows how code and generated tests can share the wrong assumption. Write the expected result independently.
Replace fixed sleeps with completion and row-identity assertions, then check a delayed preview and a deliberately wrong result in CI.
Record PR head, checked-out source, event, and workflow URL together so a valid test-merge result is not confused with stale verification.
Separate first-review waiting, review windows, author rework, and dependency waits before asking agents to open more pull requests.
Show a tolerated step failure in the job summary and record the owner, evidence, expiry, and restoration condition of its exception.
Compare a real PostgreSQL plan before and after an ordered index. Follow the rows below LIMIT, check estimates, and decide which work an index would avoid.
Separate PostgreSQL connection acquisition from query time with OpenTelemetry spans, then investigate queueing or connection setup before changing SQL.
Build a small PostgreSQL search trial with exact identifiers, full-text search, pgvector, and lifecycle checks before choosing another database.
I measured rtk, a token-saving tool, inside my Claude Code setup. It gave two wrong answers, and one is still there after the update. Its hook also approved a force push without asking me, and its own savings number fell from 125.5M to 4.8M tokens.
My site's AGENTS.md adds 1,382 tokens to every Claude Code request. I measured each part, labeled every line, and tested a slimmer version. It dropped a line that I thought was useless, and the agent tried to change the commit author.
I sent the same 200 bank support messages through keyword rules, search, three Claude models and an agent, and measured accuracy, cost per correct answer and time. The cheapest good result came from a small model that was shown similar labeled examples.
I turned measured token counts from a real classification test into a monthly cost range at 10,000 and 100,000 messages. The model bill went from cents to a few hundred dollars. Under my stated assumptions, the human time for wrong answers was far bigger.
I wrote one Claude Code skill with a vague description and a specific one, then ran 60 constructed prompts. With Haiku, the vague description loaded the skill for 18 of 30 requests that needed it, the specific one for 28. Neither loaded it for a near miss.
I built a short Claude Code skill from the Rust Failure Atlas and tested it on 30 Atlas cases with Haiku. Every fix compiled and ran, with and without the skill. In one run per case, 11 fixes with the skill changed what the program was meant to check, against 7 without it, a gap that may be noise.
I gave a reading subagent and a reviewing subagent different models, from Haiku to Opus, and let them review a small Rust crate with three planted bugs. Every setup found all three bugs. The real differences were the cost, a handoff that the main session quietly shortened, and one data-loss bug that I had not planted.
I gave 40 failing Rust programs to Haiku, Sonnet and Opus. Almost every fix compiled. Reading the fixes showed the real difference, and what one accepted fix cost.
Price per token said Haiku is 40 times cheaper than Opus. My 40 Rust fixes gave other numbers. This is how I compute cost per completed task, with failed attempts, retries, escalation and the fixed overhead of every call.
A router sends each task to a cheap model first and escalates when a check fails. I computed one from real runs on 40 Rust fixes and compared it with one strong model at low effort. It would never have escalated, and the reason was the check, not the models.