Mehdi Akiki

Writing

The main body of this writing follows one system below its interface: a compiler, a runtime, a protocol, a build, a data pipeline. It is organized around four engineering questions.

Investigations run over several parts and keep the programs and experiments that produced their evidence. Articles stand alone. Shorter notes, beginner material and tool guides stay in the archive and in search.

Investigations

Multi-part work backed by code, measurements or reproductions that can be run again.

  1. Through the layers · 10 parts

    Types under the hood

    Ten parts on what a type is and where it goes: sets of values, layouts and padding, erasure and monomorphization, the bits the processor actually sees, and what the checker knows that the binary forgets. Small programs in Rust, C, TypeScript and Python, with one script that reproduces every output.

    Evidence: experiments/rust-atlas/types-under-the-hood/

    1. What Is a Type? A Set of Values, a Promise About Operations, or Both
    2. Does a Type Exist at Runtime? Following One Value From Source to Register
    3. A Type Is a Set: Why Human Has Two Members and Option of Human Has Three
    4. Shape Without Behavior: Two Structs With the Same Fields
    5. Behavior Without Shape: Traits, Interfaces, and Types That Own No Bytes
    6. Where a Type Becomes a Layout: Fields, Padding, and Reordering
    7. Two Ways to Make a Type Disappear: Erasure and Monomorphization
    8. The Processor Has No Types: What add Does to Bits It Does Not Understand
    9. Runtime Type Tags: What Dynamic Languages Keep That Static Ones Throw Away
    10. Types as Proofs: What the Checker Knows That the Binary Forgets
  2. Interrupted execution · 3 parts

    Async Rust under the hood

    Five async Rust problems reproduced in one small tokio program: an oversized future, a future that is not Send, a blocking call, a select! loop that loses data, and a shutdown that hangs. Then the cost of async fn through dyn Trait, and cleanup when Drop cannot await.

    Evidence: experiments/rust-atlas/async-symptoms/

    1. How Async Rust Works Under the Hood: Five Symptoms, One Model
    2. Async fn in a dyn Trait: What the Box Costs, Measured
    3. Async Cleanup in Rust When Drop Cannot Await
  3. Through the layers

    Rust Failure Atlas

    Rust failures that are difficult to name and easy to misdiagnose, organized from symptom to mechanism, with a failing and a repaired fixture for most cases.

  4. Through the layers · 20 parts

    From source to a running process

    Twenty parts that follow small C and Rust programs from the linker to the exit of the process: ELF files, relocations, static and dynamic linking, the loader, the PLT and the GOT, LD_PRELOAD, the code before main, the first stack, the memory map, page faults, fork, malloc, system calls, stack overflows and thread-local storage. One script reproduces every output.

    Evidence: experiments/from-source-to-process/

    1. What Happens When You Run a Program: From execve to main
    2. What Is Inside an Executable File: ELF Headers, Segments and Sections
    3. Where Your Variables Live in a Binary: .text, .rodata, .data and .bss
    4. Why "Undefined Reference" Is a Linker Error, Not a Compiler Error
    5. Relocations: The Holes the Compiler Leaves for the Linker
    6. Static or Dynamic Linking: What Changes in the Binary and at Startup
    7. How the Dynamic Loader Finds a Shared Library
    8. How a Call to printf Finds printf: The PLT and the GOT
    9. How LD_PRELOAD Replaces a Function Without Recompiling
    10. The Code That Runs Before main: _start, Constructors and init_array
    11. What Is on the Stack When a Process Starts: argv, envp and auxv
    12. Reading /proc/self/maps Line by Line
    13. Why the First Touch of Memory Is Slow: Demand Paging Measured
    14. What fork Really Copies: Copy-on-Write Measured
    15. When malloc Asks the Kernel for Memory: brk and mmap
    16. Why Freeing Memory Does Not Shrink Your Process
    17. What a System Call Costs, and Why Some Calls Skip the Kernel
    18. How a Stack Overflow Is Detected: Guard Pages and Signals
    19. How Each Thread Gets Its Own Copy of a Variable: Thread-Local Storage
    20. What Happens When a Process Exits
  5. Interrupted execution · 20 parts

    What survives a crash

    Twenty parts on what really reaches the disk: write and the page cache, the cost of fsync, saving a file with rename, the directory fsync, torn writes, a crash simulator built from strace output, a small write-ahead log with checkpoints and recovery, SQLite journal and WAL modes, group commit, mmap, checksums and crash testing. Small programs in Rust, Python and C, with one script that reproduces every output.

    Evidence: experiments/what-survives-a-crash/

    1. Your Data Is Not on Disk When write() Returns
    2. What fsync Costs on a Laptop SSD
    3. How to Save a File Without Losing It: Write, fsync, Rename
    4. Why the Directory Needs fsync Too
    5. The Zero-Length File Problem and What ext4 Does About It
    6. What a Crash in the Middle of a Write Leaves Behind
    7. Building a Tiny Crash Simulator From strace Output
    8. An Append-Only Log That Survives a Torn Last Record
    9. Write-Ahead Logging From Scratch
    10. Checkpointing a Log Without Losing the Tail
    11. Recovery Code Must Survive a Crash Too
    12. When fsync Fails: What PostgreSQL Learned in 2018
    13. How SQLite Commits a Transaction, Followed With strace
    14. SQLite WAL Mode: Fewer fsyncs, a Different Promise
    15. Durability Settings That Trade Your Data for Speed
    16. Group Commit: How Databases Make fsync Cheaper
    17. Writing Through mmap: Who Decides When Data Reaches the Disk
    18. Checksums: Catching Corruption the Disk Did Not Report
    19. Updating Several Files at Once With One Rename
    20. Testing Crash Recovery: What kill -9, Fault Injection and Simulation Each Catch
  6. Measurement · 20 parts

    When the benchmark lies

    Twenty ways a performance number goes wrong, each reproduced on one laptop: a loop the compiler deleted, debug builds, warm-up and the JIT, CPU frequency and efficiency cores, averages and coordinated omission, clocks, run counts, blocks against interleaved runs, environment size, code alignment, caches, branch prediction, the allocator, and a microbenchmark that lost inside the real program. One script reproduces every output.

    Evidence: experiments/when-the-benchmark-lies/

    1. The Benchmark That Measured Nothing
    2. A Debug Build Measures a Different Program
    3. When the Compiler Computes the Answer Before the Benchmark Runs
    4. Why the First Run Is Slower
    5. JIT Warmup: Why a Function Gets Faster After Thousands of Calls
    6. My Laptop Changed Its CPU Speed During the Benchmark
    7. The Benchmark Ran on an Efficiency Core
    8. The Average Hides the Slow Requests
    9. Coordinated Omission: How a Load Test Hides Its Worst Latency
    10. Which Clock to Use When You Measure Time
    11. Measuring Something Faster Than the Clock
    12. How Many Benchmark Runs Are Enough
    13. Compare Two Versions by Interleaving the Runs
    14. Can Environment Variables Change How Fast Your Code Runs?
    15. Does Code Alignment Change the Speed of the Same Code?
    16. When the Working Set No Longer Fits in Cache
    17. Why Sorted Data Can Make the Same Loop Faster
    18. The Benchmark Measured the Allocator
    19. The Microbenchmark Won and the Program Got Slower
    20. A Benchmark Result Someone Else Can Check
  7. Through the layers · 10 parts

    Following a TCP connection

    Ten parts that follow one TCP connection on loopback, from connect() to a connection pool: the handshake, the accept queue, Nagle and delayed ACK, the byte stream, full socket buffers, what close() sends, TIME_WAIT, getaddrinfo and the cost of TLS. Only what ss, the kernel counters, timings and system calls show, with one script that reproduces every output.

    Evidence: experiments/following-a-tcp-connection/

    1. What Happens When Your Program Calls connect()
    2. The Accept Queue: What Happens When a Server Accepts Too Slowly
    3. Why a Small Write Can Wait 40 Milliseconds: Nagle and Delayed ACK
    4. Why recv() Returns Less Than You Asked: TCP Is a Byte Stream
    5. What a Full Socket Buffer Does to the Sender
    6. What close() Really Sends: FIN, RST and Unread Data
    7. TIME_WAIT: Why a Closed Connection Stays Around
    8. How a Name Becomes an Address: What getaddrinfo Really Does
    9. What a TLS Handshake Costs
    10. What a Connection Pool Saves, and the Stale Connection It Gives You Back
  8. Derived state · 10 parts

    Inside a database index

    Ten parts on what an index really is, measured on a SQLite table of one million rows: a second sorted copy of the data, SCAN against SEARCH, the cost on every insert, column order, covering indexes, the cases where the planner ignores an index, statistics that go stale, partial and expression indexes, free pages and VACUUM, and a search index kept in sync.

    Evidence: experiments/inside-a-database-index/

    1. An Index Is a Second Copy of Your Data
    2. Reading EXPLAIN QUERY PLAN: SCAN Versus SEARCH
    3. What an Index Costs on Every Insert
    4. Why Column Order Matters in a Composite Index
    5. A Covering Index Never Touches the Table
    6. When the Database Ignores Your Index
    7. Planner Statistics Go Stale Too: ANALYZE and a Plan That Changes
    8. Partial and Expression Indexes: Index Only What You Query
    9. Deleted Rows Do Not Shrink the File: Free Pages and VACUUM
    10. Keeping a Derived Index in Sync Outside the Database

Articles

Standalone pieces, grouped by the question they answer.

Through the layers

Following a system through its layers

Source, runtime, protocol, compiler, machine. Each piece follows one behaviour down to the layer that decides it.

Investigations: Types under the hood, Async Rust under the hood, Rust Failure Atlas, From source to a running process, Following a TCP connection

All 86 articles in this theme →

Derived state

Derived state, build artifacts and invalidation

What gets stored, what gets recomputed, what becomes stale, and how the system knows.

Investigations: Rust build times, Inside a database index

All 40 articles in this theme →

Interrupted execution

Correctness when execution is interrupted

Crashes, retries, cancellation, partial progress, ownership, recovery and idempotency.

Investigations: Async Rust under the hood, What survives a crash

All 48 articles in this theme →

Measurement

Measurement-driven systems work

Treating an assumption as a hypothesis, building an experiment, following a surprising result into a lower layer, and rejecting a change when the evidence says it loses.

Investigations: This site as an engineering laboratory, Rust build times, When the benchmark lies

All 44 articles in this theme →

Practical series

  • CI and coding agents

    Find the delay, preserve the test, and connect each CI result to the code it checked.

    10 published articles

  • Practical engineering problems

    Small investigations into slow requests, runtime changes, browser work, and data contracts.

    10 published articles

  • Agent security boundaries

    Follow the file, process, network, permission, and runtime boundaries around an agent.

    8 published articles

  • SQL databases from Rust

    Use PostgreSQL, MySQL, SQLite, MariaDB, SQL Server, Oracle, Db2 and DuckDB from Rust. Work through transactions, locks, types and query plans.

    30 published articles

  • Rust backend problems in practice

    Follow failed requests, shutdowns, traces, migrations, sessions, uploads and job leases through small Rust examples.

    8 published articles

  • Rust and database failures

    Follow request waits, uncertain writes, row streams, memory budgets and deployment dependencies through reproducible Rust examples.

    7 published articles

  • Rust database recovery

    Follow duplicate requests, external work, commit failures and import recovery through complete Rust reproductions.

    6 published articles

  • Rust database boundaries

    Check index write costs, tenant isolation, and database permissions.

    3 published articles

  • Rust service runtime decisions

    Trace shared locks, queue admission, task ownership, retained memory, and cleanup.

    9 published articles

  • Rust builds, tests, and measurements

    Keep failing inputs, diagnose build inputs, and compare performance with observable controls.

    10 published articles

  • Rust API integration recovery

    Keep provider outcomes, pagination, caches, durable acceptance, and repairs explicit.

    10 published articles

  • Rust AI tool boundaries

    Carry identity, permissions, approval, and resource limits to the place effects happen.

    12 published articles

The archive

All 554 published pieces, newest first, each labelled as an investigation, an article or reference. Reference covers beginner Rust, framework and tool guides, AI explainers, interview preparation and older short pieces. The 61 short engineering notes have their own page.

Other ways in