Mehdi Akiki
Writing / Series

Reliable AI Systems in Production

Evals, retrieval, tools, and observability

A software-engineering view of AI-enabled products: where models help, where deterministic boundaries remain necessary, and how to operate the whole system responsibly.

The model behind the series

A useful AI feature is still a production system. It depends on permissions, identity, current data, tool contracts, latency budgets, evaluation, and a recovery path when a probabilistic component behaves differently than expected.

This series focuses on those engineering boundaries. The model is important, but it is one component inside a system that still needs clear invariants and evidence that it works.

Core principles

01

Keep deterministic boundaries explicit

Use models for useful ambiguity, while permissions, state transitions, money, and irreversible actions remain enforceable in code.

02

Evaluate the real workflow

Measure retrieval, tool selection, outputs, latency, and cost against representative tasks instead of judging a demo by feel.

03

Make failure inspectable

Record the inputs, versions, sources, tool calls, and decisions needed to explain and improve a production result.

Start here

Entry points into this subject, ordered as a reading path.

Articles

Through the layers

Start an internal AI tool with read-only access

Roll out one narrow internal assistant before adding write actions. Read-only still requires authorization, output limits, and checks against private-data exposure.

Through the layers

Only offer tools that belong to the current workflow

Build tool manifests for the current task instead of exposing an entire backend. Keep independent server authorization because hiding a tool does not prevent a fabricated call.

Through the layers

A denied tool call must leave no side effect

An error response does not prove that a tool did nothing. Validate and authorize before effects, then assert unchanged storage and zero external calls after rejection.

Through the layers

A fallback model must not get wider permissions

An automatic model fallback should preserve actor scope, tools, approvals, and budgets. Never turn a provider failure into a switch to an administrative execution context.

Through the layers

Who Writes the Instructions Your Coding Agent Follows?

Before I type a word, my coding agent already has instructions from files, hooks, plugins and online services. I list who wrote each one, which ones can change without a commit, and how to review them.

Through the layers

When an Agent Rule Needs a Hook, Not a Sentence

I tested one CLAUDE.md rule, never run git push, in 81 Claude Code runs, with and without a deny rule and a hook. The sentence held every time. The gaps were in the other layers.

Derived state

Even a trusted tool needs an output limit

Bound tool results before they become model context or logs. Use explicit pagination or overflow errors so partial data cannot look like a complete answer.

Measurement

Keep the broken input before you fix the bug

Capture a small regression fixture before changing code or asking an agent for a fix. Prove that the test detects the original failure and preserves the real requirement.

Measurement

A Record-and-Replay Harness for Agent Tests

A working record-and-replay harness for agent tests: a JSON cassette of decisions and tool results, deterministic replay with zero live calls, and a readable diff when the trajectory changes.

Measurement

Red-Team the Workflow, Not Only the Prompt

A strong system prompt cannot secure retrieval, tools, authorization, retries, or side effects. Red-team the complete AI workflow and its recovery paths.

Measurement

What rtk Really Saved in My Claude Code Setup, and What It Changed

I measured rtk, a token-saving tool, inside my Claude Code setup. It gave two wrong answers, and one is still there after the update. Its hook also approved a force push without asking me, and its own savings number fell from 125.5M to 4.8M tokens.

Measurement

Auditing an Agent Instruction File Line by Line

My site's AGENTS.md adds 1,382 tokens to every Claude Code request. I measured each part, labeled every line, and tested a slimmer version. It dropped a line that I thought was useless, and the agent tried to change the commit author.

Measurement

Estimate the Monthly Cost of an AI Feature Before Building It

I turned measured token counts from a real classification test into a monthly cost range at 10,000 and 100,000 messages. The model bill went from cents to a few hundred dollars. Under my stated assumptions, the human time for wrong answers was far bigger.

Measurement

Turning the Rust Failure Atlas Into an Agent Skill, and What It Did to the Fixes

I built a short Claude Code skill from the Rust Failure Atlas and tested it on 30 Atlas cases with Haiku. Every fix compiled and ran, with and without the skill. In one run per case, 11 fixes with the skill changed what the program was meant to check, against 7 without it, a gap that may be noise.

Measurement

Which Subagent Can Use the Cheap Model: the Reader or the Reviewer?

I gave a reading subagent and a reviewing subagent different models, from Haiku to Opus, and let them review a small Rust crate with three planted bugs. Every setup found all three bugs. The real differences were the cost, a handoff that the main session quietly shortened, and one data-loss bug that I had not planted.

Measurement

Cost per Completed Task: How I Compare AI Models on Price

Price per token said Haiku is 40 times cheaper than Opus. My 40 Rust fixes gave other numbers. This is how I compute cost per completed task, with failed attempts, retries, escalation and the fixed overhead of every call.

Measurement

Try One Model at Lower Effort Before You Build a Router

A router sends each task to a cheap model first and escalates when a check fails. I computed one from real runs on 40 Rust fixes and compared it with one strong model at low effort. It would never have escalated, and the reason was the check, not the models.

Shorter pieces and reference

Older, introductory or narrower pieces on the same subject.

Engineering context

AI capability with ordinary engineering discipline

I help teams turn AI-enabled workflows into maintainable products with clear data, evaluation, permission, and failure boundaries.