Mehdi Akiki
Published on

Does My Skill Load When It Should? I Tested Two Descriptions With 60 Prompts

Authors
  • Mehdi Akiki avatar
    Name
    Mehdi Akiki
    Twitter

Article · Measurement

I wrote a small skill for Claude Code. It has one job: write a new entry in CHANGELOG.md from my staged changes, or from the commits since the last release tag. The steps are in a file called SKILL.md.

But Claude Code does not read that file when a session starts. At the start, the model sees only the skill's name and its description, one line of text. For each request, the model decides if it should load the rest. So a single line decides if my steps are ever used.

I wanted to know how much that line matters. Does the skill load when I ask for a changelog entry, and stay away when I ask for a commit message? And what changes with a vague description?

One skill, two descriptions

The skill lives in a project folder, .claude/skills/changelog-entry/SKILL.md. I made two versions. They have the same name and the same five steps in the body. Only the description is different.

The vague one:

---
name: changelog-entry
description: Helps with changelogs.
---

The specific one:

---
name: changelog-entry
description: Writes a new entry in CHANGELOG.md from the staged diff or from the commits since the last release tag. Use when the user asks to add, draft or update a changelog entry, or asks what to put in the changelog for some changes. Do not use for commit messages, pull request descriptions, or code review.
---

The vague line has the shape of the "poor example" in the Agent Skills specification, description: Helps with PDFs. The specification says a description "Should describe both what the skill does and when to use it" and "Should include specific keywords that help agents identify relevant tasks". The specific line follows that advice and adds a "Do not use for" sentence.

An older article on this site, How to Write Better Skill Descriptions, gives this advice. This time I measured it.

The test

I made a scratch Git repository for a tiny Rust command line tool. It has a CHANGELOG.md in the Keep a Changelog format, two release tags, two commits after the last tag, and one staged change that renames a flag.

Then I wrote 60 prompts myself. They are constructed for this test, not taken from real sessions. 30 of them should load the skill, for example:

Add a changelog entry for my staged changes.
changelog for staged pls
Fill in the [Unreleased] section.
Draft changelog bullets for the commit that added --json.

The other 30 are near misses. They are close to the skill's job, but they are not that job:

Write a commit message for my staged changes.
Review my staged diff for bugs.
Write a short release announcement for our blog about 0.4.0.
In CHANGELOG.md, change the date of 0.4.0 to 2026-10-31.

Ten near misses belong to the three things that the specific description names in its "Do not use for" sentence. The other twenty do not: version bumps, release announcements, edits to old changelog sections, questions about Git history, and so on.

Each prompt ran once per description, in a fresh copy of the repository, with Claude Code 2.1.325:

claude -p "$PROMPT" --model haiku \
  --output-format stream-json --verbose \
  --setting-sources project,local \
  --max-turns 6 --max-budget-usd 0.10 \
  --no-session-persistence --permission-prompts none

For Sonnet, the budget was USD 0.40 per run.

--setting-sources project,local keeps my personal settings, hooks and skills out of the test. With --permission-prompts none, every action that needs my approval is denied. So the model could read files and run some Git commands. Edits were denied, and so were some shell commands, even a git show in one run. It could not edit CHANGELOG.md. Here I measure whether the skill loads, not how good the entry is.

A load is easy to see in the stream. It is a call to the Skill tool:

{
  "type": "tool_use",
  "id": "toolu_01FwbbPG7tHyeapw8y7xfrBo",
  "name": "Skill",
  "input": { "skill": "changelog-entry" },
  "caller": { "type": "direct" }
}

The first event of every run listed 20 skills: mine, and 19 that came with Claude Code, like code-review and debug.

The results

With Haiku, all 60 prompts, one run each:

DescriptionReal requests that loaded the skillNear misses that loaded it
vague18 of 300 of 30
specific28 of 300 of 30

With the vague line, Haiku skipped the skill for 12 of the 30 requests that needed it. With the specific line, it skipped only two.

The way it loaded was different too. With the specific description, the Skill tool was the first tool call in all 28 loads. With the vague one, 3 of the 18 loads came only after Haiku had first looked around with a Git command.

Then I ran every third prompt again with Sonnet, 10 real requests and 10 near misses per description. I chose that rule before I saw any result.

Model, descriptionReal requests that loaded the skillNear misses that loaded it
Haiku, vague (same 20 prompts)5 of 100 of 10
Haiku, specific (same 20 prompts)9 of 100 of 10
Sonnet, vague10 of 100 of 10
Sonnet, specific10 of 100 of 10

On these 10 prompts, Sonnet loaded the skill every time it should, even with "Helps with changelogs." So in this test, the vague description hurt the small model and not the bigger one. With one run per prompt, I cannot say how stable that is. But if you test your skill only with a strong model, a weak description can look fine.

Sonnet also cost more. The 40 Sonnet runs cost USD 1.47 as reported by Claude Code, about 4 cents per run. The 120 Haiku runs cost USD 0.40 together.

Near misses did not separate the two descriptions

The "Do not use for" sentence had no chance to show its value here. The near misses did not load the skill with either description and either model, not even the twenty that the sentence does not name.

Hmm, so even "Helps with changelogs." did not pull in a request like "Set up a GitHub Action that fails when CHANGELOG.md is not updated". I think the model simply judged that a changelog skill is not needed to write a GitHub Action, or to change one date in a file.

So in this test the description decided recall, the share of real requests that loaded the skill. It did not decide false loads. For a skill with side effects, false loads may matter much more, and my near misses cannot tell you about that.

Three misses I learned from

"Draft changelog bullets for the commit that added --json." Neither description loaded the skill for this one. With the specific description, Haiku ran one git show, wrote two bullets in its reply, and stopped. It ended with: "I haven't written anything to CHANGELOG.md. If you want the entry added there, I can do that next." I think the word "draft" made it read as a request for text, not for a change to the file. The specific description even contains the word "draft". It was not enough.

"Make sure CHANGELOG.md mentions the empty CSV fix and the --json flag." This is the second miss of the specific description. The request sounds like a check, not like "write an entry". Haiku tried to edit the file directly (the edit was denied) and gave the entry in its reply. The entry looked fine, but the reply says: "I didn't check the diffs of 7ffe980 and c28c9ab beyond their commit messages, so the wording is based on those messages." The first step of my skill is to read the changes. This is what a missed load costs: the answer looks finished, but my steps were skipped.

"Describe the staged diff as changelog lines for end users." The vague description missed this one. The answer had a line about the README: "README usage line now lists the options". Step 4 of my skill says: "say what changed for them, not which files changed." Again, the work was close, but it missed the one rule that the skill exists for.

All three misses have the same pattern. Without the skill, Haiku still knew how to write a changelog entry. What it did not know were my rules. A missed load is not a crash. It is a quiet difference in quality, and you only see it when you read the output.

What Codex documents

I could not test Codex for this article. Its usage limit was reached on my account, so the Codex side is not tested here.

Its documentation tells the same story, though. The Codex skills page (formerly developers.openai.com/codex/skills) says "Skills build on the open agent skills standard." About automatic use, it says "Implicit invocation: ChatGPT or Codex can choose a skill when your task matches the skill description", and then "Because implicit matching depends on description, write concise descriptions with clear scope and boundaries." And under best practices: "Test prompts against the skill description to confirm the right trigger behavior."

Claude Code's skills documentation says the same about loading: "In a regular session, skill descriptions are loaded into context so Claude knows what's available, but full skill content only loads when invoked."

Limits

  • One skill, one repository, one laptop, on 2026-11-14.
  • The 60 prompts are constructed by me. Real requests may be shorter, longer or messier.
  • One run per prompt and description with Haiku, and one run on a third of the prompts with Sonnet. Small differences, like 9 against 10, can be noise. I did not repeat the runs to measure that.
  • Edits were denied, and I set --max-turns 6. Six of the 120 Haiku runs hit that limit, four of them after the skill had loaded. In the Haiku runs, every load of the skill was the first or the second tool call.
  • My skill was the only project skill. With many skills, Claude Code drops some descriptions to fit the skill listing's character budget, so the results may change.

What I would do with my own skills

Write 20 or 30 prompts the way you really type them, including the lazy ones. Add a few near misses. Run each one with --output-format stream-json --verbose and search for "name":"Skill" in the stream.

Then read the answers of the misses, not only the count. In my test, a missed load still gave an answer that looked right. The difference was in the rules that only the skill knew.

And if your skill matters, do not leave its description as "Helps with" something. A second sentence that says what the skill does and when to use it is cheap.