Write the description in the words of someone with the problem

The nomad-drain skill was written by the site-reliability engineer who owns the decommissioning runbook, reviewed by two colleagues, and merged with the description Procedure for draining a Nomad client node prior to decommissioning, including allocation migration and eligibility toggling. Nomad is HashiCorp's cluster scheduler, which places the running copies of a job — it calls them allocations — onto the worker machines it calls client nodes; draining one of those machines means moving its allocations off it and marking it ineligible for new ones before the machine itself goes away. Every word of that description is true. It uses the terminology from the Nomad documentation, it names the two operations the procedure performs, and it would survive any technical review you put it through.

In forty days it fired zero times. The team pulled the agent's session transcripts for the same period and found eleven requests that the skill existed to serve:

• "take box i-0a4f out of the cluster"
• "we're returning the r5.2xlarge next week, can you get the jobs off it"
• "node ip-10-2-31-8 is being replaced tomorrow, move everything"
• "how do I safely remove a worker"
• "i-0b91 is flapping, get it out of rotation"

None of those sentences contains the word drain. None contains decommissioning, allocation, or eligibility. Two of them contain cluster, and the rest contain an instance id, a verb of removal, and a deadline. The description and the requests are describing the same event in two disjoint vocabularies, and the description is the one that never gets to speak first.

Two people naming the same thing agree less than you expect#

This failure is old enough to have a name and a measurement. In 1987, Furnas, Landauer, Gomez and Dumais published The Vocabulary Problem in Human-System Communication in CACM, reporting a series of studies in which people were asked to name commands, objects and categories. The probability that two people spontaneously chose the same term for the same familiar thing was between 10% and 20%, depending on the domain, and it did not improve much when the participants were experts. Their conclusion was that any system whose access path depends on a single chosen term will fail most of the people trying to reach it, and their proposed fix — unlimited aliasing, so that many terms reach the same entry — is essentially what a good skill description does in prose form.

Semantic matching, where the comparison runs on what a request means rather than on the words it happens to share with the description, softens this without repealing it. A model that knows Nomad will happily accept that "take a box out of the cluster" is related to draining a client node, so the failure is not that the connection is unmakeable. It is that the connection has to be strong enough to beat every other option on the list, including the option of not selecting anything and just answering. The description gives the model no situational hook at all: nothing in it describes a person with a machine they need to get rid of. It describes a procedure, from the point of view of somebody who already knows the procedure is what they want.

The environment makes it worse, because nomad-drain was not alone. The library also contained ec2-teardown, described as Use when retiring or replacing an EC2 instance: deregisters from load balancers, snapshots volumes, and terminates. That description is written in the vocabulary of the request — retiring, replacing, instance — so eight of the eleven requests selected it. It ran its steps, terminated instances that still had allocations on them, and the jobs were rescheduled by Nomad the hard way, with about 90 seconds of failed health checks each time. The skill that fired was not wrong to fire; it was just the only one that had made itself findable.

Vague fails differently, and just as completely#

The team's other never-firing skill was the opposite defect. runbooks, described as Team runbooks and operational procedures., is not jargon-locked; it is written in ordinary words that anybody would use. It fired four times in forty days, twice on requests it could not help with.

A description at that altitude is weakly related to almost every operational request and strongly related to none. Selection is comparative, so a weak match loses to any skill with a specific one, and on requests where no specific skill exists the weak match still has to clear the model's bar for preferring a skill over answering directly. Sitting just under that bar on a hundred requests looks exactly like sitting far under it, because the observable in both cases is nothing happening.

The fourth row is the one that kept the original description alive for forty days. The author tested it, and the author phrased their test the way the author talks, so it worked. Every author is the one person in the organisation guaranteed not to have the vocabulary problem with their own skill.

Replaying the log instead of guessing#

The fix is mechanical once you decide the request log is the source of truth rather than the author's sense of the domain. Session transcripts record which skill was selected on each turn, so the first step is extracting the pairs of user message and selected skill for a period long enough to contain the requests you care about.

# Extract (request, skill selected) pairs from local session transcripts
jq -r '
  select(.type == "user" or .type == "skill_load")
  | [.timestamp, .type, (.text // .skill_name)]
  | @tsv
' ~/.claude/projects/*/*.jsonl \
  | awk -F'\t' '$2=="user" {req=$3} $2=="skill_load" {print $3 "\t" req}' \
  | sort | uniq -c | sort -rn

That gives you what fired. What you need next is what should have fired, which is a human judgement over the requests where nothing fired or the wrong thing did. Forty requests is enough to be useful and small enough that one person can label them in half an hour. Keep them as a fixture, meaning a fixed set of labelled inputs that gets replayed unchanged after every edit so that two versions of a description are scored against exactly the same requests:

[
  {"request": "take box i-0a4f out of the cluster", "expect": "nomad-drain"},
  {"request": "we're returning the r5.2xlarge next week, can you get the jobs off it", "expect": "nomad-drain"},
  {"request": "i-0b91 is flapping, get it out of rotation", "expect": "nomad-drain"},
  {"request": "terminate the two idle build boxes, nothing runs on them", "expect": "ec2-teardown"},
  {"request": "what's our procedure for pager escalation", "expect": null}
]

The null rows are as important as the rest. A description tuned only against requests it should win will drift outward until it wins requests belonging to its neighbours, and the negative cases are the only thing that stops that. In this fixture, "terminate the two idle build boxes, nothing runs on them" is the row that keeps nomad-drain from swallowing all of ec2-teardown's territory, and the pager question is the row that keeps every description from growing an "operational procedures" clause.

Running the fixture means replaying each request against the current library and recording the selection. There is no API that exposes the selection decision on its own, so the replay is a scratch session per request with the full library installed and nothing else in context, and the observable is which SKILL.md the agent reads. Selection is sampled rather than deterministic, which shows up immediately: on the first run, four of the forty requests selected different skills on different attempts. Three replays per request and a rule that a row passes only when all three agree is enough to keep the noise below the size of the effects you are trying to measure, and it costs about twelve minutes of wall time for forty requests.

On the first pass the numbers were 3 correct out of 40, with 22 selections of the wrong skill and 15 turns where nothing fired at all. The 22 are worth separating from the 15, because they are different diseases: the wrong-skill rows say a neighbour's description is a better match for these requests than yours, and the nothing-fired rows say your description did not clear the bar for using any skill at all. A rewrite that fixes one often leaves the other untouched.

The rewrite, and what each clause is for#

---
name: nomad-drain
description: >-
  Use when a Nomad client node is being removed from the cluster for any
  reason — hardware replacement, instance return, an unhealthy or flapping
  box, an AMI roll, or scaling down. Covers moving running jobs off the node
  and marking it ineligible before it is stopped or terminated. Requests look
  like "take i-0a4f out of the cluster", "get the jobs off this host", or
  "drain node X". Not for terminating an instance that runs no Nomad
  allocations, and not for draining a Kubernetes node.
---

The clauses do four separate jobs. The first sentence names situations a person can be in rather than a procedure they might want, and it lists the ones that actually occurred in the log, including "flapping" which no runbook author would have thought to include. The second sentence keeps the procedural vocabulary, because requests from people who do know the term still have to match. The third quotes the request forms directly, which is the crudest of the four devices and the most reliable, since it puts the exact surface strings into the document being matched. The fourth marks the boundary against the neighbour that has been stealing the traffic, and against a plausible confusion — Kubernetes also has a drain — that had not happened yet but would.

After the rewrite the same 40-request fixture scored 36 correct, with two requests still going to ec2-teardown (both genuinely ambiguous, describing instances that may or may not have had allocations) and two where nothing fired. That is a fire rate that makes the skill worth the body it carries.

The obvious over-correction#

Everything above pushes a description outward, and there is a point past which that stops helping. A description that lists twenty synonyms and every phrasing anybody has ever used becomes a broad matcher, and a broad matcher wins requests belonging to its neighbours — the exact failure ec2-teardown was inflicting in the other direction. Keyword stuffing also produces descriptions long enough to matter at rest across a library, and it degrades in a way that is hard to see, because the skill fires more and the extra fires are wrong.

The negative fixture rows are the guard against this, and the discipline is that a phrase goes into a description because a real request contained it, not because it might. When you add a clause, rerun the fixture and look at whether the null and neighbour rows still pass; a rewrite that improves recall — the share of the requests a skill ought to win that it does win — from 3 to 36 while breaking two negative rows has not obviously helped.

There is also a case where none of this is available. A brand-new capability has no request log, because nobody has asked for something they do not know exists. The honest procedure there is to write the ten requests you expect, in the words you expect other people to use rather than the words you would use, and treat the description as provisional until the log has real traffic in it. Two of the eleven nomad-drain phrasings would not have been predicted by anyone; you get those only by looking.

What none of this fixes is a skill that is genuinely trying to be several things at once. The rewrite above worked because nomad-drain has exactly one job, so every clause added to its description pushed in the same direction. A description covering five unrelated topics has no single direction to push in, and the next chapter is about what happens when you try anyway.