Diagnosing tool selection failures

Selection failures come from a small set of causes, and once the request body has been read and confirmed, working through that set in order is faster than reasoning about any one of them in depth. Two of them account for most of what you will meet: a neighbouring tool whose description claims ground the skipped tool also claims, and a schema demanding information the conversation does not contain.

The two look identical from the outside — a tool that is not being chosen often enough — and they are fixed in opposite places. The first is a wording problem fixed in the other tool's description; the second is not a wording problem at all. Telling them apart takes one query over the arguments the tool did receive.

Overlap is fixed in the tool that is too broad#

Consider a documentation assistant where search_docs is selected on 71% of documentation questions. The remaining 29% are not the model answering from memory, which is what everyone assumes the failure mode will be. On every one of those turns it makes a tool call, and the call is to answer_faq. The two descriptions, side by side, explain it immediately:

[
  {
    "name": "search_docs",
    "description": "Search the product documentation and return matching passages. Use for questions about how a feature works, configuration, limits, or supported integrations."
  },
  {
    "name": "answer_faq",
    "description": "Answer general user questions using our curated FAQ knowledge base."
  }
]

answer_faq claims general user questions. A question about how a feature works is a general user question, so both tools apply, and the model is choosing between a specific claim and a broader one that subsumes it. Nothing in either description says where one stops. This is the same situation as two route patterns registered on overlapping paths, except that a web framework resolves the ambiguity by a documented rule — first match, or most specific — while the model resolves it by which description sounds like a better fit for this sentence, which varies from turn to turn and is why the split is 71/29 rather than 100/0 or 0/100.

The instinct is to strengthen the description of the tool being skipped, and it is the wrong end. search_docs is already specific; making it more emphatic does not narrow the other tool's claim, and adding "always use this for documentation questions" to one description while the other still says "general user questions" leaves the model with two instructions that both apply. The edit that changes behaviour goes into the broad tool, and it names the sibling:

{
  "name": "answer_faq",
  "description": "Answer questions from the curated FAQ: account setup, billing cycles, password resets, and refund eligibility windows. Do not use for questions about how a product feature works, configuration options, API limits, or integrations — use search_docs for those. If the FAQ has no entry, return no match rather than guessing.",
  "input_schema": {
    "type": "object",
    "properties": { "question": { "type": "string" } },
    "required": ["question"]
  }
}

Two things happened in that rewrite. The positive claim was replaced with an enumerated territory, so "general" no longer stretches over the neighbour's ground, and an explicit negative clause names search_docs as the destination for the excluded cases. On the replay set the split moved to 94% search_docs and 6% answer_faq, and the remaining 6% were questions a human reviewer also classified as FAQ-shaped.

Both halves are needed, and it is worth seeing why neither works alone. Enumerating the territory removes the overlap: "account setup, billing cycles, password resets, refund eligibility windows" is a list a configuration question is not on, whereas "general user questions" is a category it is squarely inside. But enumeration alone leaves an excluded question with nowhere to go, and a model that has ruled out the only tool it was considering will sometimes answer from memory rather than look for another. The negative clause supplies the destination, so the exclusion becomes a redirection. Naming search_docs inside answer_faq's description also puts the comparison in one place: the model no longer has to infer the boundary by weighing two independent claims, because one of the two descriptions states where the line is.

That is what makes rewriting a description a real intervention rather than a placebo: it is the only lever in the request that changes the boundary between two tools. The description is where selection criteria live, and a description that says only what a tool does leaves the criteria unstated.

Required fields with no source in the conversation#

A refunds team had a different problem with the same headline. Their tool was rarely selected, and when it was selected the arguments were wrong in a specific way:

{
  "name": "refund_order",
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": { "type": "string" },
      "customer_id": { "type": "string" },
      "idempotency_key": { "type": "string" },
      "requested_by_agent_id": { "type": "string" }
    },
    "required": ["order_id", "customer_id", "idempotency_key", "requested_by_agent_id"]
  }
}

Over 240 production calls, order_id matched a real order 97% of the time. customer_id matched 31%. idempotency_key matched nothing, ever — the values were things like refund-AB-40912-001 and idem_20250114_a7f3. requested_by_agent_id matched nothing either, arriving mostly as agent_1, support_bot, or the name of the human agent if one had been mentioned in the transcript.

Read that column downwards and it is not four unrelated numbers. It is one number — how often the value exists somewhere in the conversation — reported four times. An order id is spoken aloud by the customer in nearly every refund request, so it matches almost always. A customer id is never spoken, but a prior lookup_order result sometimes puts one into the message history, and it matches on roughly the fraction of conversations where that lookup happened first, which is what a figure around a third is describing. An idempotency key and an agent id appear in no message ever, from either side, and they match at nothing. The argument-match rate is therefore not a measure of how well the model understands your schema. It is a readout of what the conversation contains, and you can predict it before running the query by asking, field by field, where in the transcript this value would have to have come from.

Both symptoms come from one cause. Two of the four required fields have no source anywhere in the conversation: an idempotency key is a value your system mints so that a receiving service can record it against a completed write and return the original outcome instead of performing the write a second time, and an agent id lives in your session, and neither has ever appeared in a message the model can read. That single fact produces the rare selection and the fabricated arguments together. Selection drops because the tool, viewed through its schema, demands four pieces of information of which the conversation supplies at most two, which makes it a poor fit for the turn compared with asking a clarifying question or reaching for a tool it can fully populate. Fabrication follows because required means required — once the tool is chosen, the decoder must emit a string for idempotency_key, and the only strings available are ones constructed on the spot.

The selection half of that deserves its own step, because it is the less obvious one. Nothing in the protocol computes whether a tool is satisfiable and rules it out; there is no such check anywhere. What happens instead is that the model is producing the most plausible continuation of a transcript, and a tool-use block is one candidate continuation among several — a clarifying question and a plain answer being the others. A block for a tool with four required fields, two of which have no antecedent anywhere in the text above, is a less plausible continuation of that text than a block for a tool whose fields are all sitting in the preceding sentences. The schema does not gate selection. It changes how well the call fits, and fit is the whole of what is being sampled from.

Which is why the two symptoms arrive together rather than separately. On turns where the fit is poor enough, something else wins and the tool is not selected; on turns where it wins anyway, the required fields still have to be filled and there is nothing to fill them from. One cause, two appearances, and a fix that addresses only the wording will move neither.

The tempting fix is to describe the fields better: explain what an idempotency key is, add a pattern of ^idem_[a-f0-9]{16}$, give an example. It is tempting because it is the fix that works for every other kind of wrong argument, and because the resulting calls do look better. What it actually buys is inventions that match the pattern. A value like idem_3f9a1c72b40e8d55 is indistinguishable from a real key by shape, so it stops failing your input validation and starts failing further in. The log line that used to say unknown idempotency key: refund-AB-40912-001 now says nothing at all until two refunds land against the same order. Better wording made the failure quieter without making it rarer.

That is worth stating as a general shape, because the same move is available and equally wrong in several places. A pattern constrains the form of a string and has no access to the set of strings that exist in your database. Constraining form when the problem is existence converts a loud failure into a silent one, and a silent failure is strictly worse: the loud version was caught by a validator on the way in, and the quiet version is caught by a customer noticing they were refunded twice. Debugging gets harder in exactly the proportion that the invented values got more convincing.

The correction is to the schema. Fields whose values live in your infrastructure come out of the tool definition entirely, and the dispatcher supplies them when it executes the call:

def dispatch_refund_order(args: dict, session: Session) -> dict:
    order_id = args["order_id"]
    # Not model-supplied: authoritative in the session.
    customer_id = session.customer_id
    # Not model-supplied: derived so a retry of the same intent collapses.
    idem = hashlib.sha256(
        f"{session.conversation_id}:{order_id}:{args['amount']}".encode()
    ).hexdigest()
    return refunds.issue(
        order_id=order_id,
        customer_id=customer_id,
        amount=args["amount"],
        idempotency_key=idem,
        requested_by_agent_id=session.agent_id,
    )

What stays in the schema is order_id, which the customer states or which a prior lookup_order result put into the conversation, and amount, which is either stated or defaults to the order total. What stays in required is order_id, because a refund with no order is not a refund.

A list of causes that terminates#

The table below is the order worth working through, because it runs cheapest-to-check first.

Scoring the edit on every tool, not on the one you fixed#

A description change is a change to the boundary between two tools, so measuring it on the tool you were worried about answers half the question. The measurement that answers the other half is a replay set: conversations captured from production, each labelled by a human with the tool a competent agent would have reached for, replayed against the current payload with tool_choice on auto and nothing executed.

def replay(cases: list[Case], tools: list[dict]) -> Counter:
    chosen = Counter()
    for case in cases:
        resp = client.messages.create(
            model="claude-sonnet-4-5-20250929",
            max_tokens=512,
            tools=tools,
            tool_choice={"type": "auto"},
            messages=case.messages,
        )
        blocks = [b for b in resp.content if b.type == "tool_use"]
        chosen[(case.expected, blocks[0].name if blocks else "no_call")] += 1
    return chosen

Running that over 400 labelled turns before and after the answer_faq rewrite gives the pair of numbers the fix should be judged on.

The rewrite moved 23 points onto search_docs and took 14 off answer_faq, which is the shape almost every boundary edit has: an exclusion clause pushes the line, and it pushes it slightly too far. Whether that is a good trade depends on what the two tools cost when they are wrong — a documentation search returning FAQ-shaped content is a mildly worse answer, while an FAQ lookup on a configuration question is a confidently wrong one — and it is a trade nobody can evaluate from a single-tool metric. The bottom row deserves the same attention: a narrowing edit that pulls turns away from no call at all has made the assistant more eager rather than more accurate, and five turns out of sixty is small enough to accept and large enough to watch.

The two figures in that first sentence are percentage points, and percentage points from different rows are not the same size. Twenty-three points of a 180-turn row is twenty-three hundredths of 180, which is around forty turns gained; fourteen points of a 90-turn row is fourteen hundredths of 90, which is around thirteen turns lost. Subtracting one point figure from the other would suggest a modest net gain of nine of something, and the something has no meaning, because the two percentages are fractions of different denominators. Converting each back to turns before comparing is the step that makes the trade legible: roughly forty recovered against roughly thirteen given up, plus the two smaller rows.

Do the same conversion across all four rows and the set-level figure appears. Multiply each row's before-rate by its turn count, add the four products, and divide by the 400 turns: a little over eighty-four correct in every hundred. Repeat with the after-rates and it is around ninety-one. That is the number to quote when somebody asks whether the edit was worth shipping, and it is the number that would have been missed entirely by watching search_docs alone — which would have reported a 23-point win and said nothing about the thirteen turns it was funded by.

Where the rate moves inside a single conversation#

The history row of the table is worth a second look, because it is the only cause whose symptom moves during a conversation and it therefore hides from any measurement taken on first turns. The documentation assistant selected search_docs on 94% of first-turn questions and on 41% of questions asked at turn five or later, and the difference was entirely explained by what the earlier calls had returned: two results of {"passages": []} sitting in the message history, after which the model had reasonable evidence that this tool does not answer questions. Splitting the selection rate by turn index makes the pattern obvious in one query, and the remedy is in the result rather than the description — an empty search that returns which index was queried and what terms were tried gives the model something to act on, and dropping a useless result from the history entirely is the blunter version that also works. What a result should contain is the subject of chapter 7.

The last row is the one to keep honest about. A model that responds to "I want my money back" by asking which order, rather than calling refund_order with a guess, is doing the right thing, and counting that turn as a selection failure will push you toward changes that make the assistant worse. Before treating a rate as a bug, read twenty of the misses and decide how many of them a competent human agent would have handled with a question.

None of these fixes produce determinism either. Selection is a sample, and at temperature 1.0 the same conversation replayed twice can take different paths, so the honest target for a well-specified tool is a rate in the nineties rather than a guarantee. Measure it as a rate over a labelled replay set, keep the set in version control, and re-run it when you change a description, because a wording change that helps one tool routinely costs a neighbour a few points and only a set shows you that.

Two questions, then, answered in different files. Whenever a tool with a correct payload is passed over on auto, the first place to look is the description of whatever it was passed over for, and the edit goes there. Whenever a tool is both rarely chosen and called with values matching nothing in your systems, the field with no source in the conversation comes out of the schema and moves into the dispatcher.

There is also a class of requirement that description-writing is simply the wrong instrument for. If a compliance rule says every response about account balances must be grounded in a live lookup, no rate in the nineties satisfies it, and the lever people reach for is the tool choice field — forcing the model to call a specific tool on every turn. That lever exists, it does what it says, and it removes something from the model in the process, which chapter 6 takes apart. Before that comes the situation that appears the moment tools are being selected reliably: a model that has stopped hesitating starts asking for four things at once.