Tool-call failure modes

Read these first

• Tool design — granularity and surface

What you will know

• Reproduce a hallucinated tool argument

  Loose schemas and ambiguous descriptions let plausible-looking values pass validation while violating the tool’s actual business assumptions.

• Show selection accuracy degrading as tools are added

  A growing catalog creates near-neighbor choices whose descriptions overlap, making an otherwise capable model select the wrong operation.

• Measure where that degradation starts for your model, rather than trusting a number

  Use a fixed labeled evaluation set and report accuracy by catalog size, because aggregate success can hide failures concentrated in particular tools.

• Apply namespacing or dynamic tool loading and measure the recovery

  Reducing the active catalog can improve routing, but it also introduces missed-tool failures and loading latency that belong in the comparison.

• Detect and break a tool-call retry loop

  Retries can repeat a side effect or consume the entire request budget, so track attempts and failure identity and require an explicit stopping condition.

Chapters

1. 1

   Selection failures and argument failures

   One trace answers a refund question by calling search_orders, which returns a valid empty list, and tells the user there is no record. Another calls refund_order with reason_code CUSTOMER_UNHAPPY, which the backend silently maps to OTHER. Only the second is an argument problem, since no schema constrains which tool gets chosen — and both return 200, so neither appears in error monitoring. Across 200 hand-labelled conversations the two had different victims and a 4-day against 41-day median time to discovery.

   Read chapter
2. 2

   Reproducing a hallucinated argument

   Loose schemas accept plausible values that violate the tool's real business assumptions, nothing errors, and the evidence is destroyed at the moment the failure occurs, so there is no log line to search for. The reproduction has to be constructed: stratify conversations so most of them cannot determine the true value, drive the agent against a recording stub, and diff every emitted value against the backend's actual set. On one refunds corpus that produced 223 out-of-set values in 720 calls.

   Read chapter
3. 3

   Checking arguments against a value set

   A reproduction runs on cases you constructed; the same failure is running in production on cases you did not, and by construction it writes no error line to search. Move the membership predicate into the dispatch path, record it as a metric even when the backend accepts the value and quietly maps it away, and observe before enforcing — on one deployment production ran 6.2% out-of-set against a corpus figure of 31%, and 27% of the mismatches turned out to be formatting rather than invention.

   Read chapter
4. 4

   Near-neighbour density in a tool catalogue

   Ninety-six percent at twelve tools and 71% after eighteen more is not a fact about the number thirty: 94 of the 116 failures fall into seven pairs out of 435 possible ones. Adding eighteen orthogonal tools instead costs 2.8 points at the same catalogue size, and going on to forty-eight costs 1.5 more — which is the control that separates near-neighbour density from tool count, and the reply to the received wisdom that models degrade past roughly twenty tools.

   Read chapter
5. 5

   Per-tool and per-pair selection accuracy

   Replay one fixed labelled request set at increasing catalogue sizes with the model, prompt and tool ordering held still, and read the results per tool and per confused pair. The aggregate is the figure to refuse: 89% across thirty tools was entirely consistent with refund_order falling from 95.8% to 45.8% while twelve other tools held steady, and with eleven of the sixteen new failures sitting in a single pair.

   Read chapter
6. 6

   Namespacing a tool catalogue

   Prefixing forty tools with crm., billing. and support. moved selection accuracy from 87.5% to 87.6%, which is inside the seed spread. That non-result is informative: 53 of the 67 failures are between two tools a prefix places in the same bucket. Read the per-pair table before committing to a scheme, be clear that a prefix resolves reference rather than intent, and price the rename against what it breaks in metrics, labels and stored history.

   Read chapter
7. 7

   Dynamic tool loading

   Loading the eight most relevant tools out of sixty lifted selection accuracy from 71% to 94%, but those two numbers are computed over different populations: 97 of 540 requests never had their correct tool retrieved and were skipped as out of scope. Scored end to end the recovery is 6.1 points, not 23, and the k that maximises end-to-end accuracy is not the k that maximises the quoted figure. Latency, multi-turn drift and prompt caching belong in the comparison too.

   Read chapter
8. 8

   Breaking a tool-call retry loop

   Fourteen create_ticket calls in one turn, each timing out after the backend had committed, each with a slightly reworded summary — so a framework that deduplicates identical calls never fired and fourteen tickets exist. Key attempts on the fields that determine the side effect plus the error class, cap them hard, and end the turn with an explicit terminal action and tool_choice none, because refusing the call without ending the turn sends the model to a neighbouring tool.

   Read chapter