Selection failures and argument failures

A language model, left to itself, emits text and nothing else. It cannot read an orders table, move money, or find out that an order came back on the 3rd, because none of that is in its weights and none of it arrived with the question. What closes the gap is an arrangement in which the model is handed a catalogue — the list of operations it is allowed to ask for, each one a name, a sentence or two of description, and a JSON Schema declaring its parameters — and is permitted to end its turn by requesting one of them instead of replying. That request, naming a single tool and supplying an object of arguments, is a tool call. Your code receives it, runs the real function behind the name, and feeds the result back into the conversation, and the model continues with something it had no way of knowing a moment earlier. Providers ship the mechanism under two names, tool use and function calling, and they mean the same thing.

You meet it wherever a model has to do something rather than say something. It is the tools parameter on a chat completions request, the tool definitions registered in an agent framework, and the catalogue published by an MCP server — the Model Context Protocol, a standard interface for exposing a set of tools to any agent that speaks it, and the reason a single agent can arrive at sixty tools by the end of this topic without anyone deciding to give it sixty. A support agent built this way is a loop around that exchange: send the conversation and the catalogue, receive either prose or a tool call, execute the call, append the result, send again. Two decisions sit inside every request of that kind, and the model makes both of them unaided. It chooses which tool from the catalogue, and it produces the values that fill that tool's arguments. A tool-call failure mode is one of the ways either decision comes out wrong — a call the model was free to make, that the transport accepted, that the backend served correctly, and that should never have been made in that form.

Two names, because two mechanisms#

Here is the first of two traces from the same support agent, trimmed to the tool calls. Neither produced an exception, a non-2xx response, or a single line in an error dashboard.

{
  "user": "I returned the boots on the 3rd and I still haven't seen the refund. Order 88214.",
  "calls": [
    {
      "tool": "search_orders",
      "arguments": {"query": "boots refund", "customer_id": "c_40912"},
      "result": {"orders": [], "total": 0},
      "status": 200,
      "latency_ms": 91
    }
  ],
  "final_message": "I can't find any record of a return for boots on your account."
}

Every argument in that call is well-formed. query is a string, customer_id exists and belongs to the person asking, the backend returned a syntactically perfect result, and the search index genuinely holds no full-text match for "boots refund" because it indexes product titles and order status, not return events. The correct call was get_order with order_id: "88214", followed by get_return_status. The user was told, in a confident sentence, that a return they had made did not exist.

The second trace, from a different conversation:

{
  "user": "This blender arrived and it's just not what I expected from the photos. I want my money back.",
  "calls": [
    {
      "tool": "refund_order",
      "arguments": {
        "order_id": "91733",
        "reason_code": "CUSTOMER_UNHAPPY",
        "amount_cents": 8499
      },
      "result": {"refund_id": "rf_5512", "state": "pending", "reason_code": "OTHER"},
      "status": 200,
      "latency_ms": 340
    }
  ],
  "final_message": "I've started a refund of $84.99 for the blender."
}

The refund is correct. The money moves, the customer is happy, the conversation closes cleanly. What went wrong is CUSTOMER_UNHAPPY, which is not one of the six reason codes the refunds service actually recognises — those are DAMAGED_ON_ARRIVAL, WRONG_ITEM_SHIPPED, LATE_DELIVERY, NOT_AS_DESCRIBED, DUPLICATE_CHARGE and OTHER. The schema declares reason_code as a plain string, the service accepts any string and maps anything it does not recognise onto OTHER, and so a refund that should have been coded NOT_AS_DESCRIBED landed in the bucket that merchandising ignores. The vendor whose photos are misleading keeps shipping the same blender.

The first trace is a selection failure: the model picked the wrong tool from the catalogue. The second is an argument failure: the model picked the right tool and invented a value for one of its parameters. The mechanisms diverge at the point where the model is generating tokens. When it emits reason_code, it is producing a value into a slot whose type is declared and whose legal contents are, in principle, knowable — there is an authoritative set of six strings sitting in the refunds service. Nothing in the request told the model what that set was, so it produced a token sequence that reads like a reason code, which is exactly what a language model does with an unconstrained string field. When it emits "tool": "search_orders", it is choosing among names and descriptions that were all offered as equally legal. There is no slot, no type, and no authoritative set to check the choice against, because every tool in the catalogue is a permitted answer.

Why the model reaches for CUSTOMER_UNHAPPY in particular is worth walking, because it is the step that makes the behaviour predictable instead of capricious. The model is producing the next tokens that fit what it has read. What it has read is a customer describing a product that did not match its photographs, a tool named refund_order, and a parameter named reason_code declared as a string. Those three things tell it the shape of what goes in the slot: something that reads like a machine-readable reason, in the register such things are written in. What is nowhere in the request is the list of six strings the refunds service will branch on, because nothing put it there. So the highest-scoring continuation is a well-formed-looking code that describes the situation accurately, which is exactly what a competent writer would produce given the same evidence. The failure is not the model reasoning badly. It is the model reasoning correctly from a request that never contained the answer.

The same walk explains why a selection failure cannot be repaired by supplying information. The evidence that would settle which tool is right — that a return event lives in one system and a full-text index in another, and that this customer's sentence is about the first — is not a set anybody can attach to the request. It is a judgement about intent, and there is no field to put it in.

The fix for B is close to trivial and takes one line. Declaring the parameter as an enum moves the failure from a silent remapping into a hard rejection at the boundary, and with a schema-constrained provider — one that narrows, at each generated token, the set of tokens the model is allowed to emit so that only a string satisfying the schema can be formed — it moves further still, into a decode that cannot produce the bad value at all:

{
  "name": "refund_order",
  "description": "Issue a refund against a delivered order.",
  "input_schema": {
    "type": "object",
    "required": ["order_id", "reason_code", "amount_cents"],
    "properties": {
      "order_id": {"type": "string"},
      "amount_cents": {"type": "integer", "minimum": 1},
      "reason_code": {
        "type": "string",
        "enum": ["DAMAGED_ON_ARRIVAL", "WRONG_ITEM_SHIPPED",
                 "LATE_DELIVERY", "NOT_AS_DESCRIBED",
                 "DUPLICATE_CHARGE", "OTHER"]
      }
    }
  }
}

There is no equivalent line for A. You can rewrite search_orders's description until it is a paragraph long, you can add a when_to_use field, and none of it is a constraint — it is a hint that competes with every other hint in the catalogue at generation time. Topic 5.10, on tool granularity and surface, covered how to shape descriptions and where to draw the granularity line, and that work genuinely reduces selection error. What it does not do is convert selection error into something a validator can reject, because a validator would have to know the user's intent to know that get_order was correct and search_orders was not. Intent is the thing being inferred, and there is no oracle for it outside a labelled set that a human wrote.

The two ways of applying that enum are worth telling apart before either is shipped, because they differ in what they leave behind. Validating at your own boundary lets the model emit the bad value and then refuses it, so the bad value exists for long enough to be counted, and its rate is a number that moves when a description changes. Constraining the decode stops the bad value from being formed at all, which is strictly better for the backend and removes the measurement with it: there is no out-of-set emission to record because the model was never permitted to produce one, and a metric on it will read zero whether the underlying behaviour improved or got much worse. Neither is wrong. What is wrong is switching the second one on and then reading the resulting zero as evidence about the model.

A third thing the word covers#

The taxonomy has one more member, and it belongs here because it behaves unlike either of the two above. In the same month's traces the agent emitted nine calls to get_refund_status, a tool that is not in the catalogue and never has been. The name is a reasonable guess — the catalogue holds get_return_status and refund_order, and something that reports on the state of a refund is the obvious sibling for either — and the model produced it as a continuation rather than by selecting from the list it was handed.

class UnknownTool(Exception):
    pass

def dispatch(name, args):
    handler = REGISTRY.get(name)
    if handler is None:
        raise UnknownTool(f"{name!r} not in catalogue ({len(REGISTRY)} tools)")
    return handler(**args)

That failure is loud in the way the other two are not. It raises before a single argument is deserialised, the exception carries the invented name, and it lands in the same place as every other exception the service throws, so those nine calls were on a dashboard the following morning. Providers that constrain decoding to the set of declared tool names make it structurally impossible, which is why it is close to extinct on some stacks and routine on any path that parses tool calls out of free text.

Three mechanisms travel under one word. A ghost tool is a name that is not in the catalogue; a misroute is a name that is in the catalogue and is the wrong one; an invented argument is a value that satisfies the schema and does not exist in the backend. They are caught, respectively, by a registry lookup, by nothing you currently own, and by a check somebody has to build. "Hallucination" is true of all three and useful for none of them, because it describes the model's state rather than the artefact that would have stopped the call.

The monitoring asymmetry decides where the effort goes#

As shipped, neither of the two traces produces an error. Both calls returned 200 in under half a second, both conversations ended with the assistant sending a fluent message, and any dashboard built on exception rate, 5xx count or tool latency shows a flat green line through both. With only the ordinary service instrumentation, you learn about A when a customer replies "that's not true, here's my return label" and about B when someone in merchandising asks why 40% of refunds are coded OTHER, six weeks later.

The asymmetry appears once you start fixing things. Add the enum, and B becomes loud: the provider rejects the call or your boundary throws a validation error, that error carries a stack trace and a tool name, it lands with every other exception, and by Monday it has a count and a graph. B is a failure mode that can be converted into a monitored one by a change to a schema. A cannot, by any schema, ever be converted into a monitored one, because the call was legal and no declarative artefact in the system encodes which of your twelve tools was the right answer to a sentence about boots.

Teams instrument what throws. When the two failure modes are collapsed under "hallucination", the work that gets done is the work that produces a visible number — enums, validators, retry wrappers, argument guards — and the dashboard afterwards genuinely does look better, because the class of failure that was always capable of being loud is now loud and shrinking. Meanwhile the misroutes continue at exactly the rate they did before, returning confident, well-formed, wrong answers, and the only thing that changed is that the team now believes it has addressed tool-call reliability.

The temptation to collapse them is not laziness. From the outside the two traces really do look alike: in each case the model produced something that was not grounded in the system it was talking to, and in each case the user got a bad outcome from a call that returned 200. The distinction only becomes visible when you ask what artefact would have prevented it, and the answers are a JSON Schema keyword for one and a measurement programme for the other.

Different victims, different discovery times#

Keeping them apart matters beyond having different fixes, because they also have different sizes and different victims, and both facts vanish when a single figure is reported for tool-call errors. On this deployment, 200 conversations were sampled from the escalation queue — which oversamples failure on purpose, so these are not rates over all traffic — hand-labelled, and every tool call in each one classified.

Three conversations contained one of each and are counted in both columns. The two shapes are close to complementary: a misroute damages the answer, which the person reading it has some chance of disputing, while an invented argument damages a record, which nobody reads until a quarterly report comes out wrong. The 41-day median in the last row is not a discovery latency in any useful sense — it is how long it took someone in merchandising to run a report — and the honest description of that cell is that the discovery mechanism was luck. Neither column contains a single defect that the service's own monitoring surfaced.

Two things about that table have to be read carefully or it will be quoted wrongly. The columns overlap, since three conversations appear in both, so the count of affected conversations is the two column heads added together and the overlap taken back out once, which leaves fewer than forty affected conversations out of the two hundred sampled. Adding the columns without that subtraction double-counts, and it is the standard way a labelled sample turns into an inflated headline. The rows are conditional rates inside their own column and not shares of the sample either: four out of five selection errors told a user something false, and every single argument error corrupted a record, which are two facts about two different denominators.

The last row rewards the same care, because it is not measuring detection. If a defect can only be found when a quarterly report is run, and defects arrive at no particular time relative to that report, then the wait is however long remains until the next one — which averages out to about half the gap between reports. Half of a quarter is somewhere around a month and a half, and the observed median sits a little under that. The figure is therefore an approximate measurement of the reporting calendar, and it would move by weeks if merchandising switched to monthly reports without anything about the agent changing at all.

Large value sets, constrained decoding and labelled data#

Fixing B properly is more than the enum, which is why it takes the next two chapters rather than a footnote. An enum on a six-member set is easy; the same problem on component_id, where the authoritative set is nine hundred rows in a table that changes weekly, cannot be an enum in a static tool definition, and the model will still produce plausible values like checkout-service when the real row is svc-checkout-api. Before any of that there is a harder problem: one anecdote about CUSTOMER_UNHAPPY, found by accident, does not tell you whether it happens on 0.4% of refunds or 30% of them, and you cannot find out by searching logs for errors that were never written.

Constrained decoding is worth ruling out explicitly, because it is the first thing suggested whenever these three get discussed together and it genuinely does eliminate one of them. A decoder restricted to the declared tool names makes ghost tools impossible and, applied to arguments, makes an out-of-enum value impossible. Applied to selection it does nothing at all, because search_orders is a declared name: the constraint is satisfied by every wrong answer as thoroughly as by the right one. A mechanism that narrows the output space cannot help when the output space was already exactly the set of legal tools and the mistake was choosing within it.

A is the longer road, and the shape of the road is set by what the failure is made of. There is no oracle, so there has to be a labelled set; the labelled set has to survive the catalogue growing, which means the measurement has to be designed rather than run; and the two mitigations teams reach for first — namespacing and dynamic tool loading — each recover a specific fraction of the loss and each introduce a failure of their own that the obvious way of scoring them hides.

The two threads meet again at the end of this topic, in a way worth flagging now. A tool that fails in a way the model cannot interpret produces a retry loop, and a model's response to being stuck is frequently to reach for a neighbouring tool — which is a selection failure caused by an argument failure, and the reason neither of these can be treated as somebody else's problem.