Partitioning a confusable pair

A B2B support agent has two tools that read customer data from two different systems. get_customer hits the CRM — the customer relationship management system, where sales keeps who a customer is and who owns the relationship — authenticated with a service account scoped to the sales org. lookup_account hits the billing platform, authenticated with a separate key, and returns things the CRM has never held. Neither is wrong about itself:

[
  {
    "name": "get_customer",
    "description": "Retrieve a customer record from the CRM. Returns name, company, contact details, account owner, lifecycle stage, and a summary of recent activity."
  },
  {
    "name": "lookup_account",
    "description": "Look up an account in the billing system. Returns account status, plan, billing contact, payment details and invoice history."
  }
]

Both are accurate. Both are current. Neither carries a stale example. On a labelled sample of 600 real support requests, the agent routes to the correct one 71% of the time. The 174 errors are not spread evenly — 4% of them are wild, and the rest sit in a band of requests that a reasonable person reading both descriptions would also find hard to assign. Words like contact, account, status, and history appear in both strings with different meanings, and roughly 40% of what a support agent asks in a shift touches something that could be either. The move that fixed the search_orders and search_refunds pair in chapter 1 applies here unchanged; what that case did not show is how you work out where the boundary goes when the overlap is large and the two tools are genuinely different systems rather than two views of the same domain.

Seven in ten of six hundred requests is four hundred and twenty-six routed correctly, which leaves the hundred and seventy-four. Users do not know which of two internal systems holds the answer to their question, and nothing about how they phrase a request obliges them to indicate it, which is why those errors concentrate in a band instead of scattering. The pair in chapter 1, search_orders against search_refunds, had the same property over a shorter distance: order status and shipments on one side, money going back to the customer on the other, and a user asking about a charge on an order standing exactly between them.

A good self-description is not a discriminator#

Descriptions get written one tool at a time, usually by whoever built the tool, usually reviewed by asking is this an accurate account of what this tool does. That is the wrong objective, or rather it is a necessary condition that is nowhere near sufficient, because selection is comparative. The model is not asking whether get_customer is a good match for the request; it is ranking get_customer against lookup_account and nine other candidates and taking the best. A description written in isolation optimises for a question nobody asks it.

The practical consequence is that the two tools most likely to be confused are also the two whose descriptions are most likely to be individually fine, because nothing in the review of either one ever put them side by side. You find them in the logs rather than in code review. The cheapest way to find them is to take a month of tool_use events, label a sample for correctness, and tabulate errors as an ordered pair of (chosen, correct). Misroutes concentrate: on this agent, 84% of all routing errors fell into three (chosen, correct) pairs, and get_customer → lookup_account was over half on its own.

The query is a group-by over a table you have to be logging in the first place, which is the only part of this that takes any setup:

select chosen_tool, correct_tool, count(*) as n
from routing_labels
where label_batch = '2026-07' and chosen_tool is distinct from correct_tool
group by 1, 2
order by n desc;

Those four rows add back to the hundred and seventy-four, which is the first thing to check on a table like this and the reason its share column can be trusted. Ninety-one out of a hundred and seventy-four is a bit over half; thirty-eight is a bit over a fifth; seventeen is a tenth; and the twenty-eight scattered across nine other pairs is the remaining sixth. The top three rows between them account for five sixths of everything. Confusable pairs are found this way rather than by reading code, and this arithmetic is the point at which a vague sense that routing is unreliable turns into a claim about one boundary.

Two things in that shape are worth reading before writing any prose. The concentration says this is one boundary rather than a general reliability problem, so an intervention aimed at the top pair is aimed at half the errors. The asymmetry says which side is over-claiming: get_customer takes 91 requests that belong elsewhere and loses 38, so its description is the broader of the two and its positive claim is the one that needs narrowing first. Both facts come from labelled data rather than from complaints, and the difference matters here more than usual, because a misroute that returns a real customer record produces an answer that reads as authoritative and almost never generates a support ticket. Error rates inferred from user dissatisfaction on this pair ran at about a fifth of the labelled rate.

Finding the axis, then writing it into both strings#

Once you know which pair, the work is to find the sentence that separates them, and the raw material is the contested traffic rather than the API surface. Pull the requests where the two tools compete, label each with the tool that should own it, and look for what the labels have in common. For this pair, 200 contested requests fell out like this:

The axis is who the customer is against what the account owes and is entitled to. That sentence did not exist anywhere before the exercise, because no single tool's documentation had reason to state it. Writing it down is most of the work; the descriptions then just say it twice, from each side:

[
  {
    "name": "get_customer",
    "description": "Look up who a customer is, from the CRM. Use this for contact details, company, which rep owns the account, lifecycle stage, and the history of interactions and support tickets. Use it to confirm a customer exists and to find the right person to talk to. It knows nothing about money: for balances, invoices, plan, seats, renewal dates or payment method, use lookup_account."
  },
  {
    "name": "lookup_account",
    "description": "Look up what an account owes and is entitled to, from the billing system. Use this for outstanding balance, invoices and their status, current plan and seat count, renewal date, payment method, failed payments and dunning state, and for whether an account is in good standing or past due. It holds no relationship or support history: for contact details, account ownership or past tickets, use get_customer."
  }
]

Three things carry the weight. Each description opens with the axis rather than with the backend it hits, because the user asking the question does not know or care which system holds the answer. Each enumerates the concrete nouns of its own side — dunning, the automated sequence of chase emails a billing system sends after a payment fails, along with seat count and renewal date — so the match is against words a request will actually contain. And each ends by naming the other tool and handing off the cases it does not own, which is what converts two self-descriptions into a partition.

The ambiguous middle is settled by decision rather than by cleverness. Is this customer in good standing has no correct answer in the abstract; it has an answer once you decide that "good standing" is a billing concept at this company, write that decision into lookup_account, and stop treating the resulting misroutes as a model problem. Replayed against the 600-request sample, the rewrite took selection accuracy from 71% to 93%; the residual 7% is the 42 requests the last two sections of this chapter take apart.

Points and counts are worth keeping in step for the rest of this chapter, because one point of accuracy here is six requests and every interesting quantity below is small. Twenty-two points of improvement is a hundred and thirty-two requests recovered, which is close to the hundred and twenty-nine sitting in the top two rows of the error table — so the paired rewrite fixed the boundary it was aimed at and left the third row alone, which is what a targeted intervention is supposed to look like. The residual seven points is the forty-two, and forty-two is small enough that one mislabelled request moves it by a sixth of a point.

Three interventions, and why the middle one is usually right#

Rewriting both descriptions is not the only available move, and it is worth knowing what the alternatives buy.

The rename is worth calling out because it is nearly free and it is the highest-value token in the whole definition. A name is read at selection time, sits at the top of every rendered tool entry, and is a handful of tokens; lookup_account was ambiguous in exactly the way the pair was ambiguous, since "account" means the customer relationship in the CRM and the billing entity in the billing system. Rename before you write, always. It will not carry a boundary on its own — seven points is not enough — but it makes the description's job smaller.

Rewriting one side is the intervention teams reach for, because one tool is "the one that keeps getting picked wrongly" and it feels like the tool with the bug. It leaves the other description still claiming the contested band with the same confidence as before, so the model is now choosing between one string that says not this and one that says nothing, and the ten-point shortfall against the paired rewrite is the cost of that asymmetry. Exclusions only work when the tool receiving the handoff also claims it.

The mechanism is in the word choosing. A ranking does not reward the candidate that has been argued down; it rewards whichever candidate scores highest, and lowering one score by adding a prohibition leaves the outcome to be settled by whatever the other candidate happens to say. When the other candidate says nothing about the contested band, it goes on matching the shared nouns — account, status, history — with exactly the confidence it had before. Sixty requests in six hundred is what that asymmetry costs on this pair. A paired rewrite works because it moves both scores at once, in opposite directions, along the same axis.

When the name and the description disagree, the name wins#

The 42 requests still misrouted after the paired rewrite had a pattern in them. Fourteen contained the literal word account in a sense the CRM owns — pull up this account, who looks after the Northwind account, is this account still active — and every one of them went to lookup_account, whose new description explicitly hands relationship questions to get_customer. The description was read, was current, and was outvoted by four tokens sitting above it.

That is the expected outcome given how the two strings are weighted. The name appears at the head of the tool's entry, is what the model emits when it commits, and is repeated in every prior tool_use and tool_result block in the transcript, so a conversation eight turns deep contains the string lookup_account a dozen times and its description exactly once. A surface-form match between the request and a name that is repeated that often is a strong signal, and a sentence of prose asking the model to disregard it is a weak counterweight. Renaming the tool to get_billing_account after the rewrite took the pair from 93% to 95% on the same sample, and twelve of the fourteen name-driven errors went with it.

Two points of six hundred requests is twelve requests, which is where that second figure comes from rather than from a separate count: the rename recovered all but two of the fourteen and nothing else moved. The correspondence is tight enough to be evidence rather than coincidence. It says the fourteen were one mechanism and not fourteen unrelated confusions, and it says the mechanism was the noun in the name, because the noun in the name is the only thing that differed between the two runs.

The ordering follows from that. Rename first, write the partition second, because a description whose first job is to undo its own name is spending tokens on a problem that four characters would have removed, and it is spending them on every turn of every conversation. The check is mechanical enough to run during review: take the twenty most common phrasings from the contested traffic, and ask whether any of them shares a distinctive noun with a tool that should not own them. When one does, the noun belongs to whichever tool the users mean by it, and the other tool needs a different word.

Requests that need both tools#

Accounting for the whole residual before working on any part of it is what stops the last few points being chased forever. Fourteen of the forty-two were driven by the name and left with the rename. Twenty-four are the ones this section takes apart. That leaves four unclassified requests out of six hundred, which is below the level at which any change to a description could be evaluated, since a difference of four requests sits inside the noise of the labelling itself. A residual that decomposes like that is finished work. A residual of forty-two that nobody has decomposed is an open question wearing a percentage.

Twenty-four of the requests still failing after all of that are a different category, and it is worth separating them from the routing problem so they stop polluting the measurement. Most were questions whose answer needs both systems — is this a paying customer and who is their rep, why did their card fail and who should I call about it — where any single choice is wrong by construction. A description can help at the margin by stating the order of operations, and adding if you need both identity and billing information, call get_customer first to resolve the customer id, then pass it to get_billing_account recovered five of the 24, because the failures there were the model stopping after one call rather than choosing the wrong one.

The other nineteen are not a description problem at all. They are requests for a view that neither backend holds, which is a question about the tool surface rather than about the strings — the territory of topic 5.10, on tool granularity and surface, and the sort of finding that argues for a small composed tool that reads both systems rather than a third clause in either description.

The partition move also does not fix a pair that should not be two tools. If, after working the contested traffic, the axis you write down is something like use A for simple lookups and B for complicated ones, you have not found a boundary; you have found two implementations of one capability, and the fix is on the surface rather than in the prose. No description will paper over it — a rule the model cannot evaluate from the request alone is not a rule.

Nor does it fix a boundary that depends on data the model does not have yet. Use get_enterprise_contract for enterprise customers and get_contract for everyone else is a perfectly clear sentence and an unusable description, because the model cannot know the customer's tier until it has looked something up. That distinction is a signal about the tool surface rather than about the writing, and chapter 7 is about reading that signal correctly instead of writing another clause.

A partition is two claims, not one claim and one prohibition. Each tool says what it owns in the words users use, then names its neighbour for what it hands over, and the ambiguous middle gets assigned by decision rather than left for the model to resolve. The name goes first because it outranks the prose; the paired rewrite goes second because a one-sided one leaves the other tool competing at full confidence.

The whole framing so far has been which of these. There is a third option the model treats as a choice even though it has no name and no description: calling nothing at all, or calling a tool before it has what the tool needs. That option is where invented arguments come from.