Tool count and selection accuracy

Selection accuracy is the share of eval tasks on which the model called the tool the task actually needed, scored against a labelled answer key rather than against whether the run eventually produced something. The calls it counts against you are misroutes: a call that landed on a tool other than the one the task required, whether or not the run recovered afterwards. Both definitions matter here because a surface that grows one reasonable tool at a time degrades on exactly this number while every other signal stays healthy — the tools work, the schemas validate, the runs finish.

The reason is arithmetic rather than wording, and it is the subject of this chapter along with the three things that follow from it: how to compare two candidate surfaces without over-reading a completion count, how to record the operations you decided not to expose, and how to remove a tool that already has callers.

Why the eighth addition costs more than the second#

When you add the twelfth tool to eleven existing ones, you create eleven new comparisons the model has to get right, and none of the existing fifty-five go away.

The formula behind that middle column is worth deriving once rather than trusting, because the derivation is what makes the third column obvious. Line the tools up. Each of the tools can be compared against every other one, which is all of them but itself, so there are as many comparisons as tools multiplied by one fewer than that. But every comparison has now been counted from both ends — the twelfth against the third and the third against the twelfth are the same comparison — so halve it. That is the whole of n times n minus one, over two, and the halving step is the only one anybody skips.

Now the third column falls out with no arithmetic at all. When a new tool joins, it forms a comparison with each tool already registered and with no others, so it creates exactly as many new pairs as there were tools before it. That is why the entry for nineteen tools reads eighteen. It also gives a check on the second column that costs nothing: add up the new pairs created by every addition from the first tool onward — one, then two, then three, and so on — and the running total has to equal the pair count, because every pair was created by exactly one addition, the later of its two members. At five tools that sum is one plus two plus three plus four, which is the ten in the table, with nothing left over. A column that closes on its own total this way is worth constructing deliberately: it means the growth law and the per-addition cost are two readings of one fact rather than two claims to be believed separately.

A deployment server that went from eleven tools to nineteen over two quarters grew its surface by 73% and its pair count by 211%, and the accuracy number went the other way while the eval set was itself growing.

The two growth figures in the sentence above that table come from dividing the ends of the first column, and they are worth doing because the gap between them is the point. Nineteen over eleven is about one and three-quarters, which is the seventy-three per cent rise in tools. For the pairs, nineteen times eighteen is three hundred and forty-two and eleven times ten is one hundred and ten; the halving appears on both sides of the division and cancels, so the ratio is three hundred and forty-two over one hundred and ten, a shade over three — which is the two hundred and eleven per cent rise. Those two results are the same fact seen twice, because pairs grow with the square of the count: one and three-quarters squared is very close to three. That is the sentence to keep. A surface that grows by three-quarters roughly triples what has to be told apart.

The accuracy column repays the same treatment, and this one changes the story rather than confirming it, because the denominator did not hold still. Thirteen percentage points sounds like an erosion. Convert each row back into a count of misroutes and it is not: six per cent of forty tasks is between two and three misroutes in the first quarter, twelve per cent of fifty-two is a little over six in the second, and nineteen per cent of sixty-four is a little over twelve in the third. The number of tasks the model got wrong per pass went up roughly fivefold while the rate went up by thirteen points, and it is the count that an engineer chasing individual misroutes actually experiences. Percentage points are not subtractable across rows with different denominators, and comparing them here would have made a six-fold rise look like a slide.

Most of those pairs are harmless, since nobody confuses get_logs with create_environment, and the growth law is an upper bound on the disambiguation work rather than a prediction of the accuracy curve. What matters is the subset of pairs whose task footprints overlap in the sense established in the previous chapter — pairs where some task in scope has both as a defensible answer. On this server that subset was 6 pairs at eleven tools and 23 pairs at nineteen, because the additions were not drawn uniformly from the space of possible operations. They clustered around the operations people ask about most, which is exactly where the existing tools already were. restart_service, redeploy, rollback and recycle_pods all arrived separately, each for a real request, and any task phrased as "the payments API is wedged, fix it" has four defensible answers.

There is a second cost that compounds with the first. Each disambiguating clause added to a description — "use this only when the user explicitly asks to roll back to a previous release, not to redeploy the current one" — is text that sits in context for every call, including the 90% of calls where that tool is irrelevant. The 4,100 tokens of definitions in Q3 include roughly 900 tokens of clauses that exist purely to fence tools off from each other. Disambiguation is paid for in the same budget as the task.

Nine hundred out of four thousand one hundred is a little over a fifth, and putting it that way is what makes it a design fact rather than a footnote. Better than one token in five of the tool definitions describes no capability at all. It exists to stop one tool being mistaken for another, which means it is the surface paying interest on its own shape, on every request, whether or not either of the fenced tools is relevant to what was asked. The usual instinct on seeing a large definition budget is to compress the descriptions. The arithmetic points somewhere else: compressing the eighty per cent that describes capability saves less than removing the reason the other fifth had to be written.

Running two surfaces against the same tasks#

The comparison worth running is not "is 19 too many" but two concrete candidate surfaces against one fixed set of representative tasks. Surface A mirrors the CLI with 14 tools, having already dropped the five nobody used. Surface B is a deliberately reduced set of 5: query_deployments, deploy, scale, get_logs, set_traffic. Both run against the same 30 tasks drawn from real requests in the team's support channel.

A completes 29 of 30. B completes 28.

The temptation is to read that as A winning by one and ship A, which the sample size does not support. With 30 tasks, one task is a little over three points, so the headline gap between the two surfaces is three points measured on thirty observations. It is worth counting how many observations that gap actually rests on. A failed one task and B failed two, so the tasks where exactly one of them succeeded number either one or three, depending on whether the task A failed is also among the two B failed. A paired comparison resting on that many disagreements has no power to distinguish a real 3-point gap from a coin flip, and you would need several hundred tasks before a one-point difference in completion rate meant anything. Completion counts are weak evidence, and the useful information is in what the runs did on the way to those counts.

Two rows of that table can be divided into each other, and the result is the most useful thing in it. Divide each surface's mean wall-clock by its mean number of calls and you get the mean time per call: forty-one seconds over four point eight calls is about eight and a half seconds, and twenty-six over three point one is also about eight and a half. Two surfaces, measured independently, landing on the same per-call figure. What that closes is the question of where the fifteen-second difference came from. If B's calls were individually faster — a lighter tool, a cheaper query — the per-call figures would differ and the time saving would be about the tools. They do not differ, so the entire difference is the number of calls, and the number of calls is what a misroute inflates: a wrong selection costs a call, its result, and then the corrective call. The wall-clock row therefore carries no information the calls-per-task row does not already carry, which is worth knowing before anybody argues about latency as a separate benefit.

The wrong-selection row repays the same division. Eleven wrong selections across thirty tasks is a bit over a third of a misroute per task; two across thirty is under a tenth, a difference of roughly five times. That gap rests on thirteen observations rather than on the one or three the completion row rests on, which is why it is the row to argue from even though it is not the row the eye goes to first.

Two questions decide it. The first is whether A's extra completion came from a capability B lacks or from a lucky selection. Task 22 was "roll back the payments API to yesterday's build"; A completed it with rollback, and B failed because deploy requires an explicit version and the model had no way to learn what yesterday's version was. That is a genuine gap, and it is a discoverability gap rather than a capability one, because query_deployments already returns release identifiers with timestamps — the model simply never connected the two. Adding one sentence to deploy's description pointing at query_deployments for version ids closed it on a re-run. A capability gap would have argued for a sixth tool; this argued for a sentence.

The second question is what A's misroutes touched. Eleven wrong selections is not just a slower path to the same answer, since four of them invoked a write operation before the run recovered — a restart_service on the wrong service, twice a scale that then had to be scaled back. B's two misroutes were both reads. That difference does not show up in a completion count at all, and it is the one that would show up in an incident review.

Writing down what you left out#

A reduced surface is a set of decisions about what not to expose, and those decisions are invisible six months later unless you record them next to the code. The omissions log is a table in the server's repository, updated in the same commit that removes or declines a tool.

Each row answers the question a reviewer will actually ask, which is "how do I do X now". The exec row is the one worth writing carefully, because a deliberate gap that nobody documented gets re-added by the next engineer who hits it on a Friday. Two of these rows also reduce permission exposure rather than ambiguity: a server with no exec tool cannot be talked into running a shell command, regardless of how good its other schemas are.

Removing a tool that two teams already call#

Surface B deletes restart_service and promote, and both have callers. The platform team's runbook agent — the assistant on-call engineers talk to instead of running the deploy CLI by hand — uses restart_service, and the release automation calls promote on every merge to main. Deleting them is a rollout with a schedule, not a design decision you land in one pull request.

The mechanism that makes this cheap is that a tool name and a tool implementation are separate things. The old name can keep working by translating into the consolidated call, and it can keep working while no longer appearing in the list the model sees.

const RETIRED: Record<string, (args: Record<string, unknown>) => ToolCall> = {
  restart_service: (a) => ({
    name: "deploy",
    arguments: { service: a.service, environment: a.environment, strategy: "restart" },
  }),
  promote: (a) => ({
    name: "deploy",
    arguments: { service: a.service, environment: a.to, from_environment: a.from },
  }),
};

// tools/list returns only the live surface; tools/call still honours retired names.
server.onListTools(() => LIVE_TOOLS);

server.onCallTool(async (req) => {
  const retired = RETIRED[req.name];
  if (!retired) return dispatch(req);

  const translated = retired(req.arguments);
  metrics.increment("retired_tool_call", { name: req.name, client: req.clientId });
  const result = await dispatch(translated);
  return {
    ...result,
    _deprecation: {
      retired_tool: req.name,
      replacement: translated.name,
      equivalent_arguments: translated.arguments,
      removal_date: "2026-11-01",
    },
  };
});

The _deprecation block matters because the caller might be an agent rather than a script, and an agent that reads equivalent_arguments in a successful response will frequently use the new form on its next call without anyone editing a prompt. The metric matters because it tells you when the window can close: the removal date is a placeholder until the per-client call count for a retired name has been zero for a fortnight.

Consequence, and what a misroute can reach#

Fewer tools is not a goal in itself, and the growth law does not say that a five-tool surface beats a fourteen-tool surface in general. It says that the cost of an addition is proportional to what is already registered. Plenty of servers legitimately expose thirty tools because they front thirty genuinely distinct operations with no overlapping task footprints, and their selection accuracy is fine.

The argument also says nothing about consequence. Surface B's two misroutes were both reads, which is a fact about that run rather than a property the design guarantees, and 81% accuracy over 64 tasks means roughly twelve wrong selections per pass through the eval set no matter how the surface is shaped. Reducing the tool count changes how often a selection goes wrong.

Both ends of that sentence come out of figures already on the page, and putting them in the same units is what makes the limit visible. Nineteen per cent of sixty-four tasks is a little over twelve, which is the Q3 figure. Apply Surface B's observed rate instead — two misroutes in thirty tasks, a little under seven per cent — to the same sixty-four tasks, and the expected count is between four and five. So the redesign is a change from about twelve wrong selections a pass to about four, which is a two-thirds reduction and is genuinely worth having. It is also not zero, and the arithmetic makes plain that it was never going to be: every term in it is a rate multiplied by a task count, and no rate available here is small enough that multiplying it by a working week's traffic gives nothing.

That distinction is why the pair count is a design tool and not a safety argument. Every number in this chapter is a frequency: how often the right tool wins, how many comparisons the model has to get right, how many tokens the fencing clauses cost. None of them describes what happens on the occasions the model still gets it wrong, and those occasions do not go to zero — they go from twelve a pass to four or five.

The service that got restarted last month was not primarily an accuracy statistic. It was evidence that nothing bounded what a wrong selection could reach, which is a separate design problem with separate mechanisms, and the one worth taking up next.