Consolidating a tool surface

A description has one job at selection time, which is to be a better match for the user's question than every other entry in the request. With four tools that is a comparison against three alternatives that are usually about different subject matter. With 34 tools generated from one REST API — one per endpoint, named after the operation ids, described with whatever the summary field contained — most of the alternatives are about the same subject matter.

That is a different problem from the one the previous chapter solved. search_runbooks had one serious competitor and could name it. A tool with eleven near-neighbours has no single neighbour to distinguish itself from, and every clause it adds is a distinction its siblings now also have to draw. Copywriting can carry a tool out of a two-way tie; it cannot carry a surface that was never designed, and the difference between those two situations is visible in the shape of the errors rather than in their number.

A tool surface here means the whole set a server exposes, considered together, because that is the unit the model reads. Nobody chose the 34 in this chapter. A generator produced one tool per REST endpoint, and the resulting surface inherited an API's resource decomposition — a division made for a developer who reads the reference once — without anyone deciding it was the right division for a reader choosing per call.

The burden is pairwise, and it grows faster than the tool count#

An OpenAPI document — the machine-readable file in which a REST API lists every endpoint it has along with each one's parameters, its operationId and a one-line summary — generates one tool per endpoint. Eleven of one such server's 34 tools are about orders:

A user asking "what did order 88213 contain and has it shipped" has to be routed by the model across a set where four entries are defensible and two are needed. The number of pairs a description has to survive rises with the square of the group's size, so going from four order tools to eleven does not make the problem three times harder for each description; it makes each description responsible for seven more distinctions it does not draw. Nothing in the generated text draws any of them, because the summary field in an OpenAPI document was written for a developer who would read the whole reference once and then hard-code a call, which is precisely the reader the first chapter of this topic contrasted with a model choosing per call.

The counting is worth doing, because "grows with the square" is the kind of phrase that gets nodded at and then not used. Every description has to separate itself from every other, so the distinctions the set as a whole must draw is the number of pairs in it, which is the tool count multiplied by one less than itself and halved. Four order tools make six pairs. Eleven make fifty-five. Tripling the count therefore multiplied the pairs by about nine, and from any one description's point of view the burden went from three neighbours to ten. Neither figure is large in isolation. Both are being asked of a paragraph that was generated from a summary field written for somebody else.

Writing 34 descriptions that each distinguish themselves from 33 siblings means every description has to contain a routing table, every routing table has to stay consistent with the other 33 as they change, and the whole set is re-read on every turn at a token cost the second chapter of this topic already quantified.

Default annotations as a diagnostic#

All 34 tools carry no annotations, so, by the defaults set out in the third chapter of this topic, every one of them is advertised as writing, destructive, non-idempotent and open-world — including getOrderStatus, which reads one enum value. A host with a confirmation policy therefore gates all of them equally, users learn to click through, and the gate stops meaning anything.

That is a real problem and it is not the cause of the misrouting, because annotations do not participate in selection at all. Their value here is diagnostic: a generator emitted 34 tools and nobody afterwards asked, of any single one of them, whether it reads or writes. The same absence of review is why the descriptions are summary strings. Fixing the annotations is correct, cheap, and will not change the misroute rate by a measurable amount, so doing it first and then reporting progress is how a quarter gets spent.

The reason it cannot move the misroute rate is mechanical rather than a matter of degree. Selection happens inside the model, out of the tool array in the provider request, and most hosts consume the annotations object in their own permission layer instead of forwarding it into that request. So the model never sees the flags and cannot be influenced by what they say. What the flags change is what a host does immediately before and after a selection, which is a real problem with a real symptom — the clicked-through dialog — and a different one. Both are worth fixing. They are simply not the same fix, and mistaking one for the other is what makes a two-line annotation diff feel like progress on the routing.

Designing the replacement surface#

The evidence to design against is usage rather than the API's resource decomposition. On this server the call logs showed that 9 of the 34 tools accounted for 96% of successful calls and 7 had never been called at all in six months, and one consuming team had already hard-coded four tool names in its own application code. Combining the hard-coded four, the top nine and the never-called seven gives a surface grounded in what people do.

Tool(
    name="get_order",
    description=(
        "Fetch one order and, optionally, its related records. Accepts either the "
        "internal order id (numeric, e.g. 88213) or the customer-facing reference "
        "(e.g. ORD-2026-4471) in the same argument; the format is detected. Use "
        "include to fetch line items, shipments, invoices, refunds, agent notes or "
        "the status-transition history in one call rather than making several. "
        "For finding orders when you do not have an identifier, use find_orders."
    ),
    inputSchema={
        "type": "object",
        "properties": {
            "order": {
                "type": "string",
                "description": "Internal id or customer reference. Not a customer name.",
            },
            "include": {
                "type": "array",
                "items": {
                    "type": "string",
                    "enum": ["items", "shipments", "invoice", "refunds", "notes", "history"],
                },
                "description": "Related records to include. Empty means header only.",
            },
        },
        "required": ["order"],
        "additionalProperties": False,
    },
    annotations=ToolAnnotations(readOnlyHint=True, openWorldHint=False),
)

Eleven order tools become two — this one and find_orders, which takes the filters searchOrders and listOrders split between them — and the distinction the model now has to draw is between having an identifier and not having one, which is a question the user's message answers directly. The whole server went from 34 tools to 9.

The number that defends the change#

A consolidation is a large, disruptive change that is easy to argue about on aesthetics, so it is worth arriving at the review with a measurement rather than an account of how much cleaner the surface is. The harness is a fixed set of tasks with a known correct first tool, replayed against both surfaces.

TASKS = [  # 120 tasks derived from six months of real transcripts
    {"prompt": "what did order 88213 contain and has it shipped?",
     "expect": {"get_order"}, "expect_args": {"include": ["items", "shipments"]}},
    {"prompt": "find orders for acme placed last week that are still unfulfilled",
     "expect": {"find_orders"}},
    {"prompt": "why was ORD-2026-4471 refunded?",
     "expect": {"get_order"}},
]


async def score(surface: list[Tool]) -> dict[str, float]:
    hits, misroutes, calls = 0, Counter(), 0
    for task in TASKS:
        trace = await run_task(task["prompt"], tools=surface)   # one model, fixed seed
        calls += len(trace.tool_calls)
        first = trace.tool_calls[0].name if trace.tool_calls else None
        if first in task["expect"]:
            hits += 1
        elif first is not None:
            misroutes[first] += 1
    return {"selection_accuracy": hits / len(TASKS),
            "calls_per_task": calls / len(TASKS),
            "misroutes": misroutes}

One row of that table says something the others do not, and it is easy to read past. The tool array fell to roughly a third of its former size while the tool count fell to roughly a quarter — which means each surviving description is longer than each description it replaced. The saving did not come from writing less. It came from writing fewer, and the extra room went on the routing information the generated entries never carried. That is the second chapter of this topic's arithmetic pointing the same way it did there: the cost of a description is small, and the cost of a listing is the count.

Which errors mean wording and which mean surface

The previous chapter closed on a signature it could not resolve from inside a single description: two full rewrites, no measurable movement, so look at the neighbours. The baseline run of the harness resolves it, because misroutes have a shape once you tabulate which tool the model picked against which tool the task needed. Over the 34-tool surface the mass of that matrix sat in six cells:

Two patterns there want telling apart. Confusion that runs in both directions between a pair, as getOrder and getOrderItems do by taking each other's tasks 14 times and 6 times, means the model cannot separate the two entries from the text it was given, and rewriting either one does not fix it because whatever clause you add to the first is a distinction the second now also has to draw. Confusion that runs one way into a single sink, as listOrders does by absorbing 31 selections across nine different kinds of task, is the wording problem of the previous chapter and usually has a nameable cause. Both were present here, which is why rewriting listOrders on its own would have moved the headline accuracy by a few points and left every reciprocal pair exactly where it was.

The reason a reciprocal pair resists rewriting is worth making explicit, because a team will try it before doing anything larger. Suppose a clause goes into getOrder saying it returns the header only. That clause is now a distinction getOrderItems also has to draw, from the other side, or the pair is no clearer than it was — so the edit is not one edit but two, and the second has to be kept consistent with the first as both descriptions change over time. Then do that for each remaining pair in the group. A sink is different in kind: one entry is absorbing selections it should not get, for a nameable reason such as being the shortest description in the set, and repairing that one entry repairs every task it was stealing. One is a property of a pair and the other is a property of an entry, and only the second has a local fix.

The count under listOrders is worth adding up for the same reason. Nineteen misroutes came from tasks that wanted searchOrders, and twelve more came from eight other kinds of task, so thirty-one selections across nine different kinds of question all landed on one tool. That concentration is the signature: a sink pulls from many sources, while a reciprocal pair trades with exactly one partner.

The per-tool misroute counts are the part that guides the next iteration. listOrders absorbing 19 wrong selections was a specific, fixable fact — it was the shortest description in the set and read as the generic entry point — whereas "the model picks the wrong tool constantly" supports no next step. Running the harness against a second model and a second host is a few minutes more and is what turns a local result into a claim about the server.

Retiring twenty-five tools without breaking the host that pinned four#

The migration is the unglamorous half. The fulfilment team's four hard-coded names have to keep working, so the old tools stay in the listing as aliases for a deprecation window, with descriptions that say to prefer the new tool, and the server emits notifications/tools/list_changed when the surface changes so connected sessions re-fetch rather than running on a stale list for the rest of the day.

Each transition there is gated by a counter rather than by a date on a roadmap, because a calendar deprecation removes tools that something is still calling. The server records calls per tool name and per client id, and a legacy tool leaves the listing only after 30 consecutive days with no call from any client other than the smoke test.

Each detail of that rule is carrying weight. The difference between a counter and a date is the difference between removing a tool nothing uses and removing one nothing used this quarter: a deadline is a statement about your roadmap, an absence of calls is a statement about your consumers, and only the second is evidence about whether removal is safe. The count has to be per client id as well as per tool name, because a single caller still using a name is enough to block removal and an aggregate would hide that caller behind everybody else's silence. And the smoke test has to be excluded for the mirror-image reason: it is your own traffic, so counting it would make the condition impossible to satisfy at all.

Twenty-one of the twenty-five cleared that bar within five weeks; the four pinned names took eleven, which was how long the fulfilment team needed to schedule their own change. The Dual stage is the expensive one, since the listing is briefly 34 tools again and pays the token cost of both surfaces, and it is also the stage where the misroute rate is worst, because the new tools are competing with the near-neighbours they were designed to replace.

Sessions that connected before a removal keep whatever list they fetched at initialise, so a model can call a tool that stopped existing an hour ago, which is why a removed name returns a result flagged as an error naming its replacement rather than an unknown-method protocol error the model never reads.

Consolidation moves work rather than eliminating it, which the numbers above partly hide. With include as an enumerated array, a share of the remaining failures are argument errors rather than selection errors: the model asks for ["items"] when the user's question needed ["items", "shipments"], and the fix is in the property description and the enum rather than in the tool list. That trade is usually worth taking, because a wrong argument to the right tool is recoverable within the same turn while a wrong tool is a wasted round trip, and because arguments are validated by a schema whereas tool choice is validated by nothing. It also has a floor: pushed further, nine tools become one order_operation with a mode parameter, and the description then has to explain twelve behaviours at once, which is the surface problem again in a different costume.

That floor is worth understanding rather than merely noting, because it says a consolidation has an optimum rather than a direction. Merging two tools removes one pairwise distinction from the surface and adds one argument-level distinction inside the survivor. While the merged tools were near-neighbours the trade is good, because the model resolves the argument from words the user has already supplied, and a wrong argument is correctable inside the same turn. Push past the point where the merged operations have anything in common and the trade inverts: the description now has to explain several unrelated behaviours, the enumeration grows past what a reader holds at once, and the distinctions that used to sit between tools are all still there, competing for attention inside one paragraph. The question is not how few tools you can get to. It is which distinctions belong between tools and which belong between arguments.

The signal to act on, then, is a reciprocal confusion matrix rather than a complaint, and the evidence to design against is a call log and whatever your consumers have already worked around. A generated surface is an API's resource decomposition wearing a model-facing costume, and the decomposition was made for a reader who reads once. Redesigning it is a product decision about what belongs on a model-facing surface at all, and the number that defends it is the one that also tells you what to fix next.

What the whole topic has been circling is that MCP moves a tool contract between processes without improving it. The three fields you declare are the entire briefing, the annotations describe your handler rather than constraining anyone, the enforcement and the duplicate suppression are yours to write, the error channel decides who learns what went wrong, and the wording decides whether any of it is ever reached. A server worth connecting is one where each of those has been decided deliberately and checked against a trace, and the fastest way to find out which of them you have decided is to take a server you already run, capture one provider request, and read what you actually shipped.