Namespacing a tool catalogue

Namespacing a catalogue means giving every tool a dotted domain prefix — billing.refund_order instead of refund_order — so that the names carry the domain they belong to. It is the cheapest mitigation on the list, it ships in a day, and it is the first thing most teams reach for once a catalogue crosses thirty tools and someone notices the routing getting worse.

Whether it helps is decided by a table you already have. Chapter 5's harness records the ordered pair of expected and chosen tool for every failure, so answering the question takes a GROUP BY rather than another run, and the answer is specific to your catalogue rather than general. What follows is what that table looks like on a forty-tool support catalogue, what a prefix can and cannot do to the failures in it, and what a rename costs in places nobody checks on a Friday.

The pair table, with a column added#

Adding a column for whether the two tools now share a prefix turns the failure list into a direct verdict on the rename.

Fifty-three of the sixty-seven failures are between two tools that a prefix places in the same bucket. The rename could only ever have helped the eleven that cross a boundary, and it helped roughly none of those either, because a model that cannot tell a return from a refund is not going to be rescued by learning that one of them is filed under support and the other under billing — the whole difficulty is that the customer's sentence does not say which department owns the answer.

The verdict does not depend on how the scattered row is attributed, which is what makes it safe to act on. The Failures column adds to sixty-seven with nothing outside it, so every failure the run produced is in that table somewhere. The named rows are unambiguous — forty-four of them are between two tools filed under the same prefix and eleven cross a boundary — and only the scattered singletons are mixed, since a row of one-off confusions has no single namespace answer. Attribute every one of those singletons to the boundary-crossing side, which is the most generous assumption a proposal for namespacing could ask for, and the same-bucket failures are still a clear majority of the total. A finding that survives its own worst case is one you can merge a decision on.

A prefix behaves the way a package name behaves in code. com.acme.billing.RefundService and com.acme.support.ReturnService are unambiguously different types to a compiler, and a Java package has never once stopped a developer from calling the wrong method on the right class. Inside a package, everything sits at the same level of ambiguity it always did, and the qualifier that resolved the cross-package case contributes nothing. The catalogue works the same way, except that the resolution happens in a language model's forward pass rather than in a symbol table, so the prefix is not even a hard constraint — it is one more correlated token in a name the model is weighing against every other name.

The reason the scheme is so tempting is that it looks like it should work at the level of token generation. A tool name is emitted left to right, so billing. comes out before refund_order, and it is easy to picture the model making a coarse decision first and a fine one second, with the coarse decision pruning the field. Providers that constrain decoding to the set of legal tool names make that picture even more plausible, since the prefix genuinely does restrict what can follow it. What the picture leaves out is that the model is not choosing the prefix on separate evidence — it arrives at billing.refund_order as one continuation, and the prefix is generated because the tool was already the likely answer rather than the prefix narrowing the search that produces it. The ordering is real and the pruning is not.

Collisions, policy units and a cheaper measurement#

The rename is still worth merging, for reasons that have nothing to do with the eval. Three of them are worth naming, because teams that adopt namespacing and then measure no accuracy change often conclude the work was wasted and revert it.

The first is genuine name collision, which is a different failure from confusion. Attach three MCP servers to one agent and you can easily end up with two tools called search and two called get_user, at which point the model's output does not identify a tool at all and your dispatcher has to guess which server was meant. Prefixing resolves reference; it does not resolve intent. Those two problems get discussed under the same word and have almost nothing in common.

The second is that namespaces give you a unit of policy. Permissions, rate limits, audit requirements and on-call ownership are all naturally per-domain, and billing.* is a far better thing to attach a policy to than a hand-maintained list of nineteen tool names that goes stale the week someone adds the twentieth. The same applies to the metrics from chapter 3: per-namespace failure rates are a useful intermediate view between the aggregate and the forty per-tool figures.

The third is that it makes the next measurement cheaper, because the namespace column that produced the table above only exists once names carry a domain. That is a small benefit and it is the one that pays for the Friday.

Against those, a rename is a breaking change to every identifier the system has ever emitted, and the cost lands in places nobody checks on a Friday. Metric series are tagged by tool name, so refund_order and billing.refund_order are two different series and the dashboards that chapter 3's argument check populates go flat on one and start empty on the other, with no continuity across the deploy unless someone writes an aliasing rule. Labelled eval sets store the expected tool name per request, so the 540 labels have to be rewritten in the same commit or the next ladder run scores 0%. Any few-shot examples in the prompt — the two or three worked exchanges pasted in above the conversation to demonstrate the behaviour wanted — contain the old names, cached conversations replayed from history contain the old names, and a customer-facing audit log now contains both. None of this is hard; all of it is the sort of thing that gets discovered on Monday.

TOOLS = [
    {"name": "billing.refund_order",
     "description": "Return money to the customer's original payment "
                    "method for a delivered order. Use when the customer "
                    "has the item and the charge is not contested.",
     "input_schema": {...}},
    {"name": "billing.open_dispute",
     "description": "Open a formal dispute against a charge. Use only "
                    "when the customer says they did not authorise it or "
                    "has already contacted their bank.",
     "input_schema": {...}},
]

# The prefix is a naming convention on the wire; nothing in the
# protocol constrains selection to a namespace.
assert all(t["name"].split(".", 1)[0] in {"crm", "billing", "support"}
           for t in TOOLS)

The two descriptions above are the part of that Friday's work that could move the eval, and it is worth noticing that they have nothing to do with the prefix. Rewriting refund_order and open_dispute in terms of when to reach for each — the customer has the item and is not contesting the charge, against the customer says the charge was unauthorised — is the granularity and surface work from topic 5.10, applied to the one pair that the table says is costing nineteen requests. Namespacing gets bundled with that rewrite in most renames, which is how it acquires a reputation for helping.

Making the namespace a real constraint#

There is a stronger version of the idea, where the namespace stops being a naming convention and becomes a decision the model actually makes. The agent is shown three namespaces with a paragraph describing each, picks one, and is then shown only that namespace's tools in a second call.

This does convert cross-namespace confusion into something structural, and on the forty-tool catalogue it did what it promises: the eleven boundary-crossing failures fell to two. The dotted edge is what it costs. Stage one is itself a classification the model can get wrong, and when it does, the correct tool is not in the second call's catalogue at all, so the request cannot be recovered — 3.1% of the 540 requests picked the wrong namespace, and every one of those became a failure that the flat catalogue would have got right. The net movement was −0.4 points, inside the seed noise, for an extra model call on every turn.

The token column is the honest argument for two-stage routing and it has nothing to do with accuracy. Showing nineteen billing tools instead of forty cuts roughly 1,900 tokens from the second request, which on high volume is real money and real context budget, and if that is why you are doing it then the right thing to measure is cost per resolved conversation, with accuracy as a guardrail rather than as the goal.

The prefixed column's token cost comes from the same place and is small for a reason worth stating. A domain word and a dot in front of a name is a token or two once serialised, paid once per tool definition, and forty tools carrying one apiece is where the eighty comes from. It is a rounding error against a tool block, which is the honest summary of flat prefixing as a whole: it costs almost nothing and it buys almost nothing on this axis. The two-stage row is the opposite trade and its sign is worth reading carefully, because the accuracy column moves by less than the seed spread while two effects of opposite sign sit underneath it. Boundary-crossing failures fall, since the wrong-namespace tool is no longer on offer. A new class appears in its place, one the flat catalogue could not produce at all, and on this catalogue that new class is the larger of the two. A structural fix that removes one failure mode by introducing another is only worth having when the one it introduces is the smaller, and that comparison is a measurement rather than an intuition.

Loss inside a single namespace#

Everything above operates on the boundary between groups of tools, and the majority of the failures in the table are inside a group, where no boundary exists to operate on. refund_order against open_dispute and issue_credit_note is thirty-one requests of loss that a naming scheme cannot touch, because all three return money in response to an unhappy customer and the differences between them are facts about the payment network that the conversation does not contain. The instruments that move that number are the ones aimed at the pair: rewriting both descriptions around the condition that separates them, merging them into one tool with a settlement parameter, or adding a required argument that forces the distinction into the open where a schema can constrain it.

Those thirty-one requests are the two rows naming refund_order as the correct answer, added together, and set against the sixty-seven failures in the whole table they are close to half of everything the catalogue got wrong. One tool, against two of its neighbours, accounts for nearly as much loss as the other thirty-seven tools combined. That is what "concentrated" means in practice, and it is also the argument for doing the grouping before choosing a mitigation rather than after: a fortnight spent on one triangle of three tools is aimed at half the problem, and no naming scheme, routing layer or catalogue split addresses any of it.

There is also a case this diagnosis does not cover. The pair table here is steeply concentrated, with 79% of failures in seven pairs, which is what points at surface work. A catalogue of sixty or a hundred tools assembled from four MCP servers frequently produces the opposite shape — failures spread thinly across dozens of pairs, none of them individually worth a rewrite, with the model behaving as though it is choosing under load rather than confusing two specific neighbours. Renaming that catalogue will not help it either, and neither will rewriting descriptions one pair at a time when there are forty pairs contributing one failure each.

So the reason to read the pair table before choosing a mitigation is that the table is what tells you which of those two situations you are in, and the two have almost no remedies in common. A concentrated table says spend the fortnight on four pairs. A diffuse one says the catalogue the model sees on each turn is too large, and the mitigation is to shrink it by retrieving the handful of tools relevant to the request and showing only those — which reports very well on a selection eval, partly as an artefact of what that eval stops counting.