Catalogue size and selection accuracy

Every chapter so far has improved one tool at a time — merging a confusable pair, absorbing a cursor, choosing between one broad operation and five narrow ones. This chapter is about the cost that survives all of that, because it belongs to the surface as a whole rather than to anything in it. Two surfaces can contain only well-designed tools and behave very differently, and the difference is how many tools there are and how close together they sit.

Both costs are measurable in an afternoon, and the measurement is a distractor experiment: take a surface whose tools you have already cleaned up, add tools the tasks never need, change nothing else, and re-run. The helpdesk agent from chapters 5 and 6 had twelve tools after its redesign. Distractors drawn from the rest of the company's OpenAPI document — list_billing_plans, get_sso_config, patch_workspace_settings, and a hundred and twenty-five others, all of them irrelevant to every task in the frozen set — went in, and the same 200 tasks ran against the same prompt and the same model.

What the distractors cost#

The last column is TTFT, or time to first token: the delay between sending a request and the first word of the model's reply arriving. Nothing in the twelve tools changed. Their schemas, their descriptions, their implementations and the tasks are identical across all four arms, and the arms differ by exactly one variable, which is the whole point of the harness from chapter 2. Seventeen and a half points of task success went to tools that were never called. That number is not a property of any tool in the surface, so no per-tool metric will ever show it, and it is the reason "we have too many tools" needs to stop being an aesthetic objection and start being a measurement.

Why accuracy falls when the extra tools are irrelevant#

Selecting a tool is a discrimination problem over descriptions, and the difficulty of a discrimination problem depends on how close the nearest wrong answer is. Chapter 5 took apart one confusable pair, search_docs against find_document, where the intents overlapped completely and calls scattered near a coin flip. That pair is not a curiosity produced by two teams naming things badly. It is what happens whenever a new tool lands whose description occupies territory a tool already there was covering, and the chance of that rises with the number of tools already present.

The arithmetic is unkind. A catalogue of n tools has n(n−1)/2 pairs — 66 at twelve tools and 9,730 at 140. What matters is not the raw pair count but how many of those pairs are close enough to be confusable, and that number grows roughly with the pair count rather than with n, because each new tool is tested against everything already there. Measuring it directly is a short piece of code.

That formula is worth deriving once rather than quoting, because the derivation is the argument. Each tool can be paired with every other one, which is n choices for the first member and one fewer for the second; every pair has now been counted twice, once from each end, so halving the product gives the number of distinct pairs. Twelve tools give twelve times eleven, halved, which is the sixty-six. A hundred and forty give a hundred and forty times a hundred and thirty-nine, halved, which is the figure just short of ten thousand. The practical form of the same fact is easier to carry around: adding one tool to a surface of n adds n new relationships rather than one. A thirteenth tool creates twelve new pairs. A hundred and forty-first would create a hundred and forty.

from itertools import combinations
import numpy as np

def near_neighbours(tools, embed, threshold=0.82):
    """Pairs of tools whose descriptions a router may not distinguish."""
    texts = [f"{t['name']}: {t['description']}" for t in tools]
    vecs = np.array(embed(texts))
    vecs /= np.linalg.norm(vecs, axis=1, keepdims=True)
    out = []
    for i, j in combinations(range(len(tools)), 2):
        sim = float(vecs[i] @ vecs[j])
        if sim >= threshold:
            out.append((tools[i]["name"], tools[j]["name"], round(sim, 3)))
    return sorted(out, key=lambda p: -p[2])

# 12-tool surface  ->  2 pairs above 0.82
# 140-tool surface -> 190 pairs above 0.82, 31 of them above 0.90

Cosine similarity — the closeness of two descriptions after an embedding model has turned each into a vector, running from 0 for unrelated text to 1 for identical text — is a screen over descriptions rather than a verdict, because two tools can read alike and still be routed cleanly by their arguments: chapter 5's get_document_by_id sits close to search_docs in embedding space and never gets confused with it, since the model chooses it by holding an id. The screen's job is to produce a short list of pairs to check against the routing cross-tabulation, and on a 140-tool surface that short list is 190 pairs long, which is more review than anyone will do.

The "first-call correct" column in the table is the measurement that makes this legible, and it needs one addition to the run records from chapter 2: each task in the frozen set carries the name of the tool a competent engineer would call first, and the harness compares it against the first tool the model actually called. It is deliberately a weaker signal than task success, since a run that opens with the wrong retrieval tool often recovers, and that weakness is the point — selection accuracy moves before task success does, so it gives you a few points of warning while the surface is still working. Tasks where two tools are both defensible as an opener get excluded from the denominator rather than labelled arbitrarily, which on this set removed 23 of 200.

The distractors matter even when they are never called because they are competing for the same decision. In the 140-tool arm, the tasks that failed were not failing on exotic tools; 61% of the wrong first calls went to one of nine distractors whose descriptions mentioned "search", "look up" or "retrieve", and get_sso_config alone absorbed 14 calls that belonged to search_docs because both descriptions contained the word "configuration". A tool you never intend the model to call still has to be discriminated against on every turn.

The token cost is standing, and it is the smaller half#

Tool definitions live in the prompt. They are serialised into every request the agent makes, which means every turn of every run re-sends the entire catalogue, including the 128 tools this particular user will never touch. The accounting is simple enough to do on a napkin and worth doing on a real one.

TOOLS_TOKENS = {12: 3_410, 140: 27_800}   # measured with the tokenizer
TURNS_PER_REQUEST = 6                      # median, from the run records
REQUESTS_PER_DAY = 4_000
PRICE_IN = 3.00 / 1_000_000                # USD per input token
CACHE_READ = 0.30 / 1_000_000              # cached input, 10% of base

for n, toks in TOOLS_TOKENS.items():
    per_day = toks * TURNS_PER_REQUEST * REQUESTS_PER_DAY
    print(n, f"{per_day/1e6:.1f}M tok/day",
          f"${per_day*PRICE_IN:,.0f}/day uncached",
          f"${per_day*CACHE_READ:,.0f}/day cached")

# 12   81.8M tok/day   $245/day uncached   $25/day cached
# 140  667.2M tok/day  $2,002/day uncached $200/day cached

Follow one line of that through by hand. Twelve tools serialise to a bit under three and a half thousand tokens; every request carries all of them; a run makes six requests; four thousand runs a day is twenty-four thousand requests. Multiplying those out gives a little over eighty million tokens a day of tool definitions alone. Pricing them at three dollars a million is the two hundred and forty-five, and pricing them at the cached rate — a tenth of that — is the twenty-five. The same three multiplications on the hundred-and-forty-tool figure give a result about eight times larger, because the token count is about eight times larger and nothing else in the chain moved. That proportionality is the thing to notice: the token half of the catalogue cost is exactly linear in the size of the block, which is what makes it easy to argue about and easy to discount away.

Both annual figures come from the daily ones by a subtraction and then one multiplication. Two thousand dollars a day against two hundred and forty-five is a difference a little over seventeen hundred, and a year of that is the six hundred and forty thousand. Do the same on the cached column — two hundred against twenty-five — and the difference is a hundred and seventy-five a day, which is the sixty-four thousand. The ratio between the two annual figures is ten, for the same reason the ratio between the daily ones is: a fixed discount applied to a linear quantity survives every later multiplication unchanged.

The gap is about $640,000 a year uncached and $64,000 cached, for tool definitions alone, before a single word of user content. Prompt caching, where the provider retains the computed state of a prompt prefix it has already processed and charges a reduced rate whenever that same prefix arrives again, changes the magnitude and not the shape: because tool definitions sit at the front of the prompt and rarely change, they cache almost perfectly, and a team that has enabled caching is paying roughly a tenth of the sticker price on the catalogue. That is the honest version of the cost argument, and it is why the token line loses when it is argued alone — a finance-shaped objection to 140 tools gets answered with "we cache it" and the conversation ends.

What does not cache is the accuracy. The seventeen points in the table above are not a billing artefact and do not improve when the same prompt prefix is reused, because the model still has to discriminate among 140 descriptions on every turn regardless of where the key-value cache came from. Time-to-first-token improves with caching too, so the 890 ms in the table becomes roughly 500 ms, and the run still fails a quarter of the time. Argue the catalogue on selection accuracy and use the tokens as the supporting figure rather than the headline.

That claim sounds like an assertion, so it is worth having in mechanical terms. A prompt cache stores the intermediate state the model computed while reading a prefix, so that the same prefix does not have to be processed from scratch next time. What it stores is exactly what the model would have computed anyway, which means a cache hit and a cache miss leave the model in the same state and produce the same distribution over next tokens. Caching changes what the computation costs and how long it takes, and by construction it cannot change what the computation returns. All hundred and forty descriptions are still present, still being weighed against each other, and the selection is still a comparison over the same field of candidates. That is why the two costs in this chapter respond to entirely different interventions, and why arguing the catalogue on the one that has a fix is a mistake.

There is one more per-turn effect that shows up on long runs. The catalogue occupies context that the conversation then has to share, and a 27,800-token tool block against a 200,000-token window is not a problem until turn 20, when the accumulated tool results are also in there. Teams running long agentic workflows hit context pressure with a large catalogue several turns earlier than they would otherwise, and the symptom is a truncated or summarised history rather than an error, which is difficult to attribute after the fact.

The sixth criterion for splitting a tool#

Chapter 4 set out five criteria for choosing between one broad tool and several narrow ones — task boundaries, argument ambiguity, permission scope, validation, and calls per workflow. Catalogue cost is the sixth, and it is the only one that is a property of the whole surface rather than of the tools being decided about, which is why it gets left out.

Concretely, adding a thirteenth tool to a twelve-tool surface costs about 180 to 240 tokens per turn and some amount of selection accuracy that depends entirely on what it sits next to. The question to ask before adding it is what the surface gets back.

The last two rows are where surfaces grow without anyone deciding to grow them. A distinction the arguments can express is permission to split, and chapter 4 was explicit that it is not a reason to; with the catalogue cost written down, the default for that row flips to leaving the tool alone. The bottom row is the mechanism behind essentially every oversized surface in production, because internal service boundaries are the shape the API already has and mirroring them requires no decision at all.

Count is a proxy; density is the thing#

The table at the top of this chapter invites the wrong conclusion, which is that fewer tools is always better. What the distractor experiment actually manipulated was near-neighbour density, since 128 tools drawn from one company's API document contain a great many overlapping retrieval and configuration verbs. Two surfaces of the same size can behave very differently.

Divide the pairs column by the tools column and the table reorganises itself. The redesigned helpdesk surface carries roughly one confusable pair for every six tools. The same surface with a dozen unrelated tools bolted on is barely different, because calendar and payroll verbs create pairs that no threshold counts. The same surface with a dozen retrieval verbs from one API is running at nearly one confusable pair per tool, and its first-call-correct number has fallen by more than eleven points while its tool count moved identically to the harmless case. The curated finance surface, at five times the size of the helpdesk one, carries fewer confusable pairs in absolute terms than the twenty-four-tool dense surface does.

The 60-tool curated surface outperforms a 24-tool surface built by adding near-neighbours, which means a review that counts tools and stops is measuring the wrong quantity. A large catalogue is a risk factor for high density rather than the same thing as it, and the way to hold a large surface together is to make the tools mutually exclusive by construction: one verb per intent, arguments that route, and a naming scheme where the object being acted on is unambiguous in the tool name.

Anthropic's write-up on advanced tool use is worth reading alongside this, because the platform's own answer to large catalogues — loading tool definitions on demand rather than sending all of them every turn — attacks the token half of the problem directly and leaves the discrimination half exactly where it is.

The two costs are asymmetric in a way that decides how to argue about them. Tokens are large, visible on an invoice, and mostly cacheable away. Accuracy is smaller on the invoice, invisible in every per-tool table, and does not respond to caching, deferred loading, or a bigger context window, because the discrimination problem is unchanged by where the descriptions came from. Density is what predicts it, and density is countable from your own definitions in minutes.

The measurements in this chapter are cheap: a distractor-injection arm on a frozen task set costs one afternoon and a few hundred dollars of inference, and the embedding screen costs minutes. Having them turns the next conversation from a disagreement about taste into a disagreement about numbers, and the next conversation is the one where somebody proposes generating a tool for each of 140 endpoints because the specification is right there and a generator would keep it in sync forever.