Dynamic tool loading

Dynamic tool loading replaces a static tool block with a retrieval step: the incoming request is embedded, an index of tool descriptions is searched, and the model is shown only the nearest k tools rather than the whole catalogue. It is the standard answer to a catalogue that has grown past what a model attends to well, it takes a few days to build, and it reports spectacularly on a selection eval.

The reporting is the problem. A selection eval scores the tool the model chose against the tool a human labelled, and when the correct tool is not in the request at all, the model cannot choose it, so the harness has to do something with that row. What most harnesses do — and what most people write in an afternoon — is treat a request whose labelled tool is absent from the catalogue as out of scope and skip it, because that is the sensible thing to do when you are deliberately evaluating a subset. Under retrieval, that skip silently removes the failures the new component introduces.

Two numbers, two populations#

Ninety-seven of the 540 requests never had a chance, and they are not out of scope in any sense a user would recognise — the capability exists, the agent is deployed with it, and the request failed. Scoring them as failures, which is what they are, puts end-to-end accuracy at 77.1%. Against a flat catalogue's 71%, the recovery is 6.1 points rather than 23.

The arithmetic that produces it is worth writing down explicitly, because it is the shape of every retrieval-plus-generation system and it decomposes cleanly. The cut-off is conventionally called k — here eight, the number of tools the loader is allowed to show — and recall@k is the share of requests whose correct tool appears anywhere in those k, whether or not the model then picked it.

end_to_end = recall_at_k × selection_accuracy_given_retrieved
      0.771 =        0.820 ×                             0.940

Multiplication is the right operation there for a reason that is easy to state and easy to skip. A request has to clear two gates in order for the agent to serve it, and the second gate only ever sees requests that cleared the first. The loader either puts the correct tool in front of the model or it does not, and the model then either picks it or does not — and the ninety-four percent was measured only over requests where the tool was present, which is what makes it a rate given the first gate rather than a rate over everyone. Two rates chained that way combine by multiplying, and averaging them or reporting the larger one answers a question nobody asked.

The two published figures make the same point in a blunter form: they share a numerator and differ only in what sits under it. The count of requests answered with the right tool is the same in both. One divides it by the requests that survived retrieval, the other by the requests that arrived, and every point of the difference between them is the ninety-seven that the first denominator quietly excluded. Whenever two accuracy figures for the same system differ by more than a little, the denominators are the first place to look, and a denominator that was chosen by a component under test is never a neutral choice.

The two factors move in opposite directions as k changes, which is why the sweep matters more than any single configuration.

Every row of that table is its own check, and running it takes a few seconds with a calculator. Multiply the second column by the third and the fourth appears, on all five rows, which is what tells you the sweep was produced from one set of scored requests rather than from five runs whose denominators drifted apart. A row where the product does not reproduce the end-to-end figure means requests are being counted in one column and not another, and that is the exact defect the whole chapter is about, arriving in a table instead of a summary line. The last row is the flat catalogue restated: showing all sixty tools makes recall certain, so the product collapses to the selection figure alone, and the end-to-end column returns the number the loader was brought in to improve.

The configuration that maximises the number a proposal would quote is k=4, at 96.1% selection, and it is the worst end-to-end option on the table apart from doing nothing. The best is k=16, which nobody would have chosen from the selection column. Reporting only selection accuracy over the loaded set does not merely inflate the result; it points the tuning in the wrong direction.

The reason the two factors trade off is worth naming, because it is the same mechanism chapter 4 measured. Raising k admits tools in decreasing order of similarity to the request, and the tools most similar to a request that one tool answers are that tool's near neighbours — so every increment of k buys recall by putting another plausible-looking alternative in front of the model. The loader is not adding random tools as it widens; it is adding precisely the ones the model is worst at telling apart, which is why selection accuracy falls monotonically down the third column while recall climbs.

The fix is to give the harness a third outcome, so that a retrieval miss is recorded as its own failure class rather than being folded into either the successes or the skips.

def score_request(req, loader, agent, k):
    loaded = loader.retrieve(req.messages, k=k)
    names = {t["name"] for t in loaded}

    if req.label not in names:
        return {"request_id": req.id, "outcome": "retrieval_miss",
                "expected": req.label, "chosen": None,
                "rank_of_correct": loader.rank_of(req.label)}

    chosen = agent.select_tool(req.messages, tools=loaded, temperature=0)
    return {"request_id": req.id,
            "outcome": "correct" if chosen == req.label else "selection_error",
            "expected": req.label, "chosen": chosen,
            "pair": None if chosen == req.label else (req.label, chosen)}

def summarise(rows):
    n = len(rows)
    hit = sum(r["outcome"] != "retrieval_miss" for r in rows)
    ok = sum(r["outcome"] == "correct" for r in rows)
    return {"recall_at_k": hit / n,
            "selection_given_retrieved": ok / hit,
            "end_to_end": ok / n}          # the only one that ships

rank_of_correct on a miss is the field that makes the misses actionable. When the correct tool sat at rank 9 or 12, raising k recovers it; when it sat at rank 47, the problem is the index rather than the cut-off, and no k you are willing to pay for will help. On this catalogue the misses split roughly two-thirds within rank 20 and one-third far outside it, and the far-outside group is dominated by requests phrased in customer language against tools described in internal language — "my card got charged twice" against void_charge, whose description talks about authorisation reversal and settlement windows. That is fixable by indexing example utterances alongside the description rather than by tuning retrieval, which is the single cheapest improvement available in this whole chapter.

Latency, multi-turn drift and the index#

Latency lands on every turn, including the turns where the model was going to answer without calling a tool at all. At roughly 150 ms for an embedding call plus an approximate nearest-neighbour (ANN) query over sixty vectors — an index lookup that gives up exactness for speed, returning the nearest entries it can find quickly rather than provably the nearest — on a four-turn conversation that is 600 ms of wall clock the flat catalogue did not spend. A loader that instead uses a small model to rewrite the query before retrieving — a common upgrade once recall becomes the bottleneck — runs 300 to 500 ms per turn, and the recall improvement has to be weighed against a second of added latency in a support conversation.

Multi-turn conversations break the assumption the loader is built on, which is that the request determines the tools. A customer opens with "where's my refund for order 88214", retrieval returns the eight tools nearest that sentence, and the agent calls get_refund — which comes back saying the refund was reversed by the payment processor, at which point the right next call is open_dispute, a tool that was never in the eight because nothing in the customer's opening message pointed at disputes. Retrieving fresh on every turn from the latest message helps only if the latest message contains the signal, and after a tool result the signal is in the result. The workable patterns are to retrieve against the last few messages plus the most recent tool output rather than the user turn alone, and to make the loaded set sticky — union whatever was loaded earlier in the conversation with this turn's retrieval, capped, so the model does not lose a tool it has already used. Sticky loading recovered 14 of the 97 misses here and grew the average tool block from 1,800 to 2,600 tokens by the fourth turn, which is the trade in miniature.

Prompt caching cuts the other way against the token savings, and it is routinely missed. A static tool block is a stable prefix, so providers cache it and charge a fraction of the input rate for the cached portion on subsequent turns — the mechanism topic 2.15 covers, and the one topic 5.14 is about protecting. A tool block that changes on every turn invalidates that prefix every time, which means the 13,000-token static block might cost less per conversation than an 1,800-token dynamic one, depending on the provider's cache pricing and how many turns a conversation runs. Working that out for your own numbers before claiming a cost win takes twenty minutes and occasionally reverses the conclusion entirely.

The last item is operational rather than numeric. Retrieval adds an index that has to be rebuilt when a tool description changes, which introduces a state where a tool exists in the catalogue and is unreachable because nothing has re-embedded it, and a class of failure where a capability silently stops working for the subset of phrasings that used to reach it. The Redis write-up on solving MCP tool overload with semantic retrieval is a good walkthrough of what that plumbing actually involves in production. A newly added tool is the sharpest version of the problem, because it starts with a description written by whoever built it and no evidence about how customers ask for it, so its recall is at its worst in the week it launches — precisely when someone is watching to see whether the feature works. Seeding the index with a dozen hand-written example utterances per tool at launch, then replacing them with real ones from traffic, is the usual remedy and it is worth budgeting for as part of adding a tool rather than discovering later.

Capacity problems and surface problems#

Retrieval is the right instrument for a capacity problem, where the model is choosing under load across a catalogue too large to attend to properly, and errors are spread thinly across many pairs. It is the wrong instrument for a surface problem, where a handful of near-neighbour pairs generate most of the loss, because a loader that retrieves eight tools will cheerfully retrieve both members of a confusable pair — they are semantically adjacent, which is exactly what the retriever is optimising for — and the model faces the same choice it was already getting wrong, now with the additional risk that the loader dropped the answer.

The pair table decides which situation you are in, and the answer here is mixed in a way that is common and worth reading carefully. Of the 157 failures at sixty tools, 61 sit in six pairs and the other 96 are spread across 71 distinct pairs with one or two each. The concentrated 39% is surface work — merges and description rewrites, on the pairs from chapter 6 — and the diffuse 61% is the part retrieval genuinely addresses. Doing both is not redundant, and doing them in the wrong order wastes the eval: fix the six pairs first, because the recall ceiling and the k sweep will need rerunning after any change to descriptions anyway, since descriptions are what the index is built from.

Those three counts reconcile in both directions, which is how you know the split is over the whole loss and not over a convenient part of it. Seventy-one percent of the five hundred and forty labelled requests were routed correctly at sixty tools, so a little over a hundred and fifty were not, and the concentrated and diffuse groups add back to exactly that. The shape of the diffuse group is the part that carries the diagnosis: ninety-six failures spread over seventy-one distinct pairs is between one and two failures each, and there is no rewrite you can perform on a pair that fails once. That is the numerical signature of a catalogue too large to attend to, set against six pairs carrying around ten failures apiece, which is the signature of two tools that read the same.

What would change the reading is the shape of the misses. If the same six pairs still accounted for most of the loss at k=16, retrieval would be the wrong lever regardless of what the end-to-end number said, and the honest response would be to spend the three days on the catalogue rather than on the loader. If instead the residual errors after retrieval were scattered across pairs the retriever had co-loaded, the remaining work is in the retriever's ranking rather than in the model's judgement.

Both mitigations leave one thing untouched, and it is the failure with the largest single incidents in this topic. When the loader misses, the model is left holding a request it cannot serve with the tools in front of it, and its behaviour under that condition is not to stop. It calls the nearest available tool, gets a result that does not answer the question, and calls again — which is a loop, running against a backend that is returning 200 to every one of those calls.