Absorbing transport inside the tool

The merged search_docs from the previous chapter is one tool now, with one argument name, and it is still the internal endpoint with a JSON schema bolted to the front of it. Absorbing a mechanic means moving it from the schema into the implementation: the tool still pages, still retries, still classifies status codes, but a function does it rather than a language model, and the model never learns those things exist. What is left in the schema is the vocabulary of the task.

The distinction matters because the two failures look identical from outside. A run that stops after one page of results and a run that stops because there was only one page produce the same transcript. Below is the definition as it actually shipped, and one call against it.

{"name": "search_docs",
 "description": "GET /v2/collections/{collection_id}/documents/search",
 "input_schema": {
   "collection_id": "string (uuid)",
   "query": "string",
   "cursor": "string|null",
   "page_size": "integer, default 20, max 20",
   "sort": "string, one of relevance|created_at|updated_at"}}

search_docs(collection_id="7c1e9a04-2f55-4b18-9a3d-6e0b1c8f2211",
            query="rotate service account key", page_size=20)

HTTP 200
{"items": [
   {"document_id": "01HQ8ZK3M4N5P6R7S8T9V0W1X2",
    "title": "Key rotation runbook", "score": 0.71,
    "snippet": "...rotate the service account key every 90 days..."},
   ... 19 more ...],
 "next_cursor": "eyJvZmZzZXQiOjIwLCJxaCI6ImE5ZjMifQ==",
 "has_more": true,
 "collection_version": 4412}

Everything in that call that is not about finding a document#

Nine things have to go right per call, and one of them is the query. The rest are the endpoint's own mechanics, surfaced to a component that has to re-derive them probabilistically every time.

The status codes are the sharpest of these because the endpoint passes HTTP semantics straight through and the four codes it can return mean four unrelated things. A 404 comes back when the query matches nothing, which is a completely ordinary outcome of searching. A 409 comes back while the collection is being reindexed, which happens for roughly 90 seconds after every documentation deploy — about eleven times a day — and clears on its own. A 429 is load shedding, the service refusing work it is currently too busy to accept, and it wants exponential backoff, where each successive retry waits longer than the one before it. A 403 means this user cannot read this collection and will never be able to, no matter how many times it is retried.

The 404 is the clearest of the four, because it is a category error rather than a hard case. An HTTP status code is a channel for reporting what happened to the request, and a search that ran correctly and matched nothing is a request that succeeded. The endpoint uses the not-found code for it because, from where the endpoint is standing, the thing being addressed is a set of documents and the set came back empty — a defensible reading of the specification and a disastrous one for a caller that has to decide whether to try again. So the single most ordinary outcome of a search arrives in the channel reserved for things going wrong, and any component reading that channel has to be told separately that this one member of it is not a problem. Nothing in the response distinguishes the two, which is why the telling has to happen in advance and in prose.

Across 300 production runs of that helpdesk agent, the traces show what the model does with them. Of 47 calls that returned 404, 29 ended with the agent telling the user the documentation system had returned an error, when the correct response was "there is no runbook for that." Of 22 calls that returned 409, 8 were retried immediately without backoff and 3 of those hit the run's call ceiling still retrying. Every 403 in the sample was retried at least twice. The endpoint is behaving exactly as designed; the surface is asking a language model to be a well-written HTTP client.

Why the description cannot carry this#

The cheap fix is to keep the shape and explain it. Write a description that says the cursor must be passed back verbatim, that a 404 means no results rather than a failure, that a 409 should be retried after a short wait and a 403 should not, and that page_size caps at 20. Roughly 180 tokens of prose, no code changes, ships this afternoon.

Every sentence in that description is a rule that has to be executed correctly, by a stochastic component, on every single call — and the failures listed above are failures of executing rules of exactly that kind. The team ran it as an arm against the frozen task set from chapter 2. Invalid-argument rate on search_docs went from 18.6% to 11.2%, which is a genuine improvement and still leaves roughly one call in nine malformed. Task success moved 4 points. The 180 tokens are re-sent on every turn of every request, so at 4,000 requests a day averaging 6 turns each, the explanation costs about 4.3 million input tokens a day to keep saying.

That last figure is three multiplications, and the order they come in is the teaching. The description is a fixed number of tokens and it is sent once per request to the model — not once per run, and not once per user. A run is several requests, because every tool result has to come back inside a new one, so the per-run cost is the description multiplied by the turns. The per-day cost is that multiplied again by the runs. Six turns against four thousand requests is twenty-four thousand occasions a day on which the model is handed the same hundred and eighty tokens, and multiplying those out is where a paragraph that felt free becomes a standing line item. Nothing about the paragraph got more expensive. The multiplier did.

The two rates quoted for that arm move at very different speeds, and the gap between them is worth reading rather than averaging. The invalid-argument rate fell by about two fifths while task success moved four points, because a malformed call is usually not fatal: the schema rejects it, the rejection goes back to the model, and the model often gets it right on the second attempt. What a malformed call reliably costs is a turn, and what it costs the run is a share of the call budget spent on recovering instead of progressing. An intervention that improves argument quality therefore shows up strongly in the invalid rate, weakly in task success, and directly in latency and tokens — and a team watching task success alone will conclude that a genuine improvement did almost nothing.

The structural limit is that a description changes the distribution over calls and a schema changes which calls exist. Nothing written in prose can make it impossible to invent a cursor, because the cursor is an argument and inventing an argument value is always within the model's reach. Determinism has to come from the contract itself.

The replacement contract#

The rule for what goes in the model-facing schema, following the split from chapter 3: the operations and constraints the task needs survive, the mechanics of how your services talk to each other do not appear at all. Applied here, that produces one tool with two arguments.

{"name": "search_docs",
 "description": "Search internal documentation. Returns the most relevant "
                "passages, best first, across all collections the caller can read.",
 "input_schema": {
   "type": "object",
   "properties": {
     "query": {"type": "string",
               "description": "The user's question, in their own words"},
     "area": {"type": "string",
              "enum": ["runbooks", "policies", "api-reference", "onboarding"],
              "description": "Optional. Omit to search everything."}},
   "required": ["query"]},
 "returns": {
   "results": "array of {ref, title, passage, area, updated}, best first, max 8",
   "more_matches_exist": "boolean",
   "status": "one of ok | no_matches | unavailable"}}

The cursor is gone, the page size is gone, the sort is gone, the UUID is gone, and the twenty-six-character document id is replaced by a ref — a short, server-issued, human-legible handle like runbooks/key-rotation#3 that the model can quote back to get_document and that a support engineer can paste into a browser. The status field carries the three outcomes the task cares about, and nothing about HTTP survives into it.

Every mechanic that disappeared from the schema reappears inside the implementation, where it is executed by code that does the same thing every time.

MAX_UPSTREAM_PAGES = 6
RESULT_CAP = 8

def search_docs(query: str, area: str | None = None, *, caller) -> dict:
    collections = resolve_collections(area, caller)   # names -> UUIDs, scoped
    if not collections:
        return {"results": [], "more_matches_exist": False, "status": "no_matches"}

    hits: list[Hit] = []
    truncated = False
    for cid in collections:
        cursor, pages = None, 0
        while pages < MAX_UPSTREAM_PAGES:
            try:
                page = backend.search(cid, query, cursor=cursor, page_size=20)
            except Conflict:                       # 409, mid-reindex
                if not retry_with_backoff(max_wait_s=3.0):
                    return {"results": [], "more_matches_exist": False,
                            "status": "unavailable"}
                continue
            except NotFound:                       # 404, nothing matched
                break
            except Forbidden:                      # 403, never retry
                break                              # collection silently omitted
            hits.extend(page.items)
            pages += 1
            if not page.has_more:
                break
            cursor = page.next_cursor
        else:
            truncated = True

    ranked = rerank(hits, query)[:RESULT_CAP]
    return {"results": [to_ref(h) for h in ranked],
            "more_matches_exist": truncated or len(hits) > RESULT_CAP,
            "status": "ok" if ranked else "no_matches"}

The paging loop is worth reading closely because it is the part most implementations get half-right. The tool draws from as many upstream pages as it needs, up to a cap, and then ranks the accumulated set globally before truncating to eight. Fetching one page and returning the top eight of those twenty is not the same thing: an upstream endpoint sorted by its own relevance function will happily put the best answer on page three when the query is a phrase the first-stage index scores badly, and the whole point of the redesign is that the model never learns page three exists. The more_matches_exist flag is what remains of pagination, and it is enough, because the only decision the model makes with it is whether to suggest narrowing the query.

The two constants at the top of that function are where the fetch budget is set, and multiplying them gives the tool's worst case. Six upstream pages of twenty rows apiece is a hundred and twenty records the tool may pull in order to return eight, for each collection it searches. That ratio is the price of ranking globally rather than locally, and it is the figure to take to whoever owns the search backend before shipping. It is a modest number on its own. What earns it the conversation is that it is the one quantity the redesign moved in the wrong direction, and the one the model can no longer influence. Absorbing a mechanic means the tool now decides how hard to work, on the same terms, every time.

Measured on the same frozen task set, the invalid-argument rate on this tool fell to 1.1%, retries of the same call fell from 58 across 200 runs to 4, and the median number of search_docs calls per run went from 2.3 to 1.0. Task success on retrieval-dependent tasks moved 19 points. The tool now issues a median of 1.4 upstream requests per call and a p95 of 4, so its own p95 latency went from 140 ms to 610 ms while the run's wall time dropped, because the 470 ms it added is smaller than the 1.4 seconds a removed model turn was costing.

The latency line in those measurements is a trade with two sides, and both are in one sentence. The tool got slower because it is now doing serially what the model used to do across turns: several upstream requests, a rerank, and a wait for the slowest of them, so its own p95 rose by roughly half a second. A model turn — a full request carrying the whole transcript, a decode, and the round trip back — costs several times that, and the redesign removed more than one of them from the median run. Subtracting a larger number more than once from a smaller number added once is the whole of why the run got faster while the tool got slower, and it is the reason per-tool latency is a misleading thing to gate on by itself.

Which mechanics belong in the contract#

Absorbing is not swallowing. The tool takes over the mechanics of talking to a backend; what it must not take over is the job of saying what happened, because the model's next action depends on which of several things happened and the mechanics are precisely what used to encode that.

The failure mode of this redesign is a tool that catches everything and returns an empty list. A model that receives {"results": [], "status": "no_matches"} when the search cluster is down will tell the user, in a confident sentence, that no such runbook exists. That is worse than the 409 it replaced, because at least the 409 was visible.

The distinction the implementation above preserves is between conditions the task cares about and transport that it does not. "Nothing matched" and "the search backend cannot answer right now" are two different facts about the world, and the model's next action differs for each — refine the query, or tell the user to try again in a minute. They stay as separate values of status. What gets absorbed is the encoding: that one arrives as a 404 and the other as a 409, that the second is worth retrying and the first is not, that the retry wants backoff. The test to apply to each mechanic is whether the model's behaviour should change based on it. If yes, it belongs in the contract, in the vocabulary of the task. If no, it belongs in the implementation.

The 403 case sits on the line and deserves its own decision rather than the default. Silently omitting a collection the caller cannot read is right for a search over four areas, where mentioning the existence of a restricted collection is itself a small leak. It is wrong for a tool whose whole job is one restricted resource, where the model needs to say "you do not have access" instead of "there is nothing there." Both are defensible; picking one by accident is not.

Deep paging, upstream load and debuggability#

Absorbing mechanics costs real capability, and pretending otherwise is how these redesigns lose credibility in review.

Deep paging is genuinely gone. There is no way to ask this tool for results 21 through 40, and a user who needs the fortieth most relevant runbook has to narrow the query instead. On the task set, 6 of 200 tasks touched anything past the top eight, and all six were better served by adding area than by paging, so the capability is being traded for a failure mode that was costing 19 points. That is a defensible trade at this ratio and would not be at a different one; a legal discovery agent whose job is exhaustive review needs a different contract, probably one that returns counts and lets the model request bounded slices by an explicit filter rather than by position.

Upstream load goes up, because the tool sometimes fetches 120 records to return 8. On this traffic that is about 1.9× the read volume against the search backend, which was cheap and stayed cheap; on a backend where a search page costs 200 ms of database time, the cap on MAX_UPSTREAM_PAGES is doing load-bearing work.

Debuggability moves rather than disappears. The transcript no longer shows which upstream page a result came from, so when a support engineer asks why the agent missed a document, the answer is not in the conversation. That has to be recovered with a server-side trace id logged against each tool invocation, which is a small amount of work and easy to skip until the first incident where it is needed.

And the tool now makes a decision the model used to make. Choosing global relevance over recency is a product judgement, hard-coded in rerank, and there will be questions — "what changed in the deploy policy recently" — where recency was the right sort and the tool cannot be told. The honest answer is that this is a schema question rather than a code question: if enough tasks need it, recency becomes an explicit argument with an enum of two values, which is a distinction the arguments can express and therefore a legitimate split by the test in chapter 4.

Absorbing is what turns a per-call rule into a per-deploy decision. The cursor, the page size, the sort, the UUID and the four status codes were each a thing the model had to get right several thousand times a day; after the redesign a human got each of them right once, in code, under review. The two arguments that survive are the two the task actually has, and the three values of status are the three outcomes the task can distinguish.

Every tool in the surface can be rewritten this way, and doing so leaves untouched a cost that no individual tool's design can reach: how many tools there are. A surface of a hundred perfectly-absorbed tools has a problem that a surface of twelve does not, and it shows up in two measurements that no per-tool metric from chapter 2 will ever surface.