Declaring a capped result

Every tool that returns rows has a cap, whether or not anyone chose it: a LIMIT in the query, a page size in an API client, a guard in the harness that refuses to serialise more than a fixed number of tokens. The previous chapter turned that cap into arithmetic — a per-row token cost divided into a token budget — and the moment a cap exists, some queries match more rows than it allows. What the model then receives is not the answer to the question it asked but a sample of that answer, chosen by an ordering it was not told about.

The failure that follows is not a model failure. A page of twenty rows and a page of twenty rows that happens to be the first twenty of three hundred and forty are the same payload, and any consumer reading the second will describe it as the first. This chapter is about what a capped result has to carry so that the two are distinguishable, and about the two fixes that look like solutions and are not: an instruction in the prompt, and a larger cap.

The payload made a claim the tool could not support#

Here is a handler that had looked like this for two years without anyone minding, behind a search_invoices tool:

def search_invoices(account_id: str, query: str) -> dict:
    rows = db.query(
        "SELECT id, number, po_ref, status, total_cents, issued_on "
        "FROM invoices WHERE account_id = %s "
        "ORDER BY issued_on DESC LIMIT 25",
        account_id,
    )
    matches = [r for r in rows if query.lower() in r.po_ref.lower()]
    return {"invoices": [serialize(r) for r in matches]}

The cap is applied by the database and the filter is applied in Python afterwards, so the tool searches the 25 most recent invoices rather than the account. On an account issuing about ninety invoices a month, a query for a purchase order raised in January runs against twenty-five rows covering eleven days in March, and returns {"invoices": []}.

Reading the two statements in the order the machine reads them is what makes that inevitable rather than unlucky. ORDER BY issued_on DESC LIMIT 25 runs first and produces a candidate set: the 25 most recent invoices on the account, and nothing else exists as far as the rest of the function is concerned. The Python comprehension then filters that set. A filter can only remove members of the set it was handed, so no invoice outside those 25 rows can appear in the output whatever the query string is. Every property of the result follows from that ordering — the tool is exhaustive over recent invoices, blind to everything older, and the caller's search term selects inside a window it had no part in choosing.

The window's size in time, rather than in rows, is the number that decides the outcome. Twenty-five rows on an account that issues invoices steadily reaches back as far as twenty-five invoices' worth of calendar, and whether that is useful depends entirely on how far back the question goes. A purchase order raised in January is outside a window that ends eleven days into March, so the answer was settled before the search ran. Moving the filter into the WHERE clause changes which set the LIMIT is applied to — twenty-five matching invoices rather than twenty-five recent ones — and that single relocation is what makes the query answer the question it was asked.

The first thing to settle is whose bug the empty array is, because the answer determines where the fix goes. {"invoices": []} is a statement, and the statement is that the search found nothing. A human reading that response in a debugger would conclude the same thing a model does, which is the test worth applying: if the payload is indistinguishable from the payload you would return when the account genuinely has no matching invoices, then the two situations are indistinguishable to any consumer, and no amount of prompting makes a model recover a distinction that is not encoded anywhere in what it received. The model behaves correctly on false evidence. The tool produced the false evidence.

That framing matters because the instinct in review is to add a line to the system prompt telling the agent that search results may be incomplete and to double-check before reporting an absence. Instructions of that kind change behaviour on every call, including the tens of thousands where the result was complete, and what they buy is hedging rather than accuracy — the agent starts saying "I could not find any invoices referencing PO-2231, though there may be others" on results that were exhaustive, which trains users to distrust the answers that were right. An instruction cannot tell the model which of the two cases it is in. Only the payload can.

That impossibility is structural rather than a matter of how well the instruction is worded, and being able to say why is what stops the proposal coming back at the next review. An instruction in a system prompt is a constant. The same sentence is in front of the model on the call that returned an exhaustive result and on the call that returned a sample, because the prompt does not know which call it is about to serve. A signal that separates two cases has to take different values in them. So no phrasing of that instruction can carry the one bit distinguishing an account with no matching invoices from an account whose matching invoices were never looked at — the instruction is identical in both, and identical text cannot encode a difference. The only part of the request that differs between those two calls is the tool result, which is why the bit has to live there and nowhere else.

The same shape produces a louder failure when the question is a summary rather than a lookup. A list_incidents tool caps at 20 rows; an agent asked to summarize an on-call week reported that there were 20 incidents, all in the payments service. There had been 340, across nine services, and the 20 the tool returned were the most recent, which happened to be a payments deploy gone wrong on Friday afternoon. The agent did not merely miss something. It stated a total and a distribution, both derived from a sample it had no idea was a sample, and both wrong in ways that read as authoritative.

A boolean says something was cut and nothing else#

The standard patch is "truncated": true, and it is genuinely better than nothing, because it converts a silent failure into a visible one. It is also close to the least useful signal you could design, since it answers one of the four questions the model needs answered and leaves the others open.

Read down the columns and the failure modes fall out. Without the scale, "20 of an unknown number" and "20 of 22" get the same treatment, so the model either hedges uniformly or guesses. Without the ordering, the model knows it holds twenty incidents but not which twenty, so it cannot say the true and useful thing — that these are the most recent twenty — and it certainly cannot tell that a question about the oldest incident is unanswerable from this data. Without a route to the remainder — a cursor, meaning an opaque token the model hands back to receive the next slice, or a filter that narrows the match — a model that correctly identifies its own ignorance has nowhere to go with that, and the best available behaviour becomes an apology.

Making incompleteness part of the payload rather than an implicit property of it looks like this:

{
  "matched": 340,
  "returned": 20,
  "order_by": "opened_at DESC",
  "next_cursor": "eyJvcGVuZWRfYXQiOiIyMDI2LTAzLTA5VDE4OjQxOjAyWiIsImlkIjo4ODIxfQ==",
  "narrow_by": ["service", "severity", "opened_after"],
  "incidents": [
    {
      "id": 8842,
      "service": "payments",
      "severity": "sev2",
      "title": "Checkout 5xx after deploy 4471",
      "opened_at": "2026-03-13T16:22:00Z",
      "resolved_at": "2026-03-13T17:05:00Z"
    }
  ]
}

With matched: 340 and returned: 20 in front of it, the on-call summary changes from a false claim to a true one, because the model can report that it is looking at the twenty most recently opened of 340 incidents and that a distribution across services needs either more pages or a different call. The narrow_by list matters more than it looks: for a 340-row match the sensible next move is usually not to page seventeen times but to filter, and telling the model which dimensions the tool can filter on turns an expensive traversal into one more call. A tool description teaches the model how to make the call; these fields are the same idea applied to the moment when it has a result in hand and has to choose what to do next.

Raising the cap aims the failure at the accounts that matter most#

The tempting fix is to raise the cap — from 25 to 500 — and be done with it. That does reduce the frequency of the failure, since most accounts have fewer than 500 invoices and would now be searched exhaustively. It also does nothing about an underlying filter bug, and it changes the population of victims in a direction nobody would choose deliberately. Accounts with more than 500 invoices are the accounts that have been customers for years, that spend the most, and whose questions are the ones you least want answered with a confident falsehood. Making a failure rarer while concentrating it on your largest accounts is a worse risk profile than the original at the same error rate, because the cost per occurrence has gone up.

The distribution behind that claim is worth computing before the argument, because it takes one query against a month of tool logs and it settles the discussion without anyone appealing to intuition. For search_invoices, over 41,000 calls in March:

Raising the cap removes 97% of the incidents and leaves the survivors much worse off: a capped call at 25 is missing a median of 71 invoices, while a capped call at 500 is missing 1,400, and the ninety-two accounts still exposed are enterprise accounts carrying nearly a third of revenue. The same table also prices the alternative honestly — 12% of calls returning a silent sample is not a rare edge case that a busy team can defer.

Two different divisions are at work in that paragraph and it is worth keeping them apart, because arguments about caps are usually won by quoting the one the audience was not asking for. The percentages are each call count over the 41,000 calls in the month: 4,900 over 41,000 is 12.0 percent, 160 over 41,000 is 0.4 percent, and both answer the question "how often is a caller affected". The 97% is a different operation on the same pair. Going from 4,900 incidents down to 160 removes four thousand seven hundred and forty of them, and that subtraction expressed as a share of the 4,900 you started with is the 97%, which answers "how much of the problem does this change remove". Neither figure is wrong and neither is the whole picture.

The reason they disagree is that severity moves in the opposite direction to frequency, and the honest comparison multiplies the two rather than reading either column alone. A call capped at 25 leaves a median of 71 invoices unsearched; a call capped at 500 leaves 1,400, close to twenty times as many. Multiplying calls by invoices never searched, the total quantity of invisible evidence falls by roughly a third rather than by 97%, and the third that remains has been moved onto ninety-two enterprise accounts holding 31% of revenue. Risk is a product and not a rate, which is exactly how "rarer" and "better" come apart here.

There is a token argument as well. At the 298 tokens per row measured in chapter 2, a 500-row page is roughly 149,000 tokens, which does not leave a working session on a 200,000-token window. Subtracting one from the other leaves a little over fifty thousand tokens for everything else the request has to carry — the system instructions, every tool definition, the conversation up to this point, and a reserve for the model's own output — and that has to hold for the rest of the session, since the page is re-sent on every subsequent turn. The cap is not an arbitrary limit to be relaxed under pressure; it is the page size that follows from the per-row cost and the share of the budget the tool is entitled to.

A contract computed too early describes a payload that no longer exists#

The contract is a claim about the rows the model received, and there are usually several places between the query and the model where rows can leave. The incidents tool had three: the LIMIT in the query, a permission filter that dropped incidents belonging to services the calling user cannot see, and a guard in the agent harness that trimmed any tool result over 20,000 tokens by cutting elements from the end of the largest array. Building matched and returned from the query meant the payload asserted twenty rows while the model was reading fourteen, and the assertion was more damaging than silence would have been, because the model now had an explicit and false statement to reason from.

Why an explicit falsehood is worse than an absence is the step worth taking slowly, since the instinct is that more information cannot hurt. A model holding rows and no contract has one source of belief about what it holds, which is the rows, and any claim it makes is grounded in something present. A model holding fourteen rows and a field saying returned: 20 has two sources that disagree, and the field is the more authoritative-looking of the two, because it is a stated count while the rows are something you would have to tally. So the sentence it produces — a distribution across twenty incidents, a statement that twenty of 340 were examined — is built on a number that no rows support. Silence leaves the model uncertain about the right thing. A stale contract makes it confident about the wrong one.

Two rules keep the claim true. The contract is computed at the last stage that can remove a row, which in this stack is the harness rather than the tool, so the trimming guard has to rewrite returned and re-emit the cursor for the position it actually stopped at. Anything that cannot rewrite the contract must refuse rather than truncate — an error saying the result exceeded the budget is a fact the model can act on by narrowing the query, and a quietly shortened array is not. The permission filter is the subtler of the three, because dropping rows there is correct behaviour and the model still needs to know the shape of what it is holding; carrying matched as the count after the filter, with a separate flag when anything was withheld, keeps the model from reporting a service-wide total it was never shown.

Honesty about a sample is not the same as a usable answer#

The contract makes the model's reasoning sound; it does not make every question answerable. "How many unpaid invoices does this account have?" cannot be answered from any page of rows, however well labelled, and an agent that pages through fourteen screens of invoices to count them is spending 40,000 tokens and thirty seconds on something a COUNT(*) answers in four milliseconds. When the shape of the question is aggregate, the right response is an aggregate, and the chapter on server-side rollups takes that up properly.

The fields also cost something and can be got wrong in the other direction. A response that carries matched, returned, order_by, a cursor and a filter list is about 60 tokens of overhead, which is nothing against a 20-row page and meaningful against a 2-row one; returning a full incompleteness contract on a result that is complete is noise, and worse, it invites the hedging behaviour that the prompt-instruction fix produced. The contract belongs on results that are actually capped, with matched and returned present on both so the model can see they are equal.

The two ends of that range are worth working out, because "nothing" and "meaningful" are doing a lot of unexamined work. At the 298 tokens a row measured in the previous chapter, a 20-row page is close to six thousand tokens, so 60 tokens of contract is about one percent of it and disappears into the rounding. A 2-row page is under six hundred tokens, and the same 60 is a tenth of the payload. The contract has a fixed cost and the page has a variable one, so the fraction the contract occupies rises as the page shrinks — which is the ordinary behaviour of any fixed overhead and the reason a rule of thumb stated as a token count says nothing until it is divided by the thing it sits on.

A capped result that declares itself costs about sixty tokens and converts the one failure a model cannot detect into one it can reason about. The number matched tells it the scale of what it is missing, the number returned tells it what it holds, the ordering tells it which sample this is, and the cursor or the filter list tells it where to go next — and a payload carrying all four supports a true sentence about an incomplete answer, which is the best available outcome when the answer genuinely does not fit.

One field in that payload has been carried along without being examined. next_cursor is doing the heaviest work in the whole contract, since it is the difference between a model that knows it is missing something and a model that can go and get it, and it was presented as an opaque base64 blob that resumes where the page ended.