Size guards and context budgets

A size guard is the loop most codebases have: assemble the prompt, count it, and while it is over the window, remove something and count again. A context budget is a statement made before assembly about how much of the window each component is allowed to occupy, together with a defined consequence when a component wants more. Both keep requests under the limit. Only one of them is able to say why a particular piece of text was the piece that got dropped.

The distinction sounds academic until a guard drops the wrong thing, which it does silently, because dropping text is its normal successful operation rather than an error condition. Nothing fails. The request is accepted, the answer is fluent, the latency looks ordinary, and the only trace is a log line saying that some number of messages were trimmed to fit.

What a budget has that a guard does not#

Here is the guard, in the form it usually takes:

def assemble(system, tools, chunks, history):
    parts = [system] + [json.dumps(t) for t in tools]
    parts += [c.text for c in chunks]
    parts += [m.content for m in history]
    while count(parts) > WINDOW:
        history.pop(0)                      # drop the oldest turn
        parts = rebuild(system, tools, chunks, history)
    return parts

Fourteen lines including the rebuild, and it does the job it was written to do, which is to make the provider accept the request. "It keeps us under the window" is a real property, and the temptation is to treat it as the only one that matters.

The row about acting before components are fetched is the one that does the damage, and the arithmetic behind it is worth doing rather than asserting. Retrieval runs with k=8, k being the number of chunks the retriever is asked to return, and it produces however many tokens that configuration implies — 11,890 on the deployment measured here — before the guard has any say. By the time the guard executes, three of the five slices have already been written and what is left for the fourth is a subtraction: start at the 32,000-token window, take off 1,462 tokens of system instructions, take off 3,118 of tool definitions, take off the 11,890 of retrieval that has just arrived, and the remainder is 15,530 tokens for history. A long troubleshooting session wants around 19,000, so the shortfall is a little under three and a half thousand tokens and history is the only slice the guard is in a position to reduce. History is therefore what it spends, and whether three of those eight chunks were near-duplicates never enters the calculation, because by the time the question could be asked the chunks are already in the string being measured.

Run the same subtraction with a cap in front of retrieval and the numbers move. A budget that told the retriever it had 9,000 tokens rather than letting it return whatever k=8 implied would have left the fixed slices unchanged and the remainder larger: 32,000 minus the same two fixed slices, minus 9,000 rather than 11,890, is a little over eighteen thousand tokens for history instead of 15,530. That is not by itself a fix. A session wanting 19,004 tokens still overflows, and the guard still has to remove something. What it changes is the size and the standing of the loss: the shortfall falls from 3,474 tokens to under six hundred, and the trade becomes a number somebody wrote in a config file rather than a consequence of which fetcher happened to run first.

The comparison worth holding is with memory allocation, and it is a comparison rather than an analogy — the two are the same problem with different units. A while (over_limit) free_oldest() loop keeps a process inside its resident set, and it is not what anyone means by an allocator, because an allocator answers the question "how much may this arena have" before any of the arenas start writing. The trim loop is the free-oldest loop: correct about the invariant it enforces, silent about every distribution question, and unable to distinguish between freeing a scratch buffer and freeing the page that held the request's identity. Where the comparison stops is that a process can ask for memory again after a free, and a dropped conversation turn is gone for the rest of the session — the assembler rebuilds the prompt from the session store on the next turn, and the trimmed turn is still in the store, but the same guard will drop it again for the same reason, every turn, for as long as the session lasts.

The order of operations is the bug#

Drawing the two pipelines side by side makes the failure structural rather than accidental:

In the guard, the window is discovered at the end and the caps do not exist. In the budget, the window is the first input and every fetch is bounded by a number computed from it. The B8 branch still exists, because a cap can be violated by a single oversized unit — one retrieved chunk larger than the retrieval cap, one user message longer than the history cap — but it is now an exception path with a policy attached rather than the normal way the system operates.

The loop is expensive even when it succeeds#

The trim loop's cost is not only the material it discards. Each iteration pops one message and rebuilds the prompt, and rebuilding means counting again, because measuring the total is the only way the loop can tell whether it has done enough. Take a session whose history arrives at 19,004 tokens against 15,530 tokens of room, so the loop has to shed 3,474 tokens from the front of a conversation whose opening turns were short:

iter  popped                       tokens   history    total   ceiling
  0   —                                 —    19,004   35,474    32,000
  1   user "we're on the EU instance"  102    18,902   35,372    32,000
  2   assistant acknowledgement         88    18,814   35,284    32,000
  3   user + pasted stack trace      1,204    17,610   34,080    32,000
  4   assistant analysis               940    16,670   33,140    32,000
  5   user follow-up                   610    16,060   32,530    32,000
  6   assistant remediation plan       690    15,370   31,840    32,000  send

Count the counts rather than the pops. There are six pops in that trace and seven rows, because the loop measures once before it decides to pop at all and once more after each pop to find out whether it can stop. At roughly 90 ms a call through the provider's count_tokens endpoint, seven counts are 630 ms of pre-flight latency spent deciding what to throw away, charged before the provider has seen the request at all.

Which sessions suffer most is then predictable rather than a matter of luck. The number of iterations is however many messages have to come off the front to cover the overrun, so it scales with how far over the ceiling the prompt is and inversely with the size of the messages at the front. Two sessions equally far over behave completely differently depending on how the conversation opened: one whose first message is a pasted stack trace sheds its overrun in a pop or two, while one that opens with a dozen short exchanges before anything substantial is said pops a dozen times to shed the same number of tokens and pays a count for every one of them. The worst pre-flight time observed on this deployment was 2.1 seconds across 23 iterations, which is that same 90 ms multiplied by the twenty-four counts twenty-three iterations require, and none of it appeared in the latency the dashboards attributed to the model, because the model had not been called yet.

There is a second cost the loop cannot see, and it follows from what a prompt cache is keyed on. Providers store the computed state of a prompt prefix and charge a read against it at roughly a tenth of the input price. The entry is found by matching an incoming request against the stored one from position zero forward, so a conversation whose opening turns are byte-identical from one turn to the next keeps finding the entry the previous turn wrote. history.pop(0) removes a message from the front. Position zero now holds different tokens, the match fails at the first comparison, and everything behind it is charged fresh however unchanged it is. A session that crosses the trimming threshold once therefore stops hitting the cache for the rest of its life, at exactly the point where its requests have become the most expensive ones you serve. Topic 4.14 works this mechanism through properly; what matters here is the direction, which is that the loop's repair makes every later request in that session cost more rather than less.

The third cost is a correctness problem rather than an efficiency one. history.pop(0) removes a single message, and in a session with tools the messages are not independent: an assistant turn carrying a tool_use block is followed by a user turn carrying the matching tool_result. The two are joined by an identifier: the tool_use block declares an id and the tool_result block that follows quotes it back, which is the only thing telling the model which of several outstanding calls this result belongs to. That id is resolved inside the request, because the model carries nothing over between calls and there is nowhere else it could be resolved from. Popping the assistant turn on its own therefore leaves a tool_result whose antecedent is gone — an id quoted in a request where it appears nowhere else — and the provider rejects it with a 400 naming an unexpected tool-result id. The rejection is not a strictness setting somebody could turn off. There is no answer to "which call is this the result of" once the call has been deleted. That was 0.3 percent of requests here, and because it requires a session long enough to trim and tool-heavy enough to contain pairs, it arrived in the tracker as three unreproducible bug reports filed over five weeks rather than as a pattern anyone could see.

The component the guard can never protect#

There is one slice the trim loop cannot defend under any circumstances, and it is not a matter of the loop being badly written. Follow what the loop has access to. It operates on the assembled input string. The completion is not in that string, because it does not exist yet — the whole point of sending the request is to find out what it says. So every token the loop can see is an input token, and every token it removes makes room for another input token rather than for an output one. Input and output draw on the same window, which means the loop's success condition and the completion's requirement are in direct competition, and the loop is only measuring one of them. A guard that trims until the input fits has, by construction, filled the window with input. It can bring a 34,000-token prompt down to 31,900 on a 32,000-token model, which is under the window and therefore a success by the only test the loop applies, and the space remaining for the answer is 32,000 minus 31,900, which is 100 tokens.

This is why the reserve has to be a subtraction from the window before assembly rather than a ceiling applied afterwards, and the difference between those two operations is the whole of it. max_tokens is a ceiling. It tells the provider the completion may not exceed some size, and a statement about how large something may not be says nothing about whether that size is available. A ceiling constrains the answer. It does not constrain the input, which is the thing that took the space. The reserve is a subtraction, and a subtraction changes what the assembler is allowed to fill. Tell the assembler the window is 32,000 and it will fill 32,000. Tell it the window is 32,000, the reserve is 4,000 and therefore its ceiling is 28,000, and there is no code path by which retrieval or history can reach the last 4,000 tokens, because no component is ever handed a number that would let it. The protection comes from the assembler never being told the space exists.

The smallest change that makes it a budget#

For a retrieval assistant with tools, the change is about twenty lines and requires no new infrastructure beyond a per-slice measurement of what the fixed components cost:

from dataclasses import dataclass

@dataclass(frozen=True)
class Budget:
    window: int
    reserve: int          # subtracted first — the completion's space
    system: int           # measured at deploy, asserted in CI
    tools: int            # measured at deploy, asserted in CI
    retrieval: int
    history: int

    def __post_init__(self):
        allocated = (self.reserve + self.system + self.tools
                     + self.retrieval + self.history)
        assert allocated <= self.window, f"over-allocated by {allocated - self.window}"

BUDGET = Budget(window=32_000, reserve=4_000, system=1_500,
                tools=3_200, retrieval=14_000, history=9_300)

def assemble(query, session, budget=BUDGET):
    chunks = retrieve(query, token_cap=budget.retrieval)     # cap is an input
    history = select_recent(session, token_cap=budget.history)
    req = build(SYSTEM, TOOLS, chunks, history)
    assert count(req) <= budget.window - budget.reserve
    req["max_tokens"] = budget.reserve
    return req

Three things changed and each does specific work. The caps are constructor arguments to the fetchers rather than a post-hoc trim, so retrieval stops at 14,000 tokens by returning fewer or smaller chunks instead of returning 18,000 and letting something else pay for it. The __post_init__ assertion makes over-allocation a startup failure, so the day someone raises the retrieval cap without lowering another slice, the service refuses to boot rather than silently reintroducing the original bug. And max_tokens is set from the same reserve that was subtracted, which ties the ceiling and the subtraction to one number instead of letting them drift apart.

Add the five numbers in that Budget call and the sum is exactly 32,000, which is worth noticing because it means this particular allocation has no slack in it. Every token of the window is spoken for, so the assertion passes by the narrowest margin an assertion can pass by, and the local tokenizer's small disagreement with the provider's counter — the fraction of a percent from the counting discussion above — has nowhere to go. A budget written this way works until a request lands at the top of the distribution, and the fix is a named slack line rather than a wider margin hidden inside one of the other slices.

The assertion belongs in the constructor rather than in the request path for a reason that is easy to skip past. Over-allocation is a property of the five constants, and the five constants do not change between requests, so checking them per request would run the same check on the same values a hundred thousand times a day and report the failure at whatever moment traffic first happened to reach it. Checking them once, when the object is built, reports the same failure at process start, in staging, attached to the deploy that caused it. The test is identical. The moment it fires is the whole value.

The other thing worth changing at the same time is the log line. trimmed 6 messages to fit records that something was removed and nothing about what it was competing with. A line emitted from a budget carries the caps alongside the spend, so a single request can be read on its own:

{"event": "budget.assembled", "window": 32000, "reserve": 4000,
 "caps":  {"system": 1500, "tools": 3200, "retrieval": 14000, "history": 9300},
 "spent": {"system": 1462, "tools": 3118, "retrieval": 13740, "history": 9300},
 "evicted": {"history_exchanges": 4}, "headroom": 380}

Every field there is a number that can be compared across requests and aggregated across a day, and the fields are related to each other in ways a reader can check. The input ceiling is the window less the reserve, 28,000. Add the four spent figures and the total falls short of that ceiling by exactly 380 tokens, which is what headroom reports, so a request whose headroom is small was a request that nearly did not fit and a request whose headroom is large was one where a slice came in well under its cap. Reading spent against caps row by row then says which slice it was: system and tools sit below their caps because the fixed material is smaller than the allowance it was given, retrieval came in under 14,000 because this query did not have that much material to return, and history is at 9,300 exactly.

A slice whose spent equals its cap is a slice that is binding — it wanted more and was refused, and the equality is the trace of the refusal rather than a coincidence. That is why evicted and caps belong on the same line as spent: the pair says that four exchanges were dropped and names the number that dropped them. Aggregated over a day, the fraction of requests where history is binding is a direct measurement of whether 9,300 tokens is the right number for this feature's traffic rather than a guess someone made in a design document. A slice that binds on almost nothing is oversized; a slice that binds on most requests is a cap the traffic has outgrown.

A budget can be the wrong budget#

Having a budget does not mean having a good one, and this is the honest limit of the argument. An allocation of 14,000 tokens to retrieval and 9,300 to history is a claim that those proportions serve the feature, and if the claim is wrong the system now fails predictably rather than arbitrarily. Predictable failure is genuinely easier to debug, because the same request shape produces the same content every time and a bad answer can be reproduced, but a budget that starves history on a feature where users build context over twenty turns is still a budget that produces bad answers. Making each slice cite evidence rather than a guess is a separate exercise, later in this topic.

A budget also says nothing about whether the tokens inside a slice are worth having. Eight retrieved chunks that fit inside a 14,000-token cap can still be three near-duplicates and one about the wrong product, exactly as they were in the session above, and the cap does not improve retrieval quality — it bounds retrieval's cost to everything else. Treating the budget as a relevance mechanism is a category error that leads to raising caps in response to bad answers.

Nor does the code above decide what happens when the caps are exceeded anyway. select_recent truncating to a cap is a policy — the most recent turns win, the oldest are dropped — and it is only one of several defensible ones, none of which the code argues for. Retrieval hitting its cap could mean fewer chunks, shorter chunks, or a re-query with a tighter filter. Those choices need to be decided in a fixed order rather than by whichever fetcher happened to be written first.

What the change buys, then, is not correctness. It is the shift from a system whose failures are determined by insertion order to one whose failures are determined by a number somebody wrote down, and the second kind can be argued with. A cap that starves history is visible in a log line and arguable in a review; a history.pop(0) that starved history is visible nowhere.

The immediate gap is the reserve. It has appeared twice now as a number subtracted before assembly, and the reason that subtraction is non-negotiable is a failure that produces no errors, no rejected requests, and a support queue full of answers that stop mid-sentence.