The context window budget

The context window is the maximum number of tokens a model will accept in one call, and for most hosted models it is shared between what you send and what the model generates. A conversation that resends its whole history on every turn approaches that ceiling monotonically, so every chat product eventually meets it. There are two ways to meet it. One is arithmetic performed before the call, on numbers you already have, in about three milliseconds. The other is a 400 from the provider, or worse, a successful call whose input was silently reduced by something that fitted it to the window without telling anyone.

{
  "error": {
    "message": "This model's maximum context length is 128000 tokens. However, your messages resulted in 129,431 tokens. Please reduce the length of the messages.",
    "type": "invalid_request_error",
    "code": "context_length_exceeded"
  }
}

That response is the same event as the arithmetic, observed after the fact and at the user's expense. Two things follow. The ceiling to compare against is not the number on the model card: tool definitions and images take their share before the first message is counted, so the figure the check needs is smaller than the published one. And detecting the boundary is separate work from having a policy for what happens at it. A measurement that fires nothing is a monitor rather than a fix, while the next chapter's compression strategies fail the other way, since nothing invokes them if nothing is watching.

The budget is one inequality#

The reason there is a hard number to check against at all, rather than a gradual slowdown, is that the maximum was fixed when the model was built. A model accepts sequences up to some length and no further, and positions beyond it have no defined behaviour rather than a degraded one, so a provider cannot serve an over-long request badly — it has nothing to serve. The length of the assembled sequence is therefore validated before any of the model runs, which is why the rejection above arrives quickly, costs nothing, and carries no partial output.

That validation is over the whole request, not over any one message, and this is where the first wrong belief usually sits. Reading "this model's maximum context length is 128000 tokens" as a per-message limit is tempting, because every other size limit in an HTTP API — body size, field length, upload size — does apply per item. Here the array is flattened into a single sequence before the model sees it, so a long conversation of short messages is in exactly as much trouble as one long message. The error's own wording settles it: your messages resulted in 129,431 tokens is one total for the array, not a figure for any message in it. And because generation continues that same sequence, every token the model emits is another position in the same fixed span, which is why room for the reply has to be a term in the budget rather than a separate allowance.

The check has three terms rather than two: the tokens in the assembled array, the tokens you are prepared to let the reply consume, and a margin for the difference between your count and the provider's.

import tiktoken

ENC = tiktoken.encoding_for_model("gpt-4o-mini")
PER_MESSAGE_OVERHEAD = 4      # role, delimiters, formatting the provider adds
WINDOW = 128_000
RESERVE_FOR_REPLY = 1_000
MARGIN = 0.02                 # 2% for counts we cannot see (tool schemas, provider framing)

def count_tokens(messages: list[dict]) -> int:
    return sum(len(ENC.encode(m["content"])) + PER_MESSAGE_OVERHEAD for m in messages)

def headroom(messages: list[dict], tools_tokens: int = 0) -> int:
    used = count_tokens(messages) + tools_tokens
    ceiling = int(WINDOW * (1 - MARGIN)) - RESERVE_FOR_REPLY
    return ceiling - used

def assemble_with_budget(conn, conversation_id: str) -> list[dict]:
    messages = assemble(conn, conversation_id)
    room = headroom(messages)
    log.info("budget", conversation_id=conversation_id,
             input_tokens=count_tokens(messages), headroom=room,
             utilisation=round(1 - room / WINDOW, 3))
    if room < 0:
        messages = apply_overflow_policy(messages, deficit=-room)
    return messages

count_tokens does the first term, and it is worth reading closely because both halves of the sum are load-bearing. The first half encodes each message's content, which means running it through the same tokeniser the model was built with and taking the length of the result. A token is a fragment of text from a fixed inventory the model was built around: a common English word is usually one fragment, while an identifier, a punctuation-dense JSON blob or a passage of non-English text is several. Nothing about that mapping is predictable from a word count or a character count, and that is why the check encodes the string instead of estimating from its length.

The second half is PER_MESSAGE_OVERHEAD, a flat four tokens charged for every message regardless of how long it is. The provider does not hand your array to the model as an array. It renders the array into one sequence, inserting markers that say where each message begins, which role it carries and where it ends, and those markers are tokens drawn from the same inventory as the words. So the count depends on how many messages there are as well as how long they are: a hundred short messages carry four hundred tokens of framing before a single word of content is counted. A conversation split into many small turns is therefore more expensive than the same text in fewer turns, which is not obvious from looking at the content alone.

The line that builds ceiling performs the other two subtractions, and what licenses them is that every quantity involved is measured in the same units against the same single sequence. WINDOW * (1 - MARGIN) with a margin of 0.02 leaves ninety-eight per cent of the window, holding back a proportional slice for tokens you cannot see, and subtracting RESERVE_FOR_REPLY removes the room set aside for output. headroom is what remains once used comes off that ceiling, so a negative value is not an error code but a count of the tokens you are over by — which is why apply_overflow_policy is handed deficit=-room, negating a negative to get a positive number of tokens that have to go.

Local counting is fast enough that there is no reason to skip it: tiktoken encodes roughly 1.5 to 3 million tokens per second on a modern core, so a 24,000-token conversation costs about 8 to 15 milliseconds, against a model call of 4,000 to 20,000 milliseconds. A provider-side token-counting endpoint costs a network round trip of 40 to 100 ms and is worth using only where no local tokeniser exists for the model.

Those millisecond figures are one division. Encoding cost is proportional to how much text there is, so the time is the token count divided by the throughput: 24,000 tokens at 3 million a second is eight thousandths of a second, and the same 24,000 at the slow end of 1.5 million a second is roughly twice that, which is the other end of the 8-to-15-millisecond figure. The division is legitimate because both quantities are counted in tokens, so the tokens cancel and what is left is seconds. The same relationship explains why the opening of this chapter quotes a smaller number than this paragraph does: the cost scales with the size of the array being counted, so a short conversation and a long one differ in what the count costs, not in which tokeniser is doing it. Set either figure against the model call it guards and the check is well under one per cent of the operation it protects, which is what makes skipping it indefensible.

The headroom value is the artefact that makes everything downstream possible. Logged on every turn, it becomes a distribution you can look at rather than a cliff you fall off: the p95 of utilisation across conversations tells you how close the fleet is running, and an alert on "5% of turns are above 80% utilisation" gives you weeks of warning before anyone sees a 400. Without that number the first signal is a customer, and the second is a graph of context_length_exceeded errors that tells you it has already been happening.

Where the ceiling actually sits#

The window is not the budget. Several things consume it before your conversation does, and the arithmetic is worth doing once per product rather than assuming.

Working one row end to end shows where the Room for history column comes from, and the long-context row is the one to work because it closes exactly. Start with the window, 200,000 tokens. Hold back the 2% margin, which leaves ninety-eight per cent of it, a hundred and ninety-six thousand. Subtract the 4,000 reserved for the reply, leaving a hundred and ninety-two thousand. Subtract the 900 the system prompt and tool definitions occupy, and what remains is 191,100 — the figure in the row. Each of those subtractions is permitted for the same reason: window, reserve and system prompt are all counted in positions in the one sequence established above, so they are commensurable and can be taken off one another.

The figures in that column carry a tilde because the margin has to be rounded somewhere, and the direction of that rounding is a decision rather than a detail. Rounding the ceiling down costs a few tokens of capacity. Rounding it up spends part of the one term in the budget that exists because the count is uncertain, and a budget that rounds in its own favour fails on its largest turn.

The last column is the same room divided by an assumed 400 tokens a turn. 191,100 divided by 400 is four hundred and seventy-seven turns and three quarters of another, and the row prints 477 because three quarters of a turn is not a turn — the four hundred and seventy-eighth does not fit, and truncating rather than rounding is what keeps the figure a guarantee instead of an average. The same division on the top row gives fifteen turns and three quarters from a room of 6,300, printed as 15. Read the column downwards and the intuition is plain: what buys turns is room, room is whatever survives three fixed deductions, and each of those deductions is paid again on every single call.

The last two rows are the ones that surprise teams. Tool definitions — the JSON descriptions of the functions you are willing to let the model ask you to run, which have a stage of their own later in the book and appear here only as a line item in the budget — are rendered into the prompt on the provider's side and counted against the window, so adding six tools with detailed JSON schemas can cost 2,500 tokens on every single turn of every conversation. Images are worse per item: a 1024×1024 screenshot costs on the order of 1,100 to 1,600 tokens depending on the model's tiling, so a support conversation where the customer attaches four photographs of a damaged parcel has spent 5,000 tokens on evidence before anybody types a word.

An image costs tokens for the same structural reason a tool schema does. The model consumes one sequence of positions and nothing else, so a picture has to become positions before it can be consulted at all: the vision front end cuts it into tiles and each tile takes a fixed number of positions. Cost therefore tracks the tiling rather than the file — a smaller JPEG of the same 1024×1024 picture buys nothing — and the tiling is a property of the model version, which is what the range above is made of. Four screenshots at the bottom of the 1,100-to-1,600 band come to four thousand four hundred tokens and at the top to six thousand four hundred: a spread of two thousand, more than twice the 900 that most rows here allot to system prompt and tool definitions together. An estimate whose uncertainty exceeds a whole line item is not one to set a threshold from, which is why the per-image figure is worth establishing by sending a single image and reading usage.prompt_tokens back, once per model version.

The same arithmetic explains a cost curve that surprises people looking at their first monthly bill for a chat product. Because every turn resends the whole conversation, the input tokens for a conversation of n turns grow with the square of n. A 60-turn escalation averaging 400 tokens per turn sends about 12,000 tokens on average per call and roughly 720,000 input tokens over the whole conversation, which is $0.11 at $0.15 per million — around thirty times what the same 24,000 tokens of content would cost if each turn were sent once. The budget check is therefore not only a correctness control. It is the place where you can see that number.

Both halves of that claim are worth deriving, because the shape is the part that transfers to any other product. Turn one sends one turn's worth of tokens. Turn two resends turn one and adds its own, so it sends two turns' worth. Turn k sends k turns' worth, for no reason more subtle than that assembly rebuilds the whole array on every call. Adding those up over a conversation of n turns is the sum of the whole numbers from 1 to n, which is n(n + 1) / 2, and that expression is dominated by its n-squared term as soon as n is at all large. Stated without the algebra: the average call carries roughly half the final conversation, and there are as many calls as there are turns.

Every figure in the escalation follows from that. 60 turns at 400 tokens each is 24,000 tokens of content, which is the conversation as a person would count it. The average call carries a little over half of that — the first call carries one turn and the last carries all sixty — which the sentence above rounds to 12,000. Sixty calls at 12,000 tokens each is 720,000 input tokens billed. Rates are quoted per million tokens, so the charge is the count divided by a million and multiplied by the rate: 720,000 divided by a million is a bit under three quarters, and a bit under three quarters of $0.15 rounds to the $0.11 above. The multiple at the end is nothing but the ratio of the two token counts, 720,000 against 24,000, which is thirty.

Prompt caching changes that curve and is worth understanding before the next chapter, because it interacts with everything done to shorten history. Providers that offer it hash a prefix of your array and charge a reduced rate — commonly around a tenth of the input price, sometimes with a small write premium — for the portion that matches a recent request on the same prefix. A conversation that only ever appends is close to the ideal case: turn 61's array is turn 60's array plus two messages, so almost all of it is a cache hit, and the effective cost of a 12,000-token call drops from $0.0018 towards $0.0004. The condition is that the prefix must be byte-identical, which is another reason assembly should be deterministic, and it is also the reason that any change to an early message invalidates the whole remainder. Editing the system prompt, re-rendering a facts block that sits near the front, or replacing forty old turns with a fresh summary all move the boundary and turn the next call into a full-price miss. That does not argue against compressing history; it argues for compressing it at fixed, infrequent boundaries rather than recomputing something near the front of the array on every turn.

The two prices in that paragraph are the same division as the bill above. 12,000 tokens at $0.15 per million is twelve thousandths of a million multiplied by the rate, which is the $0.0018 quoted. A tenth of the input price applied to the whole array would be a tenth of that, comfortably below $0.0004, so the figure given is not the floor — it is what an actual turn lands on, because the newest user message and the reply before it are new on this call and pay full price, and a prefix has to be written once at a small premium before it can be read cheaply. The saving is most of the input cost and it is not all of it.

That the prefix must be byte-identical is not a provider quirk; it follows from how the lookup can be made cheap. Recognising a reusable prefix by comparing stored and incoming sequences position by position would cost about as much as the work being skipped, so the prefix is identified by its hash instead. A hash gives no partial credit. Change one byte anywhere in it and the result bears no relation to the old one, so there is no nearest match and no notion of a conversation being mostly what it was. The behaviour is therefore a cliff and not a slope: editing an early message does not make the next call somewhat more expensive, it makes everything after the edit a full-price miss, and everything after an early edit is nearly the whole array. That asymmetry is what settles where compression belongs: a shorter array saves tokens on every call for the rest of the conversation, a broken prefix costs full price on one call only, and a rewrite that touches the front of the array every turn never holds a prefix still long enough to collect anything.

Two subtleties keep local counts from matching the provider's exactly. Per-message framing differs by model family, so the constant of 4 tokens above is a cl100k-era figure for chat completions and is not universal; the reliable way to calibrate it is to send a handful of arrays and reconcile your count against usage.prompt_tokens in the response, which typically lands within a few tokens once the overhead constant is right. Tool definitions are the larger source of drift, since the provider renders your JSON schemas into the prompt in a format you do not control and cannot count locally with confidence. Reconciling once per model and per tool set, then carrying the measured figure as a constant, is more accurate than estimating, and a 2% margin absorbs what remains.

Underneath both of those is one chain worth stating outright, because it is what stops the margin looking arbitrary. You count content. The provider counts the sequence it built from your request. Between the two sit the per-message framing, the rendering of your tool schemas through a template you do not have, and whatever else the platform prepends — each of them a string you never see, and therefore none of them encodable locally at any level of effort. The belief that catches people here is that owning the model's tokeniser makes the count exact; it is tempting because on the content the tokeniser genuinely is exact, and it is wrong because the content is not the whole request. The margin is a proportion rather than a fixed subtraction because the largest unseen term is the framing, the framing scales with the number of messages, and the number of messages is precisely what a conversation keeps adding to. It also settles what the reconciliation above has to cover: the model and the tool set actually deployed, since plain text is the one case where the local count was already right.

Detecting is not deciding#

Having the measurement makes it tempting to treat detection as the fix, and there is a family of responses that feel like handling the limit while being ways of finding out about it later or of postponing it.

Catching the 400 and retrying with fewer messages is reactive by construction: it fires after a user has waited for a failed round trip, it discards a variable and unpredictable amount of history under time pressure, and the amount it discards depends on how far over the line you happened to land. Raising max_tokens does nothing, since that reserve is part of the budget you are already exceeding. An alert on context_length_exceeded tells you the boundary was crossed, which is monitoring rather than behaviour. Moving to a model with a larger window multiplies the turn count you can survive — from 15 to 309 in the table above — without changing the fact that the conversation grows without bound; it converts a daily problem into a quarterly one, which is worth money and is not a policy.

Two of those are worth taking a step further, because each is refused for a reason rather than by assertion. max_tokens cannot help because the reserve enters the budget as a subtraction from the ceiling: raise it by some quantity and ceiling falls by exactly that quantity, so headroom drops one for one and an array that did not fit still does not fit. It feels as though it ought to help because the error names a maximum length, and raising a limit is the usual response to a limit — but the limit being raised is the one on output, while the quantity that overflowed was the total of input and output together.

The larger window works for a reason the table states and fails for one it does not. Room for history rises from 6,300 to 123,600, which is where the jump from 15 turns to 309 comes from: roughly twenty times as many turns before the ceiling. What the table does not price is what those turns cost. Input tokens grow with the square of the turn count, established above, so a conversation twenty times longer bills on the order of four hundred times as many input tokens as the short one did. The bigger window buys time, and the bill for the time it buys rises faster than the capacity does.

Client-side trimming is the most dangerous member of the family, because it produces a successful call. Anything that truncates to fit needs to be told what is not eligible for removal, and the system message is the obvious first entry on that list:

def trim_to_fit(messages: list[dict], budget: int) -> list[dict]:
    pinned = [m for m in messages if m["role"] == "system"]
    rest = [m for m in messages if m["role"] != "system"]
    room = budget - count_tokens(pinned)
    kept: list[dict] = []
    for m in reversed(rest):                       # newest first
        cost = len(ENC.encode(m["content"])) + PER_MESSAGE_OVERHEAD
        if cost > room:
            break
        room -= cost
        kept.append(m)
    dropped = len(rest) - len(kept)
    if dropped:
        log.warning("history_truncated", dropped=dropped, kept=len(kept))
    return pinned + list(reversed(kept))

Every line of that function answers a way the obvious version goes wrong. Partitioning by role first means a message carrying system is never a candidate for removal, which makes the scenario's failure structurally impossible rather than merely unlikely. Computing room after the pinned messages keeps the walk safe: pinned tokens are spent whatever else is decided, and the loop must not offer them twice. It walks newest-first because the current user message is what the reply most depends on, and it breaks rather than continues at the first message that will not fit — continuing would keep a short old message after dropping a longer newer one, leaving a gap in the middle, and a gap is worse than a truncation because the model reads position as sequence and will read straight across the join as though nothing had been taken out. reversed(kept) then puts the survivors back into chronological order before the array goes out.

Even corrected, that function is a floor rather than an answer. It emits a warning, it preserves instructions, it keeps the recent turns that the current message most likely depends on, and it still deletes the beginning of the conversation, which in a returns flow is where the order number and the original complaint were established. What makes something a policy rather than a symptom-handler is that it is defined in advance, applies at a fixed boundary, and states what happens to the information it removes — for instance, that everything older than the last twenty turns is condensed into a single summary message that remains in the array, with the recent turns kept verbatim. That statement can be reviewed, tested, and reasoned about by someone who was not in the room when it was written.

The boundary also should not sit at the limit. Firing an overflow policy at 99% utilisation means the summarisation or extraction call that the policy needs has to run inside a turn that is already at the edge, adding 2 to 4 seconds to a user's wait at the worst possible moment, and leaving no room if the resulting summary comes back longer than expected. Choosing a trigger at 60 to 70% utilisation gives the policy space to run, room to be wrong about its own output size, and — where the work can be done after the response has been streamed rather than before it — the option of running it entirely off the user's critical path. The specific figure follows from the tail of your turn sizes: if a single turn can carry four attached photographs, the headroom you need in reserve is larger than a text-only product's.

The reason that figure comes off the tail rather than the average is that one turn can move utilisation a long way. On a 128k window a trigger at 60 to 70% leaves roughly a third of the window unspent, which is tens of thousands of tokens of room for a summariser whose output comes back longer than expected; the same policy at 99% leaves about a hundredth of the window, and one overrun consumes it. Attachments are what make that tail long, because a turn carrying them shifts utilisation by thousands of tokens on its own, so a threshold calibrated on the average turn can be comfortably clear at one check and past the ceiling at the next.

Two limits on the measurement itself are worth holding. Your count of the input can be exact if you use the right tokeniser and account for per-message framing, but the output length is unknown until it has been generated, so the reserve is a bet; a model that decides to emit a 3,000-token itemised breakdown when you reserved 1,000 will be cut off mid-sentence, and the fix is a max_tokens that matches the reserve so the truncation is at least yours. And staying under the window says nothing about whether the model can use what is in it. The mid-context attention effect from chapter 3 does not care that your utilisation is 62%; a fact at position 12 of 61 can be present, within budget, and still missed.

Those two limits fail in opposite directions, and the difference is what stops a healthy dashboard being read as an all-clear. Too small a reserve returns a reply that stops mid-sentence; comfortable utilisation returns a reply that ignored something in the middle of its input. The budget governs what the provider will accept, attention governs what the model will use, and clearing one says nothing whatever about the other.

Getting the measurement in place before it is needed is a half-day of work and answers a question teams usually cannot answer at all, which is how much of their traffic is anywhere near the boundary. A histogram of utilisation bucketed by conversation length, plus a one-off query over live conversations estimating tokens from stored content length, is enough to tell you whether the overflow policy is urgent work or something to schedule. A product whose 99th-percentile conversation is 18 turns and 7,000 tokens against a 128k window is not going to overflow this year, and knowing that is worth as much as the alert.

The budget check gives you a boundary and a moment. What happens at that moment — condensing old turns into a summary, keeping a small set of extracted facts and dropping the rest, or fetching only the past turns that bear on the current message — is a choice between three answers that fail in different places, and in a refund conversation the differences are not academic. That is the next chapter.