Counting tokens per component

A model's context window is the fixed number of tokens it can take in and give back on one call, counted fresh every time, because the model carries nothing over between calls. Everything a request needs has to fit inside it at once — the standing instructions the assistant works from, the descriptions of the tools it is allowed to call, whatever the retriever pulled out of a knowledge base for this particular question, the conversation so far, and the answer that has not been generated yet. There is no overflow behaviour to fall back on, since a request one token too large is refused outright, which is why a size check runs before the call goes out rather than after it comes back.

Deciding in advance how much of that window each of those five parts may occupy, and what happens when one of them wants more, is what this topic means by a context budget; the practice of writing one and holding a system to it travels under the name context engineering. It exists because the window is a fixed resource that several independent pieces of code write into without consulting each other. The retriever does not know how long the session has become, the session does not know that the system prompt grew by 400 tokens last sprint, and nothing at all is holding room for the reply. You meet the problem wherever a prompt is assembled from more than one source — a retrieval assistant, an agent with a registry of tools, a summariser fed whole documents — and you usually meet it as requests refused at the boundary or answers that stop in the middle of a sentence. Before any of it can be allocated it has to be counted, and counting is where most deployments are already wrong.

The same guard, measured against the tokenizer#

Topic 2.14 established why a character-count estimate breaks: characters per token is a property of the text rather than a constant, so no safety margin removes a content-dependent error. Here are five samples drawn from one afternoon of production traffic on a support assistant, each counted with the target model's own tokenizer rather than estimated:

Read one row of that all the way through before trusting the other four. The English ticket body is 1,840 characters, and the tokenizer returns 402 tokens for it, so each token is standing in for 1,840 divided by 402, which is 4.58 characters. The estimator has no access to that ratio. It divides 1,840 by four and returns 460, and 460 against a true 402 is 58 tokens too many, which is fourteen percent of 402 — the figure in the last column. Fix the sign convention now rather than later: a positive error is the estimator claiming the text is larger than it is, which wastes capacity, and a negative error is the estimator claiming it is smaller, which is the direction that gets a request refused.

The ratio moves because of how the vocabulary was built, and this is the layer where the wrong belief usually sits. "About four characters to a token" is treated as a property of tokenizers. It is a property of English. A byte-pair tokenizer starts from single bytes and repeatedly merges whichever adjacent pair occurs most often in its training corpus, storing each merged sequence as one entry in a vocabulary of fixed size. A sequence that was common in that corpus therefore ends up as a single id; everything else has to be spelled out of smaller pieces that did make the list. Ordinary English words were common, so most of them cost one token and carry four or five characters each. A {" run, a four-space indent, a CamelCase identifier and a Japanese character were each rarer as sequences, so each is assembled from two, three or four pieces, and the characters-per-token figure falls accordingly. Nothing about the tokenizer differs across those five rows. The text does.

The spread across the ratio column is four to one, and the direction of the error is what makes such a failure sudden rather than gradual. English prose sits slightly above four characters per token, so a divide-by-four estimator runs conservative and every miss is in the safe direction. JSON — JavaScript Object Notation, the brace-and-quote format services write structured logs in — along with source code and non-Latin scripts sits below four, sometimes far below, so the estimator flips from over-counting to under-counting the moment the traffic mix changes. A guard that is wrong by −37 percent on a slice that is 12,000 characters long is wrong by roughly 1,800 tokens, and the two numbers that produce that figure are both in the table: the estimator hands back 3,100 for that attachment where the tokenizer returns 4,930, and the difference between the two is the entire error. A request the guard scored at 29,600 tokens is therefore really carrying a little over thirty-one thousand, and the guard has just certified it against a 32,000-token window with under a thousand tokens left over for an answer it was never measuring in the first place.

That is also the point at which the obvious repair stops working. Lowering the threshold the guard compares against, or dividing by three and a half instead of four, is tempting because it is one constant, it takes one line, and it demonstrably makes the failing requests pass. It fails because the quantity it is compensating for is not a fixed offset. The error is a percentage of a slice whose size the user chooses, so whatever margin absorbs a twelve-thousand character log is exposed by a log half again as long, and a margin set wide enough to absorb the Japanese case gives away most of the window on every English request. There is no constant that is simultaneously safe on 1.15 characters per token and not wasteful on 4.58.

The tokenizer you count with matters as well as the fact that you count. Running the same JSON attachment through two model families' tokenizers gave 4,930 tokens and 5,610 tokens, a 14 percent gap; the Japanese sample gave 531 and 402, a 32 percent gap in the other direction. Each of those percentages is the difference between the pair taken against the smaller of the two, which is the conservative way to state it. The JSON pair differ by nearly seven hundred tokens against a base of 4,930; the Japanese pair differ by a little over a hundred against a base of 402. The second is much the larger relative gap despite being far the smaller absolute one, which is the reason a percentage is the useful form here and a raw token difference is not. Byte-pair vocabularies are trained on different corpora with different vocabulary sizes, so their handling of punctuation runs, indentation and multi-byte scripts diverges most exactly where prose diverges least: the merges that turn the into one id are learned by any corpus of English, while the merges that turn a four-space indent or a common kanji into one id depend on how much code and how much Japanese that particular corpus happened to contain. A budget derived on one model family and carried across to another is a budget with an unmeasured error bar of ten to thirty percent on the slices that are hardest to bound.

Counting locally is cheaper and slightly wrong#

There are two places the count can happen, and they do not agree with each other. A vendored tokenizer file runs in-process and counts an assembled 25,000-token prompt in about 0.8 ms, which is cheap enough for every request and for a unit test. The provider's own counting endpoint runs over the network at roughly 90 ms a call and draws on the same rate limit as your traffic, and in exchange it counts the request as the provider will actually construct it. On one deployment the two disagreed by 0.9 to 1.4 percent, with the local count always the lower of the pair, because the local tokenizer sees text where the endpoint sees a request: role delimiters around every message, the serialised envelope around each tool schema, the wrapper on every tool result, and the cache-control markers the SDK inserts.

Every item in that list is text the local tokenizer never sees, because it is added by the provider or the SDK after your string has been handed over. That is the causal step, and it fixes the sign of the disagreement rather than leaving it as an observed coincidence: the endpoint counts a superset of what the local tokenizer counts, so the local number is lower, always, and it is lower by an amount that grows with the number of messages and tool schemas in the request rather than with the number of characters in them.

A 1.2 percent gap on a 12,830-token median request is 154 tokens, which nothing anywhere notices. The same gap on a 31,600-token request is 380 tokens, which is the whole distance between a request that clears a 32,000-token window and one the provider rejects. Both figures are the same multiplication — the request size times the fractional gap — and the reason one is harmless and the other is fatal has nothing to do with the gap and everything to do with what is left between the request and the ceiling. At the median there are nineteen thousand tokens of headroom and the error disappears into them; at 31,600 there are four hundred, and a 380-token error is most of that. A percentage error is dangerous in proportion to how close you are running to the wall, which is why a correction that is invisible in aggregate is the one that decides the boundary cases. The arrangement that works is to count locally on every request with the measured gap added back as an explicit correction, and to re-derive that correction nightly from a sample of a few hundred real requests, so that a model upgrade which shifts the gap is reported by a job rather than by the boundary traffic.

Five slices, five distributions#

The five components of a prompt behave differently enough that a single measurement of the total tells you almost nothing about which one to worry about.

Two of these are constants that you pay on every request and can measure once per deploy. Two are distributions whose tails are set by user behaviour rather than by your code. The fifth is not text at all, which is why it is the one that never appears in a character count and the one that gets silently spent. That asymmetry is the whole reason a per-component measurement is worth the work: the constant slices are cheap to bound and easy to forget, the variable slices need percentiles rather than a single number, and the reserve needs to be subtracted rather than measured.

Measuring them means counting each slice separately, with the target model's tokenizer, before they are joined:

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-4-5"

def count(system: str, tools: list[dict], messages: list[dict]) -> int:
    r = client.messages.count_tokens(
        model=MODEL, system=system, tools=tools, messages=messages
    )
    return r.input_tokens

def measure(req) -> dict[str, int]:
    """Count each slice by difference, so framing tokens land somewhere."""
    empty = [{"role": "user", "content": "."}]
    base = count("", [], empty)
    sys_only = count(req.system, [], empty)
    sys_tools = count(req.system, req.tools, empty)
    full = count(req.system, req.tools, req.messages)
    retrieval_only = count(req.system, req.tools, req.retrieval_messages)
    return {
        "system": sys_only - base,
        "tools": sys_tools - sys_only,
        "retrieval": retrieval_only - sys_tools,
        "history": full - retrieval_only,
        "total": full,
    }

Counting by difference rather than by tokenising each string in isolation matters because the request carries structure that no individual string contains — role framing around every message, the serialised schema envelope around each tool. Topic 2.14 covered where those tokens come from; the point here is that they have to be attributed to a slice rather than left as an unexplained remainder, or your five component numbers will sum to less than the total and you will not know by how much.

Subtraction is the right operation for a reason worth stating, because the obvious alternative looks equivalent and is not. Tokenising the system string on its own answers "how many tokens is this string", which is a question about a string. What the budget needs is "how much larger does the request get when this component is present", which is a question about a request, and the only way to ask it is to build the request twice and take the difference. sys_only - base is the answer for system instructions: base is a request with an almost empty user turn and nothing else, sys_only is that same request with the system block attached, and everything the provider wraps around that block on the way in is inside the difference. The subtractions then chain, each one adding exactly one component to the previous request, so sys_tools - sys_only is the tool block plus its envelope and nothing else. Every token in the request lands in exactly one of the five figures, and the figures add up to full by construction rather than by luck.

Run that over a week of replayed production traffic and the output is a report rather than a number:

component      p50      p95      max
system       1,462    1,462    1,462
tools        3,118    3,118    3,118
retrieval    6,240   11,890   18,430
history      2,010    9,470   22,110
---------------------------------------
total       12,830   25,940   44,120     window 32,000

The p50 column, the median size that half of all requests stay under, is why nothing looks wrong for a year at a time: 12,830 against a window of 32,000 is 40 percent of it, so a typical request has three fifths of the window sitting unused and no amount of estimator error is going to reach the ceiling. The p95 beside it is the size only one request in twenty exceeds. The max column is where refusals come from, and it also shows that they are not evenly caused. Retrieval and history are the two slices that reach 18,430 and 22,110 tokens at their worst, which is 40,540 tokens between them, and each of them individually can exceed the entire budget left over after the fixed slices are paid.

The max column is also the one most likely to be read wrongly, and the table contains the evidence for that in itself. Adding the four maxima gives 40,540 plus the two fixed slices, which is a thousand tokens more than the 44,120 on the total line — and the total line is not a mistake. Each column is a percentile taken down its own slice independently, so the max row records the largest system block ever seen, the largest retrieval ever seen and the largest history ever seen, and those three were not the same request. The largest request actually observed is what the total column reports, and it is smaller than the sum because the day someone pasted a very long log was not the day retrieval returned its densest chunks. This matters in the one direction that is easy to get backwards: the sum of maxima is not a request that ever happened, but it is a request that could, so it is the right number for sizing a budget and the wrong number for describing traffic.

Worst case is a corpus you assemble deliberately#

That report is only as honest as the traffic fed into it, and replayed production traffic under-represents exactly the inputs that break things, because the requests that failed are missing from the logs. The chain is short enough to check link by link. A pre-flight guard refuses an oversized request before it is sent. A request that was never sent produces no provider response and usually no request-log row, since the row is written when the call returns. So the log contains requests that passed the guard, and passing the guard is the same thing as being small — which means the distribution you are replaying has had its right tail removed by the very component you are trying to measure. The tempting reading of a clean max column is that nothing large arrives. The correct reading is that nothing large survives to be recorded.

Constructing a worst-case corpus is a deliberate exercise: take the largest attachment your upload limit permits and see what it tokenises to, take the longest session from the last thirty days rather than the median, run retrieval at maximum k against the documents in your corpus with the densest tables, and include every locale you accept rather than the one your team speaks.

The inputs with no characters to divide#

An image contributes tokens as a function of its pixel dimensions after the provider's internal resize, so the character count of an assembled prompt returns the same number whether a ticket carries no images or four of them.

The step underneath that is short and it removes the mystery. A model does not have a way of consuming pixels directly; the only thing it consumes is a sequence of vectors. An image is therefore cut into a grid of small square patches, each patch is projected into a vector of the same shape a text token becomes, and those vectors are inserted into the sequence alongside the text. The number of patches is the image's area divided by the area of one patch, so the cost follows the pixel dimensions and follows nothing else — not the file size, not the format, not how much of the picture is blank. The resize happens first, which is why the phone screenshot below is charged on the dimensions in the second half of its cell rather than the first. And because none of this passes through a character count at any point, an estimator that reads characters is not under-counting an image. It is counting a different thing that happens to be zero.

The two failure modes present differently enough that separating them is worth a query, because the fix for one does nothing for the other. The ratio error scales with the length of the text, so it grows smoothly as inputs get longer and a generous safety margin hides it until the traffic mix changes underneath it. The image error has no relationship to the text at all, which means a ticket whose body is 300 characters and whose four screenshots come to 8,900 tokens is scored at 75 tokens by any estimator that reads characters — 300 divided by four, and the screenshots contributing nothing to the numerator because they contribute no characters. The true size is that 75 plus 8,900, so the estimator is not out by a percentage here. It is out by two orders of magnitude, and it will be out by more the next time somebody attaches five screenshots instead of four. Bucketing the rejected requests by what the estimator had scored them tells you which one you have: ratio failures cluster just under the threshold, at 85 to 99 percent of the guard's ceiling, while image failures sit at 5 to 20 percent of it and look, from the estimator's point of view, like small requests the provider refused for no reason.

The correction happens at ingestion rather than at assembly: each image is counted through the provider's endpoint as a content block on upload, the cost is stored alongside the file, and the assembler adds those stored counts to its text count. A ticket carrying eleven screenshots then reports 19,700 tokens before a word of it is read.

Observed size is not deserved size#

Measurement tells you what each slice currently costs, and it does not tell you what each slice deserves. A report saying retrieval reaches 18,430 tokens at its maximum is a fact about a retrieval configuration, not a judgement that retrieval is entitled to 18,430 tokens, and treating the observed size as the required size is the most common way a measurement exercise ends up ratifying whatever the code already did. The step from observed sizes to defended entitlements needs evidence of a different kind — what happens to answer quality when you take tokens away — and that comes later in this topic, where each slice has to cite a measurement.

Counting is also not permanent. Tokenizer vocabularies are versioned with the model, so a minor model upgrade can shift your counts by a few percent, and a move across model families can shift them by thirty. The fixed slices drift for a more mundane reason: someone adds a tool, or expands a description from one sentence to four, and the tool block that measured 3,118 tokens at the last audit is 4,400 today with no code change that looks like a context change. A measurement that is not re-run in CI is a measurement with a shelf life of about one sprint.

What five numbers buy, then, is not safety. It is the ability to say which slice grew, by how much, and against which worst case — the thing a single total can never tell you, however carefully that total is computed.

The immediate consequence of having them is that the guard which produced them is now visibly inadequate. It knows the total, it knows the window, and when the total is too large it does the only thing it can do, which is to start removing text until the number comes down. That behaviour has a name in most codebases — the size guard — and it is not the same thing as a budget, for reasons worth being precise about.