max_tokens and length instructions

Every request to a text-generation API carries a ceiling on how many tokens the response may contain. On Anthropic's Messages API it is the required parameter max_tokens; on OpenAI's chat endpoint it is an optional field that defaults to the model's own limit. It sits in the request body next to system and messages, which is a large part of why it gets mistaken for something the model is told about.

It is not. The number is read by the serving layer — the infrastructure that drives the decode loop and assembles the response — and the model has no way to see it. This chapter is about the consequences of that separation: what the counter actually does when it expires, what a length instruction in the prompt does instead, how to size a cap from data you already have in your logs, and why a cap that was comfortable last quarter can start truncating after a model change that touched nothing else.

Where the token counter lives#

The decode loop produces one token per step. Around that loop the serving layer keeps a count, initialised to your max_tokens and decremented on each emission. When it reaches zero the loop is halted and the response is assembled from whatever has been produced so far, with stop_reason set to max_tokens. The halt is unconditional and position-blind. It does not wait for a sentence boundary, a closing brace, or the end of a word — if the count runs out between the Accou and the nt of "Account", the response ends there.

None of that is an accident of one vendor's implementation, and the reason is in how a request is taken apart before anything runs. The JSON body you send holds two kinds of field, sitting side by side and looking alike. system and messages are content: the serving layer renders them into tokens, lays those tokens into the context window, and the model reads every one. max_tokens, stop_sequences and temperature are control: they configure the loop running around the model, and they are never rendered into tokens at all. A field in the second group cannot shape the answer the way an instruction does, and the reason is mechanical rather than a matter of degree. Shaping the answer means changing the probabilities at some sampling step, the probabilities are computed from the tokens in the context, and the cap never becomes a token in the context. So the model generates as though no cap existed, and the cap is applied to that generation from outside it. Being halted between the Accou and the nt is what applying a limit from outside looks like when the thing inside was never told.

A small piece of API design keeps the misreading alive. Because max_tokens is required on the Messages API, every developer writing their first request has to choose a value before having any idea what a reasonable one looks like, and the natural way to choose is to think about how long you want the answer to be. On the endpoint where the field is optional and defaults to the model's ceiling, the same developer never thinks about it at all until something truncates. Both routes lead to the same misconception from opposite directions: one makes the number feel like a specification of the answer, the other makes it feel like a formality.

A second belief travels with the first and is harder to shake, because the rest of an engineer's experience supports it: that declaring a maximum reserves something. A buffer size allocates memory. A queue's capacity is space held whether anything is in it or not. A reserved instance is paid for while idle. A required field called max_tokens therefore reads like a request for an allocation, and the conclusion that follows is that a high value is expensive and a low one is thrifty. Neither follows. On the hosted endpoints this chapter is about, the number is a counter with a starting value and nothing is set aside; the meter runs on tokens the model actually produced. A cap many times your typical answer length therefore costs exactly what a snug one costs on every request that ends naturally, because on those requests the counter never reaches zero and the difference between the two values is never exercised. What a large cap does change is the worst case — how long a single connection may stay open, and how many tokens a runaway generation may bill you for before anything stops it. Bounding those two is the entire job of the parameter.

Contrast that with what an instruction in the prompt does. "Answer in one or two sentences" is text in the context window, so it participates in every sampling step: it shifts probability mass toward shorter formulations, toward getting to the point, and toward emitting the end-of-sequence token earlier. The model complies because compliance is what it was trained to do with instructions, and when it complies the answer ends at a natural boundary with stop_reason: "end_turn". What it does not give you is a guarantee. A model asked for two sentences will occasionally write five.

The two mechanisms have exactly complementary properties, and the mistake is using either one alone:

The last row is the newest of the four and the one that most directly targets what people wrongly expect the cap to do. Anthropic's task budgets let you declare a token allowance for an entire agentic loop, and the serving layer injects a countdown marker that the model actually sees during generation, so it can pace its work and wrap up gracefully rather than being guillotined. It is advisory rather than enforced, denominated in tokens across a whole loop rather than one response, and it is explicitly not a replacement for max_tokens — the documentation describes the two as distinct, with max_tokens remaining the hard per-response ceiling the model is unaware of. If you have ever wished the cap would make the model finish early instead of stopping it dead, the wish has a name and it is a different parameter.

Sizing a cap from logged output lengths#

Most caps in most codebases were picked by intuition and never revisited, which is why the sizing procedure is worth writing down. Log usage.output_tokens on every response for a week. Take the 99th percentile of that distribution and multiply by two to three. That is your cap.

The multiplier is there because the distribution you measured is itself censored: any request that already hit the old cap contributed a truncated length rather than its true one, so your measured tail understates the real tail, and the understatement is worst exactly where you care. If more than about 1% of your traffic is hitting the cap, the measurement is unreliable enough that you should raise the cap first, re-measure, and then size properly.

The censoring is worth working through concretely, because until you do the correction looks like superstition. Suppose the cap in force while you were logging was 120, and one request would naturally have run to three hundred tokens. That request is not logged as three hundred. It is logged as 120, since the counter halted the loop there and 120 is what usage.output_tokens counted. Every request in that situation is logged at the same value, so the histogram you end up holding has a wall standing at the cap and nothing at all beyond it. Now take the 99th percentile of that histogram. It cannot come out above 120, because no observation in the data is above 120, and a percentile is a value drawn from the observations. Sizing a cap from that number hands you back the cap you already had, and it does so most confidently for the requests whose true lengths were the ones erased. Multiplying by two or three is a blunt admission that the tail you cannot see runs some unknown distance past the wall. The one-percent rule is the point below which the wall is thin enough that the rest of the distribution is still describing reality rather than describing your configuration.

A cap sized this way costs nothing on ordinary requests, because you are billed for tokens generated rather than tokens permitted. Raising a cap from 120 to 512 changes the bill by zero when the answers that were previously truncated at 120 run to a natural 31 to 90 tokens, and the ones already under the cap were never affected. The only thing a larger cap costs is a larger worst case on the pathological request, which is precisely the thing you are keeping a cap for.

For converting between words and tokens while sizing, the working rule of thumb for English prose is about 0.75 words per token, so 512 tokens is roughly 380 words. Do not lean on that for anything that matters. Tokenizers differ between model families and even between generations of the same family — Anthropic's Sonnet 5 tokenizer produces roughly 30% more tokens for the same text than Sonnet 4.6 did — and code, JSON and non-English text tokenize far less efficiently than prose. When the number matters, count it with the provider's own endpoint (client.messages.count_tokens) against the exact model you will call, and never with a tokenizer built for a different vendor's models.

The cap covers more than the answer#

On models with thinking enabled — where the model works through the problem in a stretch of generated text before it starts on the answer you will actually show anyone — there is a second reason to size generously, and it catches people during migrations. The reasoning tokens produced before the visible answer are output tokens, and they come out of the same budget. A route tuned tightly around answer length on a model where thinking was off will start truncating when moved to a model where thinking is on by default — Claude Opus 5 thinks unless you explicitly disable it, whereas Opus 4.8 did not think unless you asked. Nothing about the request changed except the model string, and suddenly a cap that used to be comfortable is being consumed by reasoning before the answer starts.

At high effort settings this can be dramatic. The guidance for running Opus-tier models at xhigh or max effort — the request-time dial that governs how much thinking the model is willing to spend before answering — is to start max_tokens at 64,000, which is two orders of magnitude above what the visible answer needs, precisely because the budget has to cover the thinking. A tight cap on a thinking model produces a response that is almost entirely reasoning followed by a truncated answer, and a stop_reason of max_tokens to tell you so.

The symptom is distinctive enough to recognise once you have seen it. Output token counts jump by an order of magnitude with no change to the visible answers, then a fraction of responses start arriving short or empty, and the fraction grows with the difficulty of the input, because harder inputs provoke more thinking. Every one of those responses reports max_tokens. Raising the cap fixes it; lowering the effort setting also fixes it, at the cost of the reasoning depth you were presumably paying for.

Very large caps interact with one more piece of infrastructure. Non-streaming requests hold an HTTP connection open for the whole generation, and SDK and proxy timeouts are not sized for a 64,000-token response, so past roughly 16,000 tokens the recommendation is to stream. What you get otherwise is a timeout — a genuine transport error rather than a stop reason — and the failure mode is a request you paid for whose output you never received.

Three things the cap is not#

It is not a cost control in the way people assume. You are billed for the tokens that were generated, and a truncated response has generated all the tokens up to the cap and produced something you cannot use, so the marginal effect of a too-tight cap is to guarantee you pay full price for output that gets thrown away and retried. A cap protects you from the runaway case — the prompt injection that asks for a novel, the loop that never emits an end-of-sequence token — and that protection is genuinely worth having. It does not shave money off ordinary requests.

The direction of that error is worth being exact about, because a tight cap can raise a bill rather than lower it. Take one request whose natural answer is longer than the cap, and count what it costs to end up with a usable answer. Set generously, the cap never fires, and you pay for the answer once. Set tightly, the loop runs until the counter empties, so you are billed for a full cap's worth of tokens and hold something you cannot use; the retry that follows regenerates the whole answer from the beginning, at its full natural length, because nothing about the first attempt is reusable. The first attempt is therefore added to the bill rather than deducted from it. The only case where the tight cap comes out ahead is the one where the truncated text was good enough to keep, and if that were true of your traffic the answers would have been that short in the first place.

It is not a latency control either, except in the same crude sense. Output tokens dominate wall-clock time on most requests, so halving the cap does bound the worst case, but the typical response was never going to hit the cap and so its latency is unchanged. If the median request feels slow, the cap is not the lever; the effort setting, the model choice, and streaming are.

And it is not the context window. Exhausting the model's input capacity is a separate condition with its own reported reason, model_context_window_exceeded, and raising max_tokens will not help with it. A request whose conversation history no longer fits needs compaction or trimming, and treating that as a cap problem sends you to a fix that cannot work.

The division of labour that comes out of all this is narrow enough to apply while reading a diff. Length is a property of the answer, so it belongs in the prompt, where the model can act on it. The cap is a bound on the worst case, so it belongs well above the tail you measured, where it fires on pathology rather than on traffic. Any configuration that inverts those two — a tight cap doing the shortening, a vague instruction doing the bounding — produces articulate sentences that end in the middle.

Every mechanism in this topic so far hands back text that reads like prose. The support bot's truncated answers were grammatical right up to the character where they stopped, and so were the invoice extractions that lost their payment terms. That property is not an accident of these examples, and it is the reason the next chapter argues that reading the output is the wrong way to find out what happened to it.