Newline stop sequences and answer length

The two request-time bounds — a stop sequence and a token cap — are the obvious levers to reach for when answers are too long, and both of them are the wrong lever. Neither is visible to the model, so neither changes the answer the model is trying to write; both let it start a four-paragraph answer and then interrupt it somewhere. A blank-line stop sequence is the worked case here because its failure is the most complete: it deletes exactly the part of an answer that answers the question.

What a blank-line terminator does to a Markdown answer#

Set stop_sequences: ["\n\n"] and generation ends at the first blank line. The reasoning behind the change is that long answers are always multi-paragraph, so ending at the first paragraph break must produce a short answer. It does produce a short answer. Consider what a user asking how to reset a password receives:

You can reset your password from the account settings page.

And what the model was generating, before it was cut:

You can reset your password from the account settings page.

1. Open Settings and choose Security.
2. Click "Reset password" and confirm your email address.
3. Follow the link sent to that address; it expires after 15 minutes.

If the email does not arrive, check that your workspace allows external mail.

Everything the user needed was on the far side of the blank line. This is not a flaky failure that shows up under load; it is what the configuration does, every single time, for every answer with more than one paragraph. And the incidence is much higher than "sometimes", because Markdown puts a blank line before every list, every heading and every fenced code block. A sample of 500 logged answers from before such a change found that 34% contained a blank line, and of those, 71% had the actionable content — a numbered list, a command, a link — after the first one. The stop sequence did not shorten those answers. It deleted the part that answered the question and kept the part that set it up.

Two facts collide to produce that rate, and building the collision step by step explains why it is a third of traffic rather than an occasional accident. The first fact is what a blank line means in Markdown. Markdown separates block-level elements — paragraphs, lists, headings, fenced code — with a blank line, and the separator is structural rather than decorative: a numbered list written flush against the paragraph above it is not reliably parsed as a list. The second fact is what the model was asked for. A support answer written in Markdown that contains steps must therefore contain a blank line before those steps. Putting "\n\n" in stop_sequences registers, as a terminator, the most common structural separator in the format you asked the model to use. Generation ends at the first block boundary of every answer that has one, so the only block that ever survives is the first. The perverse part is what that does to good formatting: an answer that happens to be one dense paragraph passes through untouched, while the answer that carefully lifts its steps into a list is the one that loses them.

The two measured rates multiply, and it is worth saying why the multiplication is allowed rather than performing it silently. 34% is a share of all answers — those containing a blank line anywhere. 71% is not a share of all answers; it is a share of that subset, the answers with a blank line whose actionable content sits past the first one. A rate measured on a subset can be multiplied by the size of that subset to recover a share of the whole, which is what makes the two numbers combinable at all. Seven-tenths of about a third lands near a quarter, so roughly one user in four received an answer with its content removed and its preamble intact. That is not a tail case to be handled later. That is a quarter of the traffic being told nothing, fluently.

Moving the cut is not fixing the cut#

The remedy that gets proposed next is to remove the stop sequence and set max_tokens: 150 instead, on the grounds that this gives short answers without cutting at paragraph breaks. The first half of that is true and the second half is the problem. A cap does not cut at paragraph breaks; it cuts at token 150, wherever token 150 lands. Against the same 500-answer sample — median 74 output tokens, p95 210 — a cap of 150 truncates about 19% of traffic, and the cut lands mid-sentence or mid-word rather than at a paragraph boundary.

That 19% stops being a surprise once the distribution is read rather than glanced at. A p95 of 210 means, by definition, that one answer in twenty runs longer than 210 tokens. A median of 74 means half of them run longer than 74. A cap of 150 sits between those two landmarks, so the share of traffic it cuts has to fall between one in twenty and one in two, and 19% is where it lands for this particular shape of distribution. Reading a cap that way — as a position on a measured distribution rather than as a length somebody would like answers to have — is most of the sizing skill, and it takes about a minute once the distribution is on the screen. It is also the difference between the two numbers in this chapter: 512 sits past the largest answer anyone observed, which is what makes it a guard rail, while 150 sits inside the bulk of the traffic, which makes it a policy about answers dressed up as a limit.

That is arguably worse for the user, because a mid-word cut looks broken while a paragraph-boundary cut looks like a complete short answer. It is arguably better for the operator, for exactly the same reason: users report the visible failure and never notice the invisible one. What it definitely is not is a fix for the complaint, which was that the bot cuts off mid-explanation. Both configurations cut off mid-explanation. They differ only in where.

The proposal is tempting because it is directionally correct about the diagnosis. A cap is at least indifferent to document structure, where a stop sequence is actively sensitive to it. What the proposal misses is that neither bound is readable by the model, so swapping one invisible bound for another relocates the cut and changes nothing about the underlying behaviour — which is why tickets keep arriving with different wording after the change ships.

The fourth row is the configuration to ship. The instruction does the shortening because it is the only one of these mechanisms the model can read. The cap and any stop sequences stay as guard rails sized well above the p95, so they fire on pathological inputs and not on ordinary traffic. And the finish reason is checked and counted on every response, so that the day the guard rails start firing at 4% instead of 0.3% — after a model migration, a prompt change or a new class of question — it appears on a graph rather than in a support queue.

SYSTEM = (
    "You are a support assistant. Answer in at most three short paragraphs, "
    "using a numbered list when the answer is a sequence of steps. "
    "No preamble and no closing pleasantries."
)

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=512,          # guard rail, ~2.5x the p95
    system=SYSTEM,
    messages=[{"role": "user", "content": question}],
)

stop_metric.labels(reason=response.stop_reason).inc()
if response.stop_reason != "end_turn":
    raise IncompleteGeneration(response.stop_reason)

The instruction explicitly permits the numbered list, and that clause is doing more work than it looks. Length instructions written under pressure tend to say "be brief" and nothing else, which the model can satisfy either by writing a compact list or by writing a compact paragraph that omits the steps. Telling it which shape you want for which kind of question turns a length instruction into a formatting contract, and it separates short answers that work from short answers that are missing the steps.

Every number in that table came from a measurement rather than an estimate, and measuring before choosing is the step that gets skipped. It is cheap: log usage.output_tokens and the finish reason for a week, then look at the distribution before touching a parameter.

One condition attaches to that measurement, and skipping it produces a number that confirms whatever is already configured. A distribution logged while a tight bound is in force is a distribution of bounded answers, not of the answers the model wanted to write. Under max_tokens: 150 no logged response can exceed 150 tokens, so the p95 you compute is at most 150 and the sizing exercise recommends the cap you already have. Under a blank-line stop sequence the same thing happens in a different currency: every logged length is the length of a first paragraph. The sample that settled the support bot was drawn from before either change for exactly this reason. Where no such earlier sample exists, the way out is to lift the bound well clear of anything plausible for a day, log that traffic, and size from it — and the finish reason is what tells you whether the lifted bound was high enough, since a run with no truncations in it is a run whose tail you have finally seen. Almost every badly-sized cap in production was chosen by someone reasoning about how long they wanted answers to be, without ever looking at how long the answers actually were.

The task sets a floor the instruction cannot go below#

There is a limit to how far the instruction can take you, and a lot of length-control effort is spent trying to push past it. A question whose honest answer is five steps has an answer that is five steps long. You can compress the prose around them; you cannot compress the number of steps without removing steps, and a model asked hard enough to be brief will do exactly that — it will drop the caveat about the fifteen-minute link expiry, which is the part that generates the follow-up ticket.

When the honest answer is genuinely longer than the surface can display, the problem has moved out of the request parameters. The answers are the right length and the widget is the wrong shape, and the fixes live in the product: a one-line summary with the steps behind a disclosure, a two-call design where the first call classifies and the second answers at appropriate length, or a scrollable panel. Reaching for a cap at that point is choosing to truncate a correct answer rather than to display it, which is a decision that should at least be made deliberately.

The two-call design dissolves the tension rather than trading against it. Route the question first, cheaply, into a small set of shapes — a one-line factual answer, a short procedure, a diagnostic conversation — and give each shape its own instruction and its own cap sized from its own measured distribution. A cap of 128 is entirely reasonable for the one-line shape and catastrophic for the procedure shape, and a single global cap has to be wrong for one of them. Most support-bot length complaints are really complaints that one shape's answers are being written in another shape's register.

Where a newline terminator is the right choice#

None of this makes stop_sequences: ["\n\n"] a bad parameter value. It is the correct configuration whenever a newline genuinely is the terminator of the thing you are generating, and there are several such cases.

A classifier that must return a single label on one line has a real structural boundary at the newline, and stopping there is both cheap and exact — it saves the tokens the model would otherwise spend explaining its choice, and the finish reason tells you the boundary fired as designed. Completion-style tasks with the same shape, such as generating the next line of a config file or a single commit-message subject, are the same case. So are the chat-turn terminators the parameter was originally built for, when you are formatting a conversation into a single string yourself.

One question separates the good uses from the bad one: if the model kept generating past your stop sequence, would the extra text be part of the answer or noise? For a single-label classifier the extra text is the model explaining itself, which you do not want, so cutting is a saving. For a support answer the extra text is the numbered list, which is the answer, so cutting is a loss. The parameter behaves identically in both cases; only the relationship between your delimiter and your content differs. Applying a terminator to text that has internal structure of the same shape as the terminator is the whole of the bug.

Answer length, then, is a property of the answer, and the only mechanism that can act on it is the one the model reads. Stops and caps are bounds on the worst case: sized from a measured distribution, placed well above the tail, and paired with a check so that a bound firing is an event somebody learns about rather than a shorter answer nobody questions. Any configuration that inverts that — a bound doing the shaping, an instruction doing the bounding — produces the two failures in this chapter, one of which looks broken and one of which looks fine.

Three questions attach to every request you write from here, and each takes about a minute. What is allowed to end this generation, and if it is a stop sequence, what structural boundary does it correspond to? What happens when each bound fires — which exception, at which boundary, followed by which deliberate policy? And who finds out, which means a counter labelled by finish reason emitted on every response. Go and answer those three for one request in your own codebase, the one whose output is written somewhere durable. Most of the time the third answer is nobody, and that is the one worth fixing first.