Shortening an answer is not the same as truncating one

The change was one line in a config file: max_tokens: 800, down from unset. Spend on the feature fell 20% the following week, the evaluation suite's aggregate score moved from 4.42 to 4.41 out of 5, and the work was written up as a clean win. Twenty-three days later an account manager forwarded a renewal-risk note from the second-largest customer, whose analysts had stopped using the policy explanation flow because, in their words, it stops mid-sentence about a third of the time.

Both facts were true simultaneously. The median response was unaffected, the evaluation set was dominated by short-answer tasks where nothing had changed, and the requests that broke were concentrated in a small slice of traffic that happened to be the slice enterprise accounts used almost exclusively. The saving was real and so was the damage, and the config line contained no way to tell them apart.

The term the cache cannot reach#

The feature bills $40,400 a month across 1.7 million requests, with prompt caching already running at a genuine 88% hit rate on the 6,000-token prefix. Its per-request decomposition is not the one from chapter 1, because caching has already landed:

Output was 36% of the uncached request and is 57% of what remains, which is the general pattern: caching is the lever that makes output the dominant term. Output tokens carry two disadvantages that compound. They are priced at 5× input on this model, and there is no cache, no batching trick and no prefix engineering that reduces them, because each one is computed fresh. The only way the number falls is if the model writes less.

Before touching any of it, the distribution has to be measured rather than the average, and the ledger from chapter 2 already carries the column. The mean of 900 tokens is the number that appears in the cost model and it is the wrong statistic for every decision in this chapter, because output lengths are right-skewed: the median is 640, so more than half of all responses are below the average and the average is being pulled up by a tail. Any lever expressed as a threshold — a cap, a warning, a rail — has to be positioned against percentiles, and any lever expressed as a shift has to be evaluated on the mean, because that is what the bill integrates. Producing a percentile table from the ledger takes one query, and it is the difference between a config value chosen from the distribution and a config value chosen because it is a round number.

That is also why output length gets less attention than it deserves. Caching, batching and tiering are things a provider sells you; writing shorter answers is a product decision that no vendor post is going to headline, and it competes directly with the instinct that a more thorough answer is a better one.

Two mechanisms wearing the same name#

"Put a limit on output" describes two categorically different interventions. One changes what the model decides to write. The other stops the decoder mid-stream when a counter runs out. Measured on the same traffic, they look comparable in dollars.

The last two rows are the comparison that matters. Instructed brevity with a schema saves $9,690 a month; the hard cap saves $8,287. The cap is slightly worse on money and it truncates two responses in five.

Structured output does most of its work by deleting text that was never the answer. An unconstrained response to "does the refund policy cover shipping" tends to restate the question, hedge for two sentences about the importance of checking with a representative, give the answer, then summarise what it just said. Asking for a schema with answer, citations and caveats removes the restatement and the closing summary by construction, and it moves the hedging into a field the interface can render or suppress independently. It carries its own failure in the opposite direction: a schema with a required caveats array invites the model to invent caveats on questions that have none, and a required reasoning field will be filled with several hundred tokens whether or not the question needed any. Fields cost tokens, so a schema is a budget and every optional field that is usually populated should be justified the same way a column in a hot table would be.

There is a version of the same lever that lives in the product rather than the prompt, and it is frequently the largest one available. This feature's interface renders the first three bullet points of the answer in a card and puts the rest behind a "show full response" control that 12% of users click. The model has been generating 900 tokens so that roughly 200 of them can be displayed, and asking it for the card's content directly — with the long form generated on demand when someone expands it — cuts average output far more than any brevity instruction while removing nothing anyone was reading. Checking what the interface actually shows before optimising what the model writes is a five-minute exercise that occasionally makes the rest of the chapter unnecessary.

The truncation rate follows from the distribution rather than from anything about the prompt. This feature's output lengths are roughly lognormal — a right-skewed shape in which most responses cluster below the mean and a thin tail of very long ones stretches far above it — with a median of 640 tokens and a mean of 900, which puts p90 at about 1,845, p95 at 2,490 and p99 at 4,370. A cap at 800 sits just above the median, so 39.3% of responses hit it, and the expected output falls to 575 tokens because everything past the cap is discarded rather than never generated. The saving is real money for text the user was going to see and now does not.

Truncated responses also do not stay truncated. In this product roughly 30% of users who received a cut-off answer re-asked the question, which added 200,600 requests a month at $0.0189 each, or $3,793 — 46% of the saving handed straight back, and this time as full-price traffic that also raises the cache write rate. That term appears nowhere in the config change's business case and is exactly the kind of uncounted call chapter 2 was about.

Where the damage lands#

A cap's regressions are invisible in a blended score because they are concentrated. Sorting this feature's responses by length and looking at what produced the long ones: of responses over 800 tokens, 61% came from two task types — policy explanation and multi-document comparison — which together are 9% of traffic. Those two task types are 100% of what the enterprise tier uses the feature for. Every other slice, which is 91% of requests and most of the evaluation set, was untouched, so an aggregate score moved 0.01 and the product broke for the customers paying the most.

That asymmetry is structural rather than bad luck. Long answers correlate with hard questions, hard questions correlate with users doing serious work, and serious work correlates with revenue. A lever whose damage is proportional to answer length is a lever whose damage is aimed at your best traffic.

None of which makes max_tokens useless. Set well above p99 it is a runaway rail: at 6,000 it truncates 0.34% of responses, costs essentially nothing in quality, and stops the pathological case where a model falls into a repetition loop and generates until something else stops it. The distinction is between a limit that shapes the answer and a limit that catches a failure, and the same parameter serves both roles depending on where you put it relative to the distribution you measured.

resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=6000,                       # rail above p99, not a shaping lever
    system=[{"type": "text", "text": SYSTEM_PROMPT,
             "cache_control": {"type": "ephemeral"}}],
    messages=[{"role": "user", "content": user_turn}],
    tools=[ANSWER_SCHEMA],                 # answer, citations[], caveats[]
    tool_choice={"type": "tool", "name": "answer"},
)
assert resp.stop_reason != "max_tokens", "rail hit — investigate, do not raise silently"

Asserting on stop_reason is the part teams skip. Without it, a cap that is doing damage produces no signal at all: the response is well-formed JSON up to the truncation point in the best case and malformed in the worst, and either way the only complaint arrives through an account manager.

The compression that costs no model call#

Sitting next to output length is the 25% of the request that is retrieved context, and a share of it is text the model has already read further up the same prompt. Three mechanisms put it there. The chunker uses 512-token chunks with a 64-token overlap, so adjacent retrieved chunks share text by construction. Documents get republished with small edits, so a passage and its near-identical successor both score well against the same query. And every chunk carries a document header — title, owner, last-updated, classification — which is 40 tokens repeated across eight chunks.

Measuring it on this feature: of 2,000 average context tokens, 570 were duplicated content, or 28.5%. The header boilerplate alone accounted for 320 of those tokens — 40 tokens of title, owner, last-updated and classification, repeated across eight chunks — which is 16% of the entire retrieved context spent restating metadata the model needed once. Chunk overlap contributed another 180 tokens on average, and genuine near-duplicate passages across documents the remaining 70. Removing all of it costs no model call and no latency worth measuring.

import re, hashlib

def dedupe_chunks(chunks: list[str], min_sentence_chars: int = 40) -> list[str]:
    seen: set[str] = set()
    out = []
    for chunk in chunks:
        kept = []
        for sent in re.split(r"(?<=[.!?])\s+", chunk):
            norm = re.sub(r"\W+", " ", sent.lower()).strip()
            if len(norm) < min_sentence_chars:
                kept.append(sent)          # too short to fingerprint safely
                continue
            key = hashlib.blake2b(norm.encode(), digest_size=16).hexdigest()
            if key not in seen:
                seen.add(key)
                kept.append(sent)
        if kept:
            out.append(" ".join(kept))
    return out

That runs in about 1.2 ms on eight chunks and cuts context from 2,000 to 1,430 tokens, taking the variable term from $10,200 to $7,293 a month. On the same 400-question evaluation used in the previous chapter, exact-match accuracy moved from 91.2% to 91.4%, which at n = 400 is inside a ±1.4-point interval and should be read as unchanged rather than as an improvement.

Deduplication is not prompt compression and does not carry compression's risk profile. Nothing is summarised, no model decides what is important, and the only text removed is text that appears verbatim elsewhere in the same prompt. The exposure it does carry is that near-duplicates with a meaningful difference — a policy paragraph and its amended version differing in one number — hash differently at sentence granularity and both survive, which is the safe direction to fail. Aggressive normalisation that collapses them would be the compression failure mode arriving through the back door.

Two levers are now on the table for this feature, each worth several thousand dollars a month, alongside a proposal to move the nightly reporting workflow onto a batch endpoint and a proposal to route routine traffic to a cheaper tier. They cannot all be evaluated, built and shipped at once, and the order they go in is not the order of their expected savings.