Reasoning effort and thinking budgets

Reasoning effort on one provider and a thinking budget on another are the same control seen from two angles: a way of saying how much intermediate work a model may do before it answers. The setting looks like a quality knob, which is why it tends to be raised globally in a shared client library, and it behaves like a cost multiplier applied uniformly to traffic that responds to it very unevenly.

This chapter is about sweeping that control properly. The experiment is the per-call instrumentation from the previous chapter — reasoning tokens, visible output tokens, input tokens by cache status, wall-clock time, time to first visible token and a graded score — run three times over one frozen set, changing only the effort setting. What makes the result readable rather than misleading is splitting the set by difficulty first, so the three response curves do not average into one.

Banding the task set#

The set used below is 180 production requests labelled into three bands during collection: routine classification and lookup, multi-step planning where the answer depends on combining several facts, and genuinely hard items — ambiguous bug reports, conflicting policy documents, requests where the correct answer is that the request cannot be satisfied.

Banding is the part people skip, and without it the sweep produces one number that means nothing. If you have human labels or an escalation flag in your logs, use those. If you do not, a workable proxy is to run the cheapest configuration once, take the items it got wrong, and treat those as the hard band; that is circular enough to be worth stating out loud, since it defines difficulty relative to one model rather than in the abstract, but it separates the traffic where the dial can possibly help from the traffic where there is nothing left to win. Three bands is usually enough, and the boundaries only need to be stable across runs, not principled.

Read down the routine rows first. Accuracy is flat from low to medium and drops slightly at high, while cost rises 6.6× and tail latency 4.1×. One item that low and medium both answered correctly came back wrong at high, which on 108 items is a single flipped answer and not evidence of anything by itself; the honest reading of the routine band is that the dial bought nothing and charged for it. That is the ordinary case, because these items are not hard for the model. It knew the answer at low effort, and the extra 1,130 tokens were spent restating a decision that had already been made.

Why a band like that cannot move is worth one step further down. The answer is whichever token scores highest at the moment the answer begins, and on a routine item that token was already the right one. More deliberation cannot make an already top-scoring token score higher in any way that changes the outcome. The only thing it can do is change the context enough to move the ordering. Most of the time it does not, and the answer comes back identical. Occasionally it does, and the answer flips — which is what the single regression in that band is, and the same effect the inverse-scaling section below describes, at a scale small enough to pass for noise.

The hard band is a different function entirely. Accuracy more than doubles from low to high, seven of eighteen items flip from wrong to right, and thinking spend goes from 610 tokens to 7,050 — an 11.6× increase in tokens for a 44-point gain in accuracy. At $74.60 per thousand calls this band is over twelve times more expensive than the routine band at low effort, and if those eighteen items are eighteen escalations that would otherwise reach a human, the comparison is not against the routine band's cost but against forty minutes of an engineer's time.

The multi-step band shows why the decision is per workload rather than global. Most of the gain arrives between low and medium, and the step from medium to high buys 1.9 points for 2.5× the tokens and 2.2× the tail latency. Whether that is a good trade depends entirely on what a wrong answer costs, which is a question about your product rather than about the model.

The latency column is a p95 rather than a mean for a reason that becomes structural two chapters from here. Thinking length has a long right tail, so the mean moves gently while the tail moves violently: on the hard band at high effort the mean total time was 11.8 seconds against a p95 of 26.4 and a slowest observed call of 41 seconds. A dashboard reporting mean latency would have shown the effort change as roughly a doubling, which sounds survivable, while the requests that actually breach a timeout live entirely in the part of the distribution that dashboard does not show.

A mean and a tail move differently here for a structural reason rather than a statistical one. Raising the dial does not shift every request's thinking spend by the same amount, because most requests never approach the ceiling at all — they stop when the model stops, and the model stops where it stopped before. What a raised ceiling changes is what is now permitted to happen to the requests that were already running long, and those are allowed to run longer. So the dial moves the right-hand end of the distribution far more than it moves the middle, and the right-hand end is exactly where a timeout lives and where a mean is least sensitive.

Now put the three bands back together at production weights, where routine traffic is 92% of volume. Aggregate accuracy across the whole mix moves from 93.0% at low to 93.9% at high, which in a review reads as "slightly better", while cost per thousand calls moves from $2.72 to $18.65. The aggregate hides the 44-point gain on the band that needed it and disguises a sevenfold cost increase as a rounding difference, which is why the per-item scores from the previous chapter matter more than any run-level summary.

The arithmetic that makes both of them disappear is worth doing once, because it is what a weighted average is for and what it is bad at. Aggregate accuracy is each band's accuracy weighted by that band's share of volume, so a band carrying 92% of the traffic carries almost all of the answer. The routine band's accuracy does not move, and it is the overwhelming majority of the mix. The hard band's accuracy more than doubles, and it is a sliver of it. Multiply each by its weight and add, and the two facts collapse into a change of under a point. The cost figure is weighted the same way and does move sevenfold — but a reader who has already accepted the accuracy number as "slightly better" reads that rise as the price of a small improvement rather than as the price of no improvement at all. That is what understates the cost. Not the number, but what the number appears to have bought.

Nominal settings are hints, and they are not comparable#

The three major providers expose this control differently, and the shape of the control affects how you sweep it.

# Anthropic: an explicit token budget the model may spend, min 1024,
# and it must fit inside max_tokens alongside the visible answer.
client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=8000,
    thinking={"type": "enabled", "budget_tokens": 6000},
    messages=[{"role": "user", "content": prompt}],
)

# OpenAI: discrete named levels, no token figure exposed.
client.responses.create(
    model="gpt-5",
    reasoning={"effort": "medium"},   # minimal | low | medium | high
    input=prompt,
)

# Google: a token budget, with -1 meaning "decide per request"
# and 0 meaning off on models that permit switching it off.
client.models.generate_content(
    model="gemini-2.5-flash",
    contents=prompt,
    config={"thinking_config": {"thinking_budget": 6000}},
)

A token budget is a ceiling rather than a target. Asking for 6,000 does not cause the model to spend 6,000; on the routine band above, a 6,000-token budget produced a mean of 1,340 because the model stopped when it had finished. That is the property which makes budgets tolerable as a default on mixed traffic — you are setting the worst case, not the typical case — and it is also the property that makes budget numbers a poor way to control cost, since actual spend is decided per request by something you do not observe.

Two different mechanisms can end a thinking pass, and knowing which one usually fires accounts for both the ceiling behaviour and the variance. The server counts generated tokens and forces a stop when the budget is reached. The model ends the pass by sampling a token that ends it, which becomes likely once the trace has covered what it was going to cover. On an easy item the second always fires first, which is why asking for six thousand tokens produced a mean well under a quarter of that. On a hard item the first can fire, and when it does the model was cut off rather than finished — a distinction two chapters from here is built around. For a sweep the consequence is that the number you set bounds a worst case and the number you observe is a sampled quantity, so the two are not the same kind of thing and cannot be read off against each other.

Named effort levels are looser still. There is no published mapping from "medium" to a token count, the mapping changes between model versions, and it is not comparable across providers: one vendor's "low" may spend more than another vendor's "medium" on the same prompt. Any table that lines up low/medium/high between two providers and compares accuracy row by row is comparing two arbitrary labels. The only comparison that means anything is against measured spend, so record the reported reasoning tokens for every configuration and plot accuracy against tokens rather than against the setting name. That also makes the nonlinearity legible: on the hard band, low to medium cost 1,630 extra tokens for 22 points, and medium to high cost 4,810 extra tokens for another 22 points, so the second half of the gain was roughly three times the price of the first.

That instruction — plot against tokens rather than against the setting's name — rests on tokens being the thing actually bought. A name is a label the provider maps to an internal configuration, and that mapping is neither published nor stable across versions, so two runs at "medium" six months apart are two unknown quantities wearing the same label. A token count is the same unit on every provider, in every version, and it arrives in a response you already have. Plotting accuracy against it turns three incomparable configurations into points on one curve, which is what makes the knee of that curve something you can find rather than something you argue about.

GRID = [
    ("low",    {"thinking": {"type": "disabled"}}),
    ("t2k",    {"thinking": {"type": "enabled", "budget_tokens": 2000}}),
    ("t6k",    {"thinking": {"type": "enabled", "budget_tokens": 6000}}),
    ("t16k",   {"thinking": {"type": "enabled", "budget_tokens": 16000}}),
]

for label, cfg in GRID:
    for task in FROZEN_SET:            # same 180 items, same order
        rec = run(task, config=cfg, effort=label)
        rec.score = grade(task, rec)
        records.append(rec)

# report per band, never only in aggregate
for band in ("routine", "multi_step", "hard"):
    for label, _ in GRID:
        rs = [r for r in records if r.effort == label and band_of(r) == band]
        print(band, label,
              mean(r.score for r in rs),
              mean(r.reasoning_tokens for r in rs),
              quantile([r.total_ms for r in rs], 0.95))

Run the grid at least three times per configuration if you can afford it. Reasoning spend is stochastic in a way visible output length is not: the same item at the same budget can consume 800 tokens on one call and 3,100 on the next, because whether the model decides it has finished checking is itself sampled. On the hard band above, the standard deviation of thinking tokens across repeats was around 40% of the mean, which is large enough that a single-run comparison between two adjacent settings tells you very little.

More thinking is not monotonically better#

The intuition that quality rises with effort and the only question is price is wrong in a specific and documented way. Inverse Scaling in Test-Time Compute constructs tasks where longer reasoning traces make accuracy worse — simple counting problems with distractors, where extended deliberation draws the model toward the irrelevant detail; regression tasks where more thinking causes the model to latch onto spurious features in the examples; and constraint-satisfaction problems where the model reasons itself out of a correct early answer. The effect is not exotic. The single routine-band regression in the table above is the same shape at a much smaller scale: an item the model had right, thought about further, and talked itself out of.

The mechanism behind those results is the one that made deliberation useful in the first place, running the other way. Intermediate text becomes part of the context every later token is scored from, which is precisely why writing the working out helps when the working is relevant. When the working dwells on a distractor, the distractor is in that context too, and it is more present there than it was in the original question. Nothing filters a trace on its way into the context, because the trace is the context. So "more thinking" is not a direction of improvement at all. It is a change of input.

That has a practical consequence for how a sweep is read. A configuration improving the mean while flipping four previously-correct items to incorrect is not obviously better than one that improves nothing, particularly if those four sit in a category where a wrong answer is expensive. Compute the per-item transition counts — right-to-wrong and wrong-to-right — and look at them separately, because a net gain of +2 built from six wins and four losses is a different object from a net gain of +2 built from two wins and no losses.

The dial also does not address the failure people most often reach for it to fix. When a model gets something wrong because the necessary information was not in the prompt, because a tool it needed returned an error, or because the instructions genuinely contradict each other, more thinking produces a longer and more confident wrong answer. Deliberation operates on what is in the context; it does not call anything you did not give it, and it cannot resolve an ambiguity that has no resolution. It does not retrieve either, since the step that finds the missing document and places its text in the prompt runs before the model is called at all — which is why raising effort as a first response to a quality complaint, before anyone has read the failing cases, is the most common way to spend six times the budget on a retrieval bug.

The dial is therefore a per-workload decision with a per-band answer, and the sweep that produces it is three runs over one frozen set with the results reported separately by band. Routine traffic wants the setting low or off, planning wants the knee of its own curve, and the escalation path wants whatever the hard band's numbers justify against the cost of a human. A single global default is guaranteed to be wrong for at least two of the three, and the review that approves it will see an aggregate in which neither error is visible.

What a sweep cannot tell you is why a particular band refuses to move. There is one common case where a task looks hard, is expensive to get right, and gains nothing from any setting on the dial, because what is failing is not the thinking at all.