Deliberation and format precision

A field-extraction service is a good example of a task that looks like it should benefit from a reasoning model and does not. Twelve fields come out of each invoice — supplier name, invoice number, issue date, due date, currency, subtotal, tax, total, PO reference, payment terms, bank account and line-item count — against a schema that states each one's type. Every field has a single correct value that is present in the document or is not. Nothing has to be weighed against anything else.

This chapter is about telling that kind of task apart from the kind where deliberation pays, and about what to do with the errors once you know. The method is an error taxonomy: pull a month of production failures, classify them by what went wrong, and mark each class as deliberation-shaped or not. The share that is deliberation-shaped bounds the entire intervention, and the rest points at fixes that are cheaper, faster and deterministic.

Where the errors actually live#

Errors are not a homogeneous quantity a better model reduces proportionally. On an extraction service running at 94.1% field accuracy, the missing 5.9% breaks down like this:

Thirteen percent of the errors sit in a category where thinking longer plausibly helps, and that thirteen percent is 0.77 points of the 5.9. Even if extended thinking eliminated fabricated values and arithmetic slips entirely — which it does not — the ceiling on the intervention is 94.9%. The other 87% of errors are the model producing a value it recognised correctly and rendering it in a shape the schema does not accept, or drawing a boundary in the wrong place. Those are decisions made at the moment of emitting a token, and no amount of prior deliberation changes the token that comes out; the model was never uncertain about what the date was.

That last clause is the whole chapter, and it deserves taking apart rather than accepting. Deliberation acts on the context. The trace is text, it is appended to the sequence, and the answer is scored from a sequence that now contains it. What the trace buys is information — a step worked out, a case eliminated, a constraint checked against the others — and it buys nothing whatever when the information was never missing. The date is printed on the invoice. Reading it required no inference and produced no uncertainty for a trace to resolve. What went wrong happened one layer later, at the instant a token was sampled to render a value that had already been read correctly, and nothing in the context arrives at that instant as a rule about which shape to use.

The claim is checkable rather than something to take on faith, and checking it is cheap. Take the failing items and set the model's output beside the field it was extracting. Where the wrong value is a differently shaped version of the right one — the same day and month in the other order, the same amount with its separators swapped — the value was read correctly and rendered wrongly. Where the wrong value is a different value, or a value lifted from another part of the page, something was decided wrongly and deliberation has something to work on. The table above is that check applied a few hundred times, and the ratio it returns is what bounds every model change you could make afterwards.

This is why reasoning variants frequently match or trail instruct variants on well-specified extraction. Instruction tuning optimises heavily for following a stated output contract, and a reasoning variant of the same base is tuned toward a different objective.

The 12% ambiguous-source class deserves a moment on its own, because it puts a ceiling on the whole request. Those documents are low-quality scans where the tax line is genuinely unreadable, or invoices where the wanted field does not exist. No model reads what is not there, so 0.7 points of the current error rate are not addressable by any change to the model, the prompt or the effort setting; they are addressable by rejecting the scan and asking for a better one. A target of 98% is therefore a target of roughly 98% on the 88% of documents that are legible, and stating it that way changes what gets built. Every accuracy conversation about extraction benefits from separating the errors the system makes from the errors the input guarantees.

What does move format precision is cheaper and deterministic. Constrained decoding against a JSON schema — where the decoder cannot emit a token that violates the grammar — eliminates the entire class of structurally invalid output. A "format": "date" constraint with an explicit statement that dates are ISO-8601 removes the 03/04/2025 failure without a single extra token of thinking. Two worked examples in the prompt showing a European decimal comma and a right-to-left supplier block do more for the boundary class than any effort setting. Post-processing that normalises currency and trims whitespace before comparison is free. Those three changes on the original instruct model took the same task to 97.2% at 740 milliseconds and $1,010 a month.

Why those work where the expensive change does not comes down to where each of them acts. Deliberation changes what the model reads before it chooses. Constrained decoding changes what it is permitted to choose. At every position the model produces a score for each token in its vocabulary, and a decoder held to a grammar puts every token that would break the grammar out of reach before sampling happens, so an invalid token is not merely unlikely — it is unavailable. A format constraint is therefore not a stronger version of an instruction. It is a different kind of thing, acting after all the scoring is finished, which is why it is deterministic and why it costs no tokens at all.

Thinking is spent before the first visible token#

The latency change from a reasoning migration is not a proportional slowdown, and describing it as "about eight times slower" hides the part that breaks a service. Extended thinking is generated as tokens, sequentially, before the model begins the answer. From the client's perspective the connection is open and silent for the entire thinking pass.

Time to first token — the gap between dispatching the request and the first character of the answer arriving — went from 210 milliseconds to 5.4 seconds, a 26× increase, while the streaming phase that produces the answer you keep is unchanged at roughly 490 milliseconds either way. Total time rose 8.7×, and the tail rose further: p95 went from 1.1 seconds to 14.2, because thinking length has a long right tail that visible output does not.

For a batch job that difference is a scheduling problem. For anything interactive it is a design change, since every technique that hides latency behind a progressive interface depends on tokens arriving early. A form that fills fields as they stream can show the supplier name at 250 milliseconds on the instruct model and shows nothing at all for five and a half seconds on the reasoning one. The same applies to any downstream consumer that parses incrementally: a streaming JSON parser emitting field events as they close cannot begin until the thinking pass ends, so a stage that was overlapping its own work with the model's is now serialised behind it.

There is also an interaction between thinking and structured output that catches teams during exactly this kind of migration. Providers differ on whether strict schema enforcement — the mode where the API guarantees the response validates against a schema you supply — can be combined with extended thinking, and on what the response looks like when it is: the thinking content arrives as its own block or channel ahead of the constrained answer, and code assuming the first content element of the response is the JSON payload will now find prose there. Tool-use flows have the same shape, since a thinking block precedes the tool call. Neither is difficult to handle once you know, and both produce parse errors during the migration that get logged as model quality problems if nobody checks the response shape.

The cost arithmetic is the least surprising part. At $3.00 per million input and $15.00 per million output, an instruct call on this workload is $0.0057 of input and $0.0051 of output. The reasoning call has identical input and 2,740 output tokens once thinking is included, which is $0.0411, so per-document cost goes from $0.0108 to $0.0468 and a monthly bill across 90,000 documents from $972 to $4,212. Nothing about that is hidden; it follows directly from the accounting rule that reasoning tokens are output tokens.

That predictability is the useful property of the rule. Once it is believed, the cost of a migration can be worked out before the migration, from two numbers you can collect in an afternoon: take the mean thinking spend at the effort you intend to use, add the visible output, multiply by the output rate. Nothing about the model's quality enters that calculation, which means the cost side of the decision can be settled before anybody starts arguing about the accuracy side. That is the reverse of how these conversations usually run, and it is what made the prediction in the scenario below possible to write down in advance.

# The instrumented run that produced the numbers above.
for doc in FROZEN_INVOICES:            # 400 documents, frozen
    for label, cfg in [
        ("instruct", {"model": "claude-haiku-4-5"}),
        ("reasoning-med", {"model": "claude-sonnet-4-5",
                           "thinking": {"type": "enabled", "budget_tokens": 4000}}),
    ]:
        rec = run(doc, config=cfg, effort=label)
        rec.score = field_accuracy(EXPECTED[doc.id], parse(rec.text))
        records.append(rec)

# report TTFT separately; the mean of total_ms hides the whole effect
p95_ttft = quantile([r.ttft_ms for r in records if r.effort == "reasoning-med"], 0.95)

An accuracy jump is a reason to audit the run#

If this experiment comes back showing the reasoning variant reached 98.4% on the same task set, the correct first response is to distrust the experiment. Extraction from a well-specified schema is exactly the case where a large deliberation gain is implausible, so a large gain is more likely to be a defect in the comparison than a property of the model.

Distrusting a good result feels backwards, and it is the right instinct here for a reason worth stating plainly. The taxonomy has already told you the size of the available prize: a ceiling under a point. A result several times larger than the ceiling is not evidence that the ceiling was wrong. It is evidence that the two arms of the experiment differed in something besides the thing you changed, because the ceiling was computed from the errors themselves rather than from an expectation about models. The checks below are what to run before anyone gets a chance to celebrate a number that a second, unlogged edit produced.

The reversal that should worry you most is latency, because its absence proves the experiment did not run. Accuracy and cost can both stay flat for legitimate reasons; time to first token cannot stay flat if thousands of thinking tokens were generated, since those tokens take real decode time. A run showing +4 points of accuracy, +300% cost and +8% time to first token is internally inconsistent, and the inconsistency is worth more attention than the accuracy number.

There is a version of this task where the reasoning variant is the right call, and it is worth naming so the lesson does not overgeneralise. If the twelve fields were not stated in a schema — if the task were "extract whatever payment obligations this contract creates", where the number of outputs is unknown and each one requires reading several clauses together — the bottleneck moves from rendering to comprehension, and deliberation is what comprehension needs. The distinguishing question is whether a competent human doing the task would spend their time reading and deciding, or transcribing. Transcription tasks want a fast model and a tight contract. Deciding tasks are what the previous chapter's hard band was made of.

That test has a mechanical form as well, which is more reliable than intuition when the task is unfamiliar. Ask how many of the output's fields are determined by a single span of the input, and how many need spans combined that sit far apart, or need a judgement about something that is absent. The first kind is transcription however complicated the document looks, and the second kind is deciding however short it is. Twelve fields each printed verbatim on one line is the first kind twelve times over, which is why the page count of an invoice never made this particular task hard.

Before buying deliberation, then, find out what the errors are made of. A month of production failures classified into five or six classes takes an afternoon and produces a number — the deliberation-shaped share — that bounds every model change you could make. When that share is 13% of a 5.9% error rate, the ceiling is under a point, and the honest options are a tighter output contract, better examples, deterministic normalisation, and a conversation about the inputs that no model can read. Those cost four percent more; the migration costs four hundred percent more and lands inside the noise band of the baseline.

The failures here were all visible: a slow call that completed, a wrong field, a bill. The harder class arrives when the request does not complete at all, and a short, useless answer against a client deadline turns out to have at least three distinct causes that a log line reading timeout cannot distinguish between.