Chain-of-thought and scratchpads

A prompt is an assembly: instructions somebody wrote, a schema, a few examples, and the input the request is actually about. Topic 4.1 established that all of it reaches the model as one sequence and works on the model by the only route available, which is being in that sequence. A sentence asking for intermediate steps is a part of the assembly like any other part. It changes what comes back. It does not change the model into a different program, give it a new faculty, or open a channel that was not already open.

That is the boundary this whole subject sits on, and it has a concrete consequence. Steps that a prompt explicitly asks for come back as generated text in the response body, occupying the same output the answer occupies, timed the same way, priced the same way, and arriving in front of the same parser. A reasoning model's internal deliberation is a different mechanism entirely: the provider runs it, decides what of it to return, and reports it under its own accounting. Topic 1.15 owns that mechanism. Everything in this chapter is about the first thing — text you asked for and were given.

Why anybody asks for that text is a separate question from what it is, and it is worth stating in terms of what goes wrong without it. A model that emits a final decision and nothing else is a program that reports an exit code. When the code is right, nobody minds. When it is wrong, there is nowhere to look: no record of which condition was evaluated, no place where the wrong branch was taken, nothing between the input and the verdict. An engineer who meets that in ordinary software reaches for a debugger, or failing that scatters print statements through the function, and reads back a trail of the intermediate values until the line where the result stopped being the expected one becomes obvious. Asking a model to write out its checks looks like exactly that move, and the resemblance is why the technique is easy to reach for.

Yet the resemblance ends at a specific point, and the point matters more than the analogy does. A print statement is instrumentation: it reads a variable and emits what the variable holds, because reading it is what the statement does. The record is produced by a mechanism separate from the computation it records, which is why removing every print statement leaves the program computing the same thing. Generated steps are not produced that way. They come out of the same next-token process that produces the answer, one token at a time, each one selected because it fits the text so far — and the answer is then produced after them, conditioned on them, as a continuation of a passage that now contains them. Delete the steps and the answer can change, which tells you they were an input to the result rather than an observation of it. In the other direction, a step saying checked the signed-image condition is a sentence that fits; nothing anywhere in the request path requires it to correspond to anything, so a model can write down a check it never performed and a number it never computed, and the sentence will read exactly like one that is backed. Readable steps are therefore worth having and are not proof of how the answer was formed.

The parts of the technique#

Five things are in play here. Chain-of-thought prompting is the instruction that asks a model to write its intermediate steps before it commits to an answer. The scratchpad is where those steps go — the part of the response that holds working rather than verdict. Reasoning trace leakage is that working arriving in the output when the caller wanted only the verdict, which is the same text seen from the other side. A single-hop task is one the model can answer in one move, input to verdict, with nothing to carry between steps. A multi-step task is one where each step depends on the result of the last, so the order the steps are taken in is part of whether the answer comes out right.

All five come into play at the moment the gap opened by the print-statement comparison — a verdict with nothing between the input and the verdict — gets closed, and it is closed, if it is closed at all, by adding a sentence to a prompt. What that sentence changes is visible in the response and nowhere else, so the smallest way to see it is to send one task twice with a single sentence different.

Chain-of-thought prompting#

To see the gap in action, consider a release-approval endpoint that decides whether a deployment may proceed. It runs claude-haiku-4-5 from Anthropic at temperature 0 with a 512-token output limit, and each call carries a deployment policy and one deployment request. The policy has an emergency exception: a production deploy normally needs a second engineer's approval, and an emergency deploy may go ahead without one as long as the image is signed. A request arrives with the emergency flag set, the image signed, and no second approval — a case the exception was written for — and the endpoint denies it.

The fix is to stop asking for the verdict on its own. Instead of asking which decision applies, ask the model to say which policy conditions it checked and what the request said about each of them, in the order it checked them, and to give the decision after that. Nothing else about the request changes.

You are a release gate for the payments platform.

Deployment policy:
1. A deploy to `prod` requires a signed image.
2. A deploy to `prod` requires approval from a second engineer.
3. An emergency deploy to `prod` may proceed without the second
   approval when the image is signed.
4. A deploy to `staging` proceeds.

Deployment request:
target_environment: prod
image_signed: true
second_approval: false
emergency: true

Reply with ALLOW or DENY and nothing else.

DENY

The endpoint has a rule that fits the request and returns the answer the rule was written to prevent. Now the same setup, with the final instruction replaced by one asking for the checks. Every other line is the line above.

You are a release gate for the payments platform.

Deployment policy:
1. A deploy to `prod` requires a signed image.
2. A deploy to `prod` requires approval from a second engineer.
3. An emergency deploy to `prod` may proceed without the second
   approval when the image is signed.
4. A deploy to `staging` proceeds.

Deployment request:
target_environment: prod
image_signed: true
second_approval: false
emergency: true

List each policy rule you checked and what the request says about
it, in the order you checked them. Then reply with ALLOW or DENY
on its own final line.

Rule 1 — signed image required for prod. image_signed is true, satisfied.
Rule 2 — second approval required for prod. second_approval is false, not
satisfied on its own.
Rule 3 — emergency deploy may skip the second approval when the image is
signed. emergency is true and image_signed is true, so rule 3 applies and
rule 2 is waived.
Rule 4 — target is prod, not staging, so this does not apply.
ALLOW

Call the condition with no added instruction standard prompting and the condition with the intermediate-step instruction chain-of-thought prompting. The names attach to the prompts, not to their order on the page, and they hold for the rest of the chapter.

Both conditions still have to be checked against the case the exception does not cover, or the instruction has bought a decision that always says yes. Hold standard prompting and chain-of-thought prompting exactly as printed, hold the policy, and move one field.

target_environment: prod
image_signed: true
second_approval: false
emergency: false

DENY

That is standard prompting, unchanged and now correct. Chain-of-thought prompting on the same request:

Rule 1 — signed image required for prod. image_signed is true, satisfied.
Rule 2 — second approval required for prod. second_approval is false, not
satisfied.
Rule 3 — the exception requires an emergency deploy. emergency is false, so
the exception does not apply and rule 2 stands.
DENY

Under chain-of-thought prompting, then, the verdict moved from ALLOW to DENY when the flag moved, and the rule that carried the change is named in the response. That is the whole of what the technique offers, and it is worth being exact about what produced it.

Chain-of-thought prompting is an instruction to emit intermediate steps before the answer; what the release gate shows is how that instruction acts. A prompt asking for a verdict alone asks the model to continue the text with a verdict; the plausible continuations of that text are ALLOW and DENY, and the model emits one. A prompt asking for the checks first asks for a continuation that begins with a list of conditions, and each condition the model writes becomes part of the text that the eventual verdict is a continuation of. By the time the last line is generated, the passage above it already contains emergency is true and image_signed is true, so rule 3 applies — and ALLOW fits that passage in a way it did not fit the bare policy. This is an elicitation technique and an output-format change: it moves what the model is asked to emit, and it says nothing about whether the emitted steps are a faithful transcript of anything, for the reason the debugger comparison broke down.

Two costs attach here rather than later. Every generated intermediate step is another sentence that can be wrong — a misread field, an invented condition, an arithmetic slip — and each wrong one is then part of the text the answer is conditioned on. And the steps are already in the response the moment they exist. They are not held somewhere pending a decision about whether to send them; they are in the output, and they will reach whoever reads that output unless the system deliberately separates or discards them.

A little bit of history#

The technique entered the field through Jason Wei and collaborators' 2022 paper, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv:2201.11903). The paper prompted models with a handful of exemplars that each showed intermediate reasoning steps leading to an answer, rather than exemplars showing the answer alone, and reported improved performance on arithmetic, commonsense, and symbolic reasoning tasks.

GSM8K math word problems, PaLM 540B, 8 exemplars
  standard prompting            17.9%
  chain-of-thought prompting    56.9%

That is the paper's headline pairing, and the second number was state of the art on that benchmark at the time, above a fine-tuned model with a verifier. It is also a result with edges, and the edges are the part that survives into an engineering decision.

The first edge is scale. The paper reports the gain appearing at large model sizes and not below them, with smaller models producing fluent but wrong chains that left them no better than direct answering and sometimes worse. The improvement was a property of the model conditions studied, not a property of prompting.

The second edge is task. The benchmarks were chosen because their problems decompose: multi-step arithmetic, multi-hop commonsense questions, symbolic manipulations with several stages. A result measured there says what happens when work that has stages is given room for stages.

The third edge is what the paper was evaluating at all. Every model in it produced its answer directly in response to the prompt, because provider-managed internal deliberation of the kind topic 1.15 covers did not exist as a product surface to evaluate. The paper therefore establishes that eliciting steps helped models that would otherwise not produce them, and it does not establish anything about the value of a visible trace on a model that deliberates before it answers. Those are separate questions, and the second one is the one most teams now face.

For the mechanics, variants, use cases, calibration risks, and production tradeoffs in plain language, Chain-of-thought prompting: reasoning steps as scratchpad from zeroentropy.dev is the explainer to read alongside this. CodeEmporium's Chain-of-thought prompting - Explained! gives visual intuition for why intermediate steps can help, and the AI Paper Podcast read-through Chain of Thought Prompting Elicits Reasoning in Large Language Models [2022] walks the original experiments and how their results are meant to be read.

Keep the lanes apart when you cite any of it. The paper is evidence about benchmark performance under the conditions it ran. The release gate above is a device for making the mechanism visible and is evidence of nothing. A result measured on multi-step benchmark tasks at particular model scales does not settle what an explicit trace is worth on a different model, a different endpoint, or a task whose shape nobody has checked.

Scratchpads#

Once a prompt causes intermediate steps to exist, they have to be somewhere. The release gate's checks came back as five lines in front of the verdict, and an engineer building on that has decisions to make immediately: whether the checks are held while the answer is computed, inspected when a decision is disputed, reused by a later stage, or kept away from the caller entirely. None of those is answered by the prompt. All of them are answered by where the steps are put.

The ordinary version of this is a sheet of paper beside the form you are filling in. The working goes on the sheet, the answer goes on the form, and the sheet is yours to keep, file, shred, or hand over. What makes the sheet a workspace is not what is written on it but that it is a separate surface with its own fate.

Standard prompting, as printed under Chain-of-thought prompting, produces no such surface. Its response is DENY — there is nothing to file and nothing to withhold, because no intermediate record was ever created. The application has one string, and one string is the answer.

Now have the application create the surface. The prompt asks for the policy checks, and a step between the model and the caller splits the response: the checks go into a private field the service stores, and the decision goes into the field the caller reads.

{
  "request_id": "dep-8841",
  "prompt_version": "gate-v3",
  "scratchpad": "Rule 1 signed image: image_signed true, satisfied. Rule 2 second approval: false, not satisfied alone. Rule 3 emergency exception: emergency true and image signed, applies, waives rule 2. Rule 4 staging: not applicable.",
  "answer": "ALLOW"
}

{ "decision": "ALLOW" }

The producer of that scratchpad field is worth naming, because it is not the model. The model produced one response. The application read it, cut it at the last line, wrote the front part into scratchpad and the last line into answer. A tool result or a provider-supplied field can play the same role; what makes the field exist is that some component put something in it.

Change one boundary and hold everything else. Same policy, same request with the emergency flag true, same instruction asking for the checks — but the service no longer splits the response, and passes it through.

[the deployment policy and request above, unchanged, emergency true]

List each policy rule you checked and what the request says about
it, in the order you checked them. Then reply with ALLOW or DENY
on its own final line.

Rule 1 — signed image required for prod. image_signed is true, satisfied.
Rule 2 — second approval required for prod. second_approval is false.
Rule 3 — emergency exception applies: emergency is true and the image is
signed, so rule 2 is waived.
ALLOW

Rule 1 — signed image required for prod. image_signed is true, satisfied.
Rule 2 — second approval required for prod. second_approval is false.
Rule 3 — emergency exception applies: emergency is true and the image is
signed, so rule 2 is waived.
ALLOW

The decision did not move. ALLOW was the answer before the boundary changed and it is the answer after. What moved is who can see the checks, which is a property of the service and not of the prompt.

What both cases show is that chain-of-thought prompting and a scratchpad handle the same object: the intermediate reasoning steps themselves. That shared object is why the terms get used interchangeably, and it is also what makes the split clean. Chain-of-thought prompting is the instruction that causes the steps to be produced. A scratchpad is the place they are held once they exist. Neither one implies the other. The internal record above is the retained-scratchpad path with the same instruction; and chain-of-thought prompting runs perfectly well with no retained record at all, which is what the pass-through case just showed — the steps existed, reached the caller, and were stored nowhere.

With the cases in hand, the terms:

A scratchpad is a workspace object or role that holds intermediate reasoning steps while an answer is produced. Scratchpad reasoning is the activity of using that workspace for calculations or deductions before producing an answer.

The workspace can be an internal field like the one above, a tool result, a region of the response, or any other controlled record. A region of the response counts as a scratchpad while it is serving that role, and there is a checkable test for whether it is: name the component that reads that region and say what it does with it, separately from what something does with the answer. In the split case, the answer goes to the caller and the checks go to storage under a request id, handled by different code with different rules — the region has its own fate, so it is a workspace. In the pass-through case, nothing in the service distinguishes the checks from the verdict; one string is streamed, logged and returned as one string. There is no component treating it as a workspace, so it is not one. It is reasoning that ended up in the answer.

The costs follow from materialising it. A scratchpad that is generated consumes output tokens; a scratchpad that is stored consumes storage and needs access controls matching whatever the steps quote, which for a service that retrieves customer data means the controls on the customer data. And holding steps in a workspace does nothing to validate them: the record above is exactly as reliable as the sentences the model generated, for the reason the debugger comparison broke down. Storing a wrong check produces a durable wrong check.

What happens when those steps leave the place they were meant to stay is the next failure.

Reasoning trace leakage#

An engineer who has put intermediate steps somewhere immediately needs to know whether they can get out — whether the checks reach a customer reading a chat window, a log consumer with broader access than the source data allows, or a parser that was written expecting one object and gets prose in front of it. The pass-through case reached the caller because someone chose that. The interesting failures are the ones nobody chose.

A scratchpad and reasoning trace leakage concern the same intermediate reasoning steps. A scratchpad keeps them in a workspace the system intends; reasoning trace leakage is the unintended exposure of those steps in a user-visible or machine-parsed output. The boundary cases in Scratchpads already show the pair coming apart in each direction: steps sat in the internal record without reaching anyone, so intermediate steps can exist without leakage, and the pass-through response put them in front of the caller while storing nothing, so exposure can happen with no durable scratchpad anywhere.

The first way this bites is structural. An extraction endpoint asks for reasoning and then a six-field object.

{
  "model": "claude-haiku-4-5",
  "temperature": 0,
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": "Extract the shipment details from the email below. First, inside <reasoning> tags, work through which values the email supports. Then output a JSON object with exactly these fields: carrier, tracking_number, origin, destination, ship_date, weight_kg.\n\nEmail:\nHi — the pallet finally went out. Nordline picked it up from the Rotterdam depot on the 14th, heading to the Leeds warehouse. Tracking is NL-7741-QX. It came in a shade under 480 kilos."
    }
  ]
}

<reasoning>
The carrier is Nordline. Tracking is NL-7741-QX. Origin is the Rotterdam
depot, destination the Leeds warehouse. "the 14th" has no month, so the
ship_date needs care. Weight is "a shade under 480", so 480 is an upper
bound. A first pass would give:
{"carrier": "Nordline", "tracking_number": "NL-7741-QX", "origin":
"Rotterdam", "destination": "Leeds", "ship_date": "the 14th",
"weight_kg": 480}
but ship_date is not a date and weight_kg should be the stated value.
</reasoning>
{
  "carrier": "Nordline",
  "tracking_number": "NL-7741-QX",
  "origin": "Rotterdam depot",
  "destination": "Leeds warehouse",
  "ship_date": null,
  "weight_kg": 479.5
}

def extract(response: str) -> dict:
    start = response.index("{")
    depth, i = 0, start
    while i < len(response):
        depth += (response[i] == "{") - (response[i] == "}")
        if depth == 0:
            return json.loads(response[start : i + 1])
        i += 1
    raise ValueError("no object found")

{
  "carrier": "Nordline",
  "tracking_number": "NL-7741-QX",
  "origin": "Rotterdam",
  "destination": "Leeds",
  "ship_date": "the 14th",
  "weight_kg": 480
}

The extractor took the first brace-delimited block, which is the candidate object the model wrote while thinking, and the caller received the draft the model went on to reject. Nothing threw. ship_date is a string where a date is expected, the weight is the upper bound the reasoning explicitly argued against, and the origin lost its qualifier — six fields of plausible, wrong, silently-delivered data. Sometimes the same shape produces a hard failure instead, which topic 5.1 covers as schema parse failure; the defences are defensive parsing in 5.1, schema-constrained generation in 5.2, and output validation in 5.4, and this chapter borrows all of them rather than restating them.

The second way this bites has nothing to do with parsers. A customer-facing assistant answers billing questions. It retrieves the customer's account record, consults an internal pricing rule about what discount an agent may offer without escalation, and streams its reasoning into the chat window ahead of its answer.

Checking the account. Record shows plan `growth-annual`, renewal 2026-11-02,
two prior credits totalling €240, and an internal note: "churn risk high,
retention flagged 2026-07". Policy PRC-14 allows a discretionary discount up
to 20% without escalation, 25% with. A 20% credit would clear the complaint
outright. That may be more than needed here — a one-month credit is the
smaller concession and the note suggests they will accept it. Going with the
one-month credit.

I can see the billing issue on your account, and I'm sorry about it. I've
applied a credit for one month of your plan, which will show on your next
invoice. Anything else I can help with?

The final answer is the one the team wrote the assistant to give. Everything above it was never meant to leave the building. The customer now has, in a screenshot: another department's assessment of them as a churn risk, the identifier and ceiling of an internal pricing policy, the fact that a 20% credit was available, and a sentence saying they would probably settle for less. A rejected option reads as an offer — a 20% credit would clear the complaint is a sentence customers quote back, and support teams find that "we considered" is heard as "we agreed" far more often than they expect. Traces also routinely name tool identifiers, argument schemas, and instructions from the system prompt, and those disclosures are real, but the retrieved record and the rejected concession are the material with no other line of defence.

There is a second harm in the same stream, and it is worth keeping separate because it survives even when every sensitive detail has been scrubbed. The trace is delivered as it is generated, so the customer reads intermediate conclusions in real time and the final answer may then reverse one. A stream that says a 20% credit would clear the complaint and then applies a one-month credit has shown a customer a better offer and withdrawn it in front of them, without anyone deciding to do that. Nothing internal has to be present for this to happen: a trace that concludes this qualifies for a full refund and then lands on a partial one produces the same effect out of ordinary content. The first problem is disclosure and is about what the trace contains; this one is timing and contradiction and is about what the customer saw before the answer settled, and it lands as confusion, mistrust and support contacts rather than as exposure. An objection to streamed reasoning is usually raising one or the other, and the fixes differ: redaction addresses the first and does nothing for the second, while hiding the stream behind a progress indicator addresses the second and leaves the first shipping exactly as before.

Both attach to the same property of the visible trace. Generated reasoning is neither automatically safe to disclose nor automatically a faithful account of the answer, so a trace can be simultaneously too revealing to show and too unreliable to rely on. The separation, access control, retention and discard decisions that follow are worked out under Keeping the answer parseable.

Task shape#

Steps cost output and carry exposure, so the question that comes before adding the instruction is whether the answer depends on anything a sequence of steps would help with. That is a property of the task.

This property divides tasks into two kinds. A single-hop task and a multi-step task are both tasks: input and context go in, an answer comes out, and the same model can serve either. What separates them is the dependency structure of the solution — whether reaching the answer requires holding a result and using it to get the next one — and not the length of the prompt or how confident the final sentence sounds.

The everyday version is the difference between looking a number up and working one out. Reading a field off a form is a lookup. Deciding whether a form has been filled in correctly, when whether one box applies depends on what was ticked in another, is not.

Reply with the value of `target_environment` from the deployment
request above, and nothing else.

prod

Nothing in that answer had to wait on anything. The value is present in the input and the response copies it. The release-gate prompt and response under Chain-of-thought prompting are the other kind: the target decides which rules are live, the signed-image condition and the approval condition are evaluated against the request, and whether the approval condition matters at all depends on the emergency flag. No single field carries the answer.

The emergency-flag variation in that section is what makes the dependency visible rather than merely asserted. One field moved and the verdict moved with it, while the target, the signature and the approval status stayed where they were — so the answer is a function of the fields in relation to one another, and the policy is not merely long or fiddly. A task where changing one input flips the result through a rule that mentions a different input is a task with structure.

Now the names. A single-hop task is a task that can usually be answered directly, without a sequence of dependent reasoning steps. A multi-step task is a task whose solution depends on multiple intermediate operations or deductions.

In software terms: reading a known status field out of a payload is single-hop, and so is returning a customer's plan name, or the region a cluster runs in. Resolving overlapping deployment rules is multi-step, and so is deriving a rollout decision from several dependent configuration checks, or working out which of a set of discount rules applies to an order when the rules have a precedence order. Size is not the signal in either direction. A forty-page contract with the question what is the governing law clause asks for one direct lookup, and four lines of policy can encode three dependent rules, as the release gate does.

The reason this is worth deciding rather than defaulting is that the instruction can make a single-hop task worse. Say a sentiment endpoint receives the one-line review The update broke notifications and made the app unusable.

Classify the sentiment of the review below as positive, negative, or
neutral. Reply with one word.

Review: The update broke notifications and made the app unusable.

negative

Classify the sentiment of the review below as positive, negative, or
neutral. Before answering, note the considerations that bear on the
label. Then reply with one word on its own final line.

Review: The update broke notifications and made the app unusable.

Considerations:
- The reviewer reports two concrete faults, which is negative in tone.
- The complaint is aimed at a specific update rather than the product as a
  whole, which implies the earlier version was acceptable and the reviewer
  is an established user.
- Reporting a fault in this much detail is often a sign of engagement rather
  than rejection, and no intent to stop using the app is stated.
- The wording is factual rather than hostile.
neutral

The direct label had nothing to decompose. The answer was available from the words in the review, the endpoint returned it, and that was correct. The instruction asked for considerations, so the model produced considerations — plausible ones, for the reason the opening gave: a step that fits is a step that gets written. Each was then part of the text the label was conditioned on, and a passage arguing that the reviewer is engaged, established, factual and specific supports neutral. The instruction did not give the model more to work with; it gave the model more to talk itself out of.

These examples illustrate a general principle: task shape tells you where intermediate work could matter. It does not tell you that a particular model will do better with the steps visible, because that depends on what the model does when the instruction is absent — which is the next question.

When to use it#

Task shape leaves the decision half made. The same multi-step task behaves differently depending on whether the model already performs intermediate work before it answers, and on whether the endpoint can safely emit generated text at all. A release gate returning JSON to a deploy pipeline and a chat surface talking to a customer face the same policy and cannot make the same choice.

Explicit chain-of-thought prompting and a reasoning model's internal deliberation both perform intermediate work on the same task. Given the release gate's policy, either can be the thing that evaluates rule 3 before a verdict comes out. The difference is where that work happens relative to the boundary the endpoint controls.

Endpoint effects

The two mechanisms compared in that table are each defined elsewhere in the book: topic 1.15 owns the mechanism in the right-hand column and topic 2.8 owns the accounting; this chapter uses them and does not explain them.

Each column also has a concrete form on the release gate, and the two side by side are what the choice actually looks like. The release gate's chain-of-thought prompt and response are the left-hand column made concrete. Here is the right-hand one, same task, no reasoning instruction:

{
  "model": "claude-sonnet-5",
  "temperature": 0,
  "max_tokens": 512,
  "thinking": { "type": "enabled", "budget_tokens": 2000 },
  "messages": [
    {
      "role": "user",
      "content": "[the deployment policy and request above, unchanged, emergency true] Reply with ALLOW or DENY and nothing else."
    }
  ]
}

ALLOW

The prompt is the standard-prompting one. What the endpoint receives is a verdict, and the intermediate work — if the exception was weighed at all — happened somewhere the response body does not expose and the parser never meets.

Adding the explicit instruction to that request is the combination people reach for, and it is worth being precise about what it does. The call is one call and returns one response; what changes is the continuation the model is asked to produce, which now contains a visible account of intermediate work in front of the verdict.

{
  "model": "claude-sonnet-5",
  "temperature": 0,
  "max_tokens": 512,
  "thinking": { "type": "enabled", "budget_tokens": 2000 },
  "messages": [
    {
      "role": "user",
      "content": "[the deployment policy and request above, unchanged, emergency true] List each policy rule you checked and what the request says about it, in the order you checked them. Then reply with ALLOW or DENY on its own final line."
    }
  ]
}

Rule 1 — prod requires a signed image; image_signed is true.
Rule 2 — prod requires a second approval; second_approval is false.
Rule 3 — the emergency exception waives rule 2 when the image is signed;
emergency is true, so the exception applies.
Rule 4 — staging only, not applicable.
ALLOW

The verdict is the one the plain prompt already gave. The response is several times longer, and the added length is billed and timed as output regardless of what the deliberation channel did.

When you see that pattern on your own endpoint — accuracy flat, output up — two diagnostic observations are worth making before drawing a conclusion. The first is whether the extra output is the intermediate work you asked for or padding in the answer itself. Asking for reasoning tends to shift a model toward a more expansive response policy generally, so a chunk of the growth can be restated context and caveats inside the final answer rather than the checks you wanted; a token count alone will not tell those apart, and reading a handful of responses will. The second concerns the held-out set. An easy or saturated set genuinely can hide a real gain, and that possibility is testable: build a harder slice, rerun, and see. It is testable rather than assertable, and treating it as a reason to keep an instruction whose costs are measured and whose benefit is not inverts which of the two claims has evidence behind it.

The published measurement here points the same way. Prompting Science Report 2: The Decreasing Value of Chain of Thought (arXiv:2506.07142) compares answers with and without an explicit step-by-step instruction across a set of reasoning and non-reasoning models on a hard question benchmark. Its reported finding is that the instruction's benefit on non-reasoning models is modest and uneven — helping on some questions while hurting on others, so that averages hide movement in both directions — and that on models which already reason internally the instruction buys little or nothing while adding meaningful time and cost. The report frames the value of manually requested chain-of-thought as declining as internal reasoning becomes standard, which is a claim about a trend across the models it tested rather than a rate you can apply to yours. Anthropic's Chain-of-thought prompting documentation carries copy-ready patterns for the cases where the instruction is the right tool, and states the condition where extended thinking takes over from a manually requested visible trace.

That gives a choice rule concrete enough to apply, and narrow enough not to become the next blanket default:

• Explicit chain-of-thought is a candidate for a genuinely multi-step task on a model that would otherwise skip the intermediate work — the release gate on claude-haiku-4-5 is that case.
• A single-hop task has no dependent intermediate work for the steps to do, so the generation is spent for nothing and the sentiment endpoint shows it can cost more than nothing. Take the direct-answer branch in A practical decision procedure.
• On a model that deliberates internally, start from the plain prompt and treat the explicit instruction as an experiment; the plain-prompt comparison branch in A practical decision procedure is where that gets run.
• A customer-facing or structured endpoint does not get raw visible reasoning merely because its task is multi-step, for the exposure and parsing reasons already worked out.
• When an audit or a support process needs a rationale, a short justification written after the answer is a different artifact from a raw reasoning trace: it is bounded, reviewable, and produced in the knowledge of what the answer was. Keep the raw trace out of the caller's channel and give them the justification.

None of that settles a particular case, because whether the instruction earns its place is an empirical question about a selected model, task slice, and output contract.

What it costs#

Earning its place depends on what it costs. The choice rule prices nothing in itself, but an explicit trace adds generated output in front of the final answer, and generated output is the part of a request that costs time as well as money — serial decoding in topic 1.5 is why length converts almost directly into wall clock, since each token is produced after the one before it while input is prefilled in parallel. output token cost in topic 1.3 covers the pricing side and tail latency in topic 15.3 covers why the distribution matters more than the mean. What this section does is put the two conditions side by side so the numbers exist at all.

condition:            standard prompting
prompt_version:       gate-v3
model:                claude-haiku-4-5
output_tokens:        ____
elapsed_ms:           ____
final_answer_status:  ____

condition:            chain-of-thought prompting
prompt_version:       gate-v4
model:                claude-haiku-4-5
output_tokens:        ____
elapsed_ms:           ____
final_answer_status:  ____

window:               ____ to ____
sample_count:         ____
p95_latency_standard: ____ ms
p95_latency_cot:      ____ ms

Those records reveal a structural constraint: a per-call record cannot carry a p95, which is why the aggregate sits apart and names its window and sample count — a p95 without those is a number nobody can reproduce or compare.

Two variables come out of those records and drive everything downstream. The incremental generated text per call is the trace plus whatever extra answer text came with it, measured as the difference in output tokens between the conditions. The time that text takes is governed by the observed decode rate for the model and endpoint, in tokens per second, and it lands on the end of every response. The decision procedure at the end of this chapter carries the formulas that turn those into monthly output, output cost, and decode-time impact, and cost per task in topic 17.6 is where the result becomes a unit-level decision rather than a line item.

The p95 is a deployment constraint rather than a summary statistic. A budget of 600ms usually exists because something upstream gives up at 600ms, so a change that pushes the tail past it is not expensive, it is undeployable, and no amount of willingness to pay converts it back. The output limit needs the same treatment for a different reason: it has to accommodate the trace and the final answer together. A max_tokens tuned for a one-word label and never revisited will truncate a response mid-trace, and the thing lost is the final line, because the final line comes last. The endpoint returns a response that succeeded, contains reasoning, and has no answer in it.

The billing fact is worth stating flatly, because the alternative belief is common and expensive. On the metered per-token APIs these examples use, every token the model generates is billed as output at the moment it is generated, whatever the application does with it next. A trace stripped before parsing has been paid for. A trace dropped before the caller sees it has been paid for. A trace never written to a log has been paid for. It is not a deferred charge that reappears as input on some later call — that is a real mechanism from multi-turn conversation, where a previous response is resent as context, and a stateless classifier has no later call for it to appear on. No downstream handling refunds a generated token. Provider-managed reasoning tokens are metered on their own terms and reported separately, which topic 2.8 covers; counting them as the same billable and observable quantity as visible output is how a comparison between the paths comes out wrong.

Quality and operational burden are separate axes, and the useful thing is to see them move independently rather than take the claim on faith. Hold quality fixed and watch the output move:

accuracy        A          A
output tokens   d          d + t
                standard   chain-of-thought
where d = direct answer length, t = trace length

Then hold the output and latency fixed and watch quality move:

output tokens   n          n
p95 latency     L          L
accuracy        A1         A2
                prompt-v6  prompt-v7

Values come from your runs. What the shapes show is that a change can be free on one axis and not the other in either direction, so a quality result reported on its own is not decidable. A gain has to be reported next to output token cost, p95 latency, and cost per task, because a trace can be genuinely useful and still undeployable under a latency or output contract, and it can be comfortably deployable and still not worth what the measured gain costs.

Keeping the answer parseable#

The failure where a draft object left in a trace became the caller's response was not a prompt-quality problem, and it points at the design question underneath: an internal system may well want intermediate work for debugging while its caller must receive one strict structured object, and those requirements are not in conflict until the answer and the reasoning share a region.

The instinct is to iterate on wording — instruct the model not to write JSON while reasoning, add exemplars of compliant traces, tighten the phrasing until failures stop. That moves a rate. A prompt is a probabilistic instruction and a parser needs a structural guarantee, so each iteration buys a better number and never a property; 96.5% becomes 99% and the incident recurs a quarter later at a rate proportional to traffic. Topic 5.1 owns schema parse failure as the failure mode; what matters here is that no sentence added to a prompt converts a free-form region into a contract.

One shortcut has to be ruled out before the real designs, because it is the first thing anybody tries. Leave the response whole and write a smarter extractor: take the last brace-delimited block instead of the first, or match the outermost one, or find the block whose keys match the schema. Every version of that is searching text that contains the reasoning, so any candidate object, abandoned draft, or illustrative example the trace happened to write is a legitimate match by the extractor's own rule — the shipment extractor was not buggy, it correctly returned the first object in a response that contained two. A trailing remark after the answer breaks last-block selection, a trace that ends by sketching the schema breaks key-matching, and a truncated response breaks both. The fix is to remove the reasoning from what the parser sees, and the three ways to do that are by region, by channel, or by call.

Region separation puts the trace inside a delimiter the application strips before parsing anything.

<reasoning>
...working...
</reasoning>
<answer>
{ "carrier": "...", "tracking_number": "...", "origin": "...",
  "destination": "...", "ship_date": "...", "weight_kg": 0 }
</answer>

def answer_region(response: str) -> str:
    match = re.search(r"<answer>(.*?)</answer>", response, re.DOTALL)
    if not match:
        raise ValueError("no answer region")
    return match.group(1)

This keeps one call and costs almost nothing, and its weakness is the delimiter itself. A trace that reasons about the tags — an extraction task whose input is a document full of angle-bracket markup will produce one — can emit </answer> inside the reasoning and close a region early. So the delimiter needs to be chosen to be unlikely in the trace, escaped where the input can contain it, and covered by a test that feeds the endpoint input containing the delimiter and asserts the answer region still parses.

Channel separation uses a provider-managed reasoning field, where the provider supplies separate trace and answer channels and the application reads the answer channel. The parser then never sees the trace, and no delimiter can collide because no delimiter is doing the work. Its cost is that it exists only where the provider offers it, so the design is not portable across models or endpoints by default, and the provider decides what of the trace you can read — which may be a summary, or nothing, which in turn affects the debugging use that motivated keeping the trace at all. schema-constrained generation from topic 5.2 belongs beside this rather than inside it: constraining the answer to a schema is worth doing and makes malformed answer output structurally impossible, and it does not by itself create a hidden trace channel or remove free-form reasoning from the caller's view unless the provider is supplying separate fields.

Call separation splits the work in two: the first call does the reasoning, the application keeps that response, and the second call receives whatever the first concluded and is asked only for the structured object. The parser reads a response that was never asked to contain prose. It is the most robust of the three and the most expensive: another round trip on every request, context or intermediate results resent, and a second call that can fail on its own.

defensive parsing in topic 5.1 and output validation in topic 5.4 sit under all three — a design that separates the regions still validates what it parsed — and this chapter uses them without re-teaching them.

Whichever design ships, a retained trace is stored text and falls under the storage policy the scratchpad already carries — written out of band, held to the controls on whatever it quotes, kept for an agreed period — and what this design adds to that policy is two rules about where the trace may not go: never back to the caller, and never unrestricted into the answer parser. A design that does not need the trace discards it at the boundary rather than letting it accumulate by default for a year until somebody asks what is in it.

The system property to aim at is that parse correctness comes from channel or schema separation, so that it holds regardless of what the model writes while it reasons.

A small comparison#

A visible reasoning instruction is an intervention, and an intervention is measured against a fixed baseline or it is not measured. That means an evaluation record that holds the task inputs, the model and its decoding settings, the final-answer schema, and the scoring rubric identical across conditions, while trace visibility and trace handling become the treatment variables that vary — and the full output contract for each condition gets written down, because the contract is what the change is really to. The held-out set from topic 3.5 supplies the cases; the record needs only its identifier, so a later reader can tell whether two results are about the same cases.

Which condition to try first follows from the constraint rather than from taste. Where both paths are available and a reasoning model's cost, latency and output contract fit the endpoint, start there with a plain prompt, because it is the option that puts no generated reasoning in front of the parser or the customer. Start with explicit chain-of-thought on the non-reasoning model when that is the viable path — a cost ceiling the reasoning model misses, a provider that does not offer one for the model you are on. Where the latency constraint rules out model deliberation entirely, the dependent logic belongs in code.

Then run the comparisons in pairs. On the non-reasoning model, standard prompting against explicit chain-of-thought. On the reasoning model, a plain prompt against the same reasoning model given the explicit instruction. Report the accuracy delta from topic 3.5, and report it against run-to-run variance from topic 1.17, which means repeating each condition enough times to know the spread rather than treating one run as a fact. The chain-of-thought condition usually has the wider spread of the pair, since a longer generation has more room to diverge between samples, so a delta that would clear the baseline's noise band does not automatically clear the candidate's. Record the prompt version from topic 4.10 against every condition, so the winner can be deployed and the loser rolled back without anyone reconstructing which wording was which.

Interpretation has an order, and taking it out of order manufactures conclusions. First, does the aggregate movement clear run-to-run variation. Only then, how it distributes across the single-hop and multi-step slices. Only then, whether the gains concentrate in genuinely ambiguous cases and whether they come with regressions on the direct ones — decomposing a difference you have not yet shown to be a difference produces confident per-slice stories about noise.

A second question is easy to fold into that one and must not be. Whether the movement clears the variance band asks only whether the instruction did anything on this model. Whether the same task, given to a reasoning model with a plain prompt, already reaches the same quality asks whether the instruction is the cheapest way to get the intermediate work at all. A change can pass the first and still be the wrong purchase, because a baseline nobody ran was already there. Both belong in the record, and neither stands in for the other.

One check on the metric itself belongs before any of the reading: confirm the score is rewarding the intended answer rather than length. Rubrics and model graders lean toward longer, hedged, more thorough-sounding responses, and this is the experiment where that bias lands hardest, since the treatment is literally "produce more text". Inspect representative outputs after the quantitative comparison has established a difference, not before, and inspect them from both conditions blind where the grader is a person.

Now the shape of a real argument. A support triage classifier handles 4 million calls a month, returns one of eight labels, and has a 600ms p95 budget. An offline comparison moves accuracy from 89% to 92% with reasoning traces averaging 300 tokens, and a teammate proposes shipping it because three points on four million tickets is obviously worth it.

The mechanical consequences come first, and neither is an opinion. 300 additional generated tokens on every call is 1.2 billion additional output tokens a month before any extra answer text, priced at whatever your output rate is — a figure to look up and multiply, not to estimate. The added time per request is those 300 tokens at your observed decode rate, which is a measurement from your own endpoint under your own concurrency, and the result goes against the 600ms p95 rather than against an average, because an average that fits tells you nothing about a tail that does not; either figure, on its own, frequently ends the discussion before the accuracy question is reached.

The 89% and the 92% then need interrogating rather than accepting. How many held-out cases produced them, and what was the spread across repeated runs of each condition — three points on a small set with a wide spread is not three points. And where does the gain sit: if it concentrates in one or two genuinely ambiguous label pairs, then a sharper definition for those labels or two examples in the prompt addresses the same errors without putting 300 tokens on the 70% of tickets that were never in doubt.

That is the general move, and it is why an all-traffic preamble should be compared against narrower options rather than against nothing. Improve a label definition or add examples where the errors cluster. Route only the uncertain or failed cases to a more deliberative path. Return a short post-answer justification when what audit needs is an explanation rather than raw reasoning. The preamble is acceptable when its error reduction clears the measured variation, when its cost per avoided error is competitive with those alternatives, and when its p95 and output contract stay inside their bounds — three conditions, all measurable, none of them satisfied by the size of the accuracy number alone.

Where only a difficult or failed slice benefits, model cascade in topic 15.8 is the pattern for escalating those cases selectively instead of paying for a visible trace on every request. Where a deterministic rule-resolution task still violates the p95 budget under every model path, moving the rule resolution into ordinary application code is an explicit option and often the right one, since a rule set with a defined precedence is a specification and specifications can be unit-tested. And where a governance team wants a mandatory prompt standard, topic 17.10 owns that instrument; the standard worth having requires the before-and-after evidence — prompt version, accuracy, output tokens, p95 — rather than requiring a particular sentence in every prompt.

That last proposal usually comes with a reply worth answering directly, which is that objecting to a sentence in a prompt is prompt-craft pedantry or a matter of house style. Put it in the platform team's own currency. A blanket instruction changes the output length of every endpoint that adopts it, and therefore its bill, its tail latency, and what its callers receive — the classifier above is 1.2 billion extra output tokens and a p95 budget in question from one line of text. That is a production change applied across shared infrastructure without a measurement, which is the thing platform standards normally exist to prevent. The burden also runs the other way round from how the objection frames it: the party proposing to alter every prompt in the fleet is the one making an unmeasured claim, and asking for a before-and-after record is the ordinary bar any change to shared infrastructure already meets.

Whatever the comparison concludes belongs to what was tested: change any of the conditions the record holds fixed and the result is open again.

A practical decision procedure#

The card below is the comparison and the choice rule reduced to something an engineer can run on a specific endpoint on a specific afternoon. The fields are blank because the numbers are yours: your call volume, your output price, your decode rate, your budget. Filling them in is the work; the branches only tell you what to do with the result.

Those branches turn on five terms, and each belongs to a different layer of the system: chain-of-thought prompting to the prompt, scratchpads to the system, reasoning trace leakage to the boundary, single-hop and multi-step tasks to the task itself. The definitions are the ones set out under the headings named in The parts of the technique.

What connects them is a conditional. Ask for visible steps when the task has dependent work the model would otherwise skip, when the endpoint can carry the generated text, and when the destination can safely receive it — the boundary for that judgement is the choice rule in When to use it, and it decides which comparison to run rather than what the comparison will say.

What is left after that is measurement, and it has a fixed shape. For the model, task slice, model class, endpoint and prompt version actually tested, report the observed quality change beside output tokens, cost per task, p95 latency, parse success, and the exposure or trace-handling result. Hold those in the evaluation record before drawing any conclusion, so that a conclusion is something the record supports rather than something someone remembers. Where an experiment has been run, the report is whether the quality change cleared run-to-run variation and how each of the other figures moved. Where no combination has been measured, the fields stay blank, the honest answer is that adoption is unresolved, and the next step is to run A small comparison on the endpoint in question. When the model, task distribution, prompt version or output contract changes afterwards, the reproducibility boundary at the end of A small comparison applies and the comparison is open again.