Logits and the decoding policy

Topic 1.0 established what a model does when it is called. It consumes a sequence of token IDs, runs them through its weights once, and produces a score for every entry in the vocabulary it was trained against. Topic 1.1 established where those IDs come from and what a vocabulary is. Neither of them produces text. A score per vocabulary entry is an answer to a question about one position — what belongs next, given everything before it — and a support agent's drafted reply is a few hundred of those answers laid end to end.

Laying them end to end is a matter of running the same call again with its own last answer appended to what it reads. The forward pass is unchanged between one run and the next; what changes is the sequence handed to it, which is one token longer each time.

Which token gets appended is a separate question, and nothing so far settles it. A score per vocabulary entry leaves the choice wide open: the highest-scoring entry, an entry drawn from the top of the distribution, an entry drawn under a temperature. The loop runs the same way whichever rule is plugged into it, and the rule itself belongs to topic 1.6. The order in which the tokens of an answer come into existence, and what that ordering costs, is settled without reference to the rule at all.

Where that cost lands is a latency budget, and a latency budget gets written before anyone has watched the loop run. An engineer is building a support-ticket endpoint. A request carries a ticket and its conversation history; the response is a JSON verdict with two fields. risk_score is a number that routes the ticket to a queue. reasoning is a paragraph explaining the number, which gives the human on that queue something to read before they open the thread. The latency budget gets written the way every latency budget on this team has been written, from the size of what goes in. The requests that worry the engineer are the sprawling escalations: forty messages of back-and-forth, with a stack trace pasted into three of them, because those are the requests with the most to read.

Then the endpoint goes to staging and the readings arrive the wrong way round. The forty-message escalation comes back promptly. A brand-new one-line ticket — card declined, please help — leaves the caller waiting noticeably longer, and the only thing that separates it from the escalation is that the one-line ticket drew a long reasoning paragraph while the escalation drew a short one. Nothing in the request bodies predicts that ordering. The prompt was the biggest thing in the request, and it turned out not to be the expensive thing.

A little bit of history#

The framing that makes an endpoint like this legible is older than the endpoint and simpler than it. Microsoft Learn's LLM Fundamentals sets out the whole of what a decoder model does at inference as repeated next-token prediction: the model is trained to answer one question, the answer to that question is a distribution over the vocabulary, and text generation is that question asked over and over. Topic 1.1's tokenizer supplies the units the question is asked in. Topic 1.0's forward pass is the machinery that answers it once. Everything in this topic is about what that repetition costs.

A Survey on Efficient Inference for Large Language Models, a 2024 survey, is the formal account this topic leans on, and it is leaned on for one specific fact: decoding is inherently sequential, each token's forward pass depending on the token the previous pass produced. That dependency is the reason a decode's cost cannot be collapsed the way the cost of reading a prompt can, and the argument for it is built up later here rather than borrowed as a given. For now it is a flag planted where the mechanism will land.

Where the survey states the repetition formally, Andrej Karpathy's Deep Dive into LLMs like ChatGPT runs the sampling loop end to end, so the repetition can be watched rather than read about. zeroentropy.dev's Autoregressive generation: how decoder LLMs produce text supplies the engineering vocabulary a serving team actually uses for it — prefill, decode, KV caching. Prefill and decode belong to topic 1.9 and the KV cache to topic 1.10, which define them; they are used below and not re-taught.

The moving parts of a decode#

Those three concepts play out across five moving parts. The decoding loop is the repeating control flow — the thing that runs, and runs again, until something stops it. The decoding step is one iteration of that loop, and it is the unit every cost in this chapter is denominated in. The generation prefix is what gets fed back in: the run of tokens produced so far, which each step reads and each step lengthens. Causal token dependency is what that feeding-back implies about a token once it has been chosen. Greedy decoding is the simplest possible rule for making that choice, and a rule with nothing left to chance is a rule a trace can be read against.

The loop, the step, the prefix and causal dependency describe one piece of machinery at four grains. Greedy decoding is the policy in the swappable slot, and swapping it changes not a word of what the other four say.

The decoding loop#

The decoding loop is the control flow that runs a forward pass over the sequence it has, appends one chosen token to that sequence, and repeats until a stop condition ends it. A stop condition is an end-of-sequence token the model itself produces, a maximum output length the caller set, or a stop string the caller nominated; topic 1.8 owns which of those fires and when.

That loop is easiest to watch on a short run, and the smallest useful case is the ticket endpoint answering one ticket. The prompt holds the system instructions, the JSON schema the verdict has to match, and the ticket thread. The response has to open with a brace, and the first field in the schema is reasoning.

step  forward pass runs over          token chosen   sequence so far
1     prompt                          {              {
2     prompt {                        "              {"
3     prompt {"                       reason         {"reason
4     prompt {"reason                 ing            {"reasoning
5     prompt {"reasoning              ":"            {"reasoning":"
6     prompt {"reasoning":"           The            {"reasoning":"The
7     prompt {"reasoning":"The        ␣customer      {"reasoning":"The customer

Seven rows, seven forward passes, seven tokens. The word reasoning — a field name the schema fixed in advance, one the model had no discretion about — still took two steps to write, because the vocabulary splits it in two and each half is its own choice. The column on the left grows by exactly one token per row, and the column on the right is what the next row's left column will be.

That table describes what a run looks like from the outside. Describing what happens inside one row needs a name for a row.

The decoding step#

That row has a name: a decoding step, one iteration of the loop — one forward pass over the sequence as it currently stands, one token chosen and appended before the next iteration begins. Loop and step are the same machinery seen at two grains — the loop is the control flow that repeats, the step is what repeats — and every quantity in this chapter is counted in steps rather than in loops, because a request runs one loop and spends many steps inside it.

What a step's forward pass actually produces is a row of scores called logits, and the rule that turns them into a single token is a decoding policy. Topic 1.6 owns both: what a logit is, how softmax turns a row of them into a distribution, and which policies exist to pick from that distribution. A step plugs a choice into that slot and carries on. What matters at this grain is only that exactly one token comes out of it, and that the token is appended to the sequence before anything else happens.

The loop therefore runs once per output token. Not once per request, not once per field, not once per sentence: a 2,000-token drafted reply is 2,000 iterations of the control flow above, each one a full pass over the model's weights, and the response arrives when the last of those iterations finishes. That is the exchange rate the engineer's staging readings were obeying. The forty-message escalation was read; the long reasoning paragraph on the one-line ticket was written, at one pass apiece.

Something got appended in every row of that trace, and the thing being appended to grew with each row. It needs a name of its own, because the next question is what exactly goes back into the model.

The generation prefix#

The generation prefix is the sequence of tokens the model has produced so far, fed back in as conditioning input for every step after the one that produced them. The prompt and the prefix are alike in what they are: token sequences that a forward pass reads, drawn from the same vocabulary, indistinguishable once they are inside the model. They differ in where they come from and in what they do over a request. The prompt arrives whole with the request and does not change while the loop runs. The prefix starts empty and gains exactly one token per step.

step   prompt tokens   prefix tokens   generation prefix, in full
8      1,180           7               {"reasoning":"The customer
9      1,180           8               {"reasoning":"The customer was
10     1,180           9               {"reasoning":"The customer was charged
11     1,180          10               ...was charged twice
12     1,180          11               ...twice and
13     1,180          12               ...and the
14     1,180          13               ...the second
15     1,180          14               ...and the second charge
16     1,180          15               ...charge has

That one-token-per-step accumulation is visible in the trace as movement: the middle column is the only one that moves. Rows 11 onward print the tail of the prefix rather than the whole of it, because the head of it has not changed since row 8 and will not change again.

Name what that movement represents — what passes between a step and the forward pass inside it — and it takes one sentence each way. Each step's forward pass consumes the prompt followed by the prefix as it stands at that moment, and each step's chosen token is exactly what lengthens the prefix for the step after it. The prompt is the fixed portion of that input and the prefix is the moving portion, which is why step 16 and step 8 are the same operation over different data rather than different operations.

Every step therefore conditions on the whole prefix built before it. That raises a question about the other direction: once a token is in the prefix, can a later step do anything about it.

Causal token dependency#

It cannot. Causal token dependency is the constraint that a step's choice is conditioned on the entire prefix through the previous step, and that the token it chooses is never revised once emitted. There is no second pass over the finished answer, no editing stage, no moment at which the model reads what it wrote and reconsiders. Information flows one way along the sequence: from earlier positions into later choices, never backwards.

That constraint is invisible on a trace of seven tokens and decisive on a schema. The ticket endpoint's verdict has two fields, and a JSON schema fixes the order they are emitted in.

{
  "type": "object",
  "properties": {
    "risk_score": { "type": "number" },
    "reasoning": { "type": "string" }
  },
  "required": ["risk_score", "reasoning"]
}

{"risk_score":0.4,"reasoning":"The customer was charged twice and the second charge has not settled, which is a duplicate-capture pattern rather than a disputed transaction."}

{
  "type": "object",
  "properties": {
    "reasoning": { "type": "string" },
    "risk_score": { "type": "number" }
  },
  "required": ["reasoning", "risk_score"]
}

{"reasoning":"The customer was charged twice and the second charge has not settled, which is a duplicate-capture pattern rather than a disputed transaction.","risk_score":0.62}

Same endpoint, same ticket, same model, same policy. The schemas differ in the order of two keys and the responses differ in a number.

To see why a reordered key changes the number, work the mechanism from the trace. Under the score-first schema, the token that carries the digit after "risk_score": is chosen at a step whose prefix holds {"risk_score": and nothing else. No sentence about duplicate captures exists anywhere in the sequence at that moment, because no step has produced one, so nothing about the reasoning can have conditioned the choice. The paragraph that follows is then generated at steps whose prefix already contains the score, and it is written in the presence of a number it had no part in setting.

Under the reasoning-first schema the order of those events is reversed. Every token of the sentence about duplicate captures is emitted first and sits in the prefix. When the step that chooses the digit finally runs, that sentence is part of the input its forward pass reads, exactly as the prompt is.

What the mechanism guarantees is the difference in conditioning, not a guarantee that any particular ticket's number moves. Two orderings can land on the same digit. What cannot happen is a score-first digit that was informed by reasoning tokens, because those tokens did not exist when it was chosen.

The same constraint explains why an early token has influence out of proportion to its size. If the endpoint forces the response to open by prefilling {"risk_score": into the prefix before the loop starts, every step for the rest of the request reads a sequence that commits to a score-first answer, and no later step can undo the commitment. Change one token near the beginning of a generated answer and the whole remainder diverges, not because anything propagates backwards but because every subsequent choice was conditioned on a sequence that now differs.

The cost of causal dependency is that early commitments compound. A forced opening token, a field emitted in the wrong order, a first sentence that misreads the ticket: none of them can be repaired by a later step, and each of them is in the prefix that every later step reads. The loop has no mechanism for regret.

Greedy decoding#

Every step so far has been described as choosing a token without saying how. The simplest rule available is worth naming, because a rule with no randomness in it is what makes a trace reproducible enough to reason about.

Greedy decoding always selects the single highest-scoring token from the logits at each step. No draw, no threshold, no randomness: the entry with the largest score wins, and ties are broken by whatever ordering the implementation uses.

Run the ticket endpoint twice against the same prompt with that rule in the slot.

same prompt, same weights, greedy at every step

run   step 1   step 2   step 3   step 4   step 5   step 6   step 7
one   {        "        reason   ing      ":"      The      ␣customer
two   {        "        reason   ing      ":"      The      ␣customer

The rows are identical, and the reason they are identical is worth spelling out rather than accepting as a property of the word "greedy". A forward pass is a deterministic function of the sequence it reads: the same tokens through the same weights produce the same logits. Greedy decoding then applies a deterministic function to those logits. Step 1's input is the prompt, which is fixed, so step 1's token is fixed; step 2's input is the prompt plus step 1's token, both now fixed, so step 2's token is fixed; and the argument runs all the way down the sequence. A fixed prefix and a fixed policy leave nothing to draw from, so there is nothing that could come out differently.

Policies that draw from the distribution rather than taking its maximum are where run-to-run variation comes from, and topic 1.6 is where they are set out. Naming the difference is what lets an engineer tell two failures apart. When two runs of the ticket endpoint disagree, either something about the input changed or the policy in the slot has a draw in it, and those call for opposite investigations. Under greedy decoding the second explanation is unavailable, so a disagreement is evidence that the sequence differed.

Determinism is what makes greedy the rule to reason a trace against, and it is a property of the rule rather than a recommendation to ship it. Greedy is one member of the policy class topic 1.6 opens, and choosing between the members of that class is a decision that belongs there.

What a decode costs#

Choosing a policy settles what the model does with the distribution at each step; it does not settle what that step costs. The cost lives one level down, in the structure the policy runs inside. Prefill and decode are the names topic 1.9 gives to the halves of a request, and each of them is a forward pass over the sequence built so far. Same weights, same operation, same kind of input. That shared description is what makes the difference between them worth deriving instead of asserting.

Every position of a prompt is known before the request starts. The token at position 900 of a ticket thread does not wait on the token at position 899 to be computed, because both arrived in the request body. One forward pass can therefore take all of those positions at once, scoring each of them against the positions before it in the same pass.

A decoding step's forward pass has no such luxury, and causal token dependency is why. The input to step 301 includes the token step 300 chose. Until step 300 has run, that token does not exist, so step 301 has nothing to read. There is no arrangement of the same work under which the answer's positions are evaluated together, because the later positions are not knowable until the earlier ones have been decided.

That asymmetry has an arithmetic, and two requests to the ticket endpoint make it visible — same model, same hardware, an idle server, and nothing differing between them but the shape of the prompt and the length of the answer asked for.

                          Request A                    Request B
                          20,000-token thread,         200-token new ticket,
                          terse verdict                long drafted reply

prompt tokens             20,000                       200
output tokens             50                           2,000
prefill forward passes    1, over all 20,000           1, over all 200
                          positions at once            positions at once
decode forward passes     50, one after another        2,000, one after another
total forward passes      51                           2,001

Request A carries a hundred times Request B's prompt and spends one pass on it. Request B spends forty times Request A's decode passes, and every one of those passes waits for the one before it. A single pass over 20,000 positions costs more than a single pass over 200 — more memory moved, more arithmetic — but it is one pass either way, and the gap between one pass and forty extra passes is not a gap that a wider prompt closes. Request A finishes first, and usually by a wide margin.

Ranking those requests by total tokens gets the order backwards, because a total sums a quantity paid once with a quantity paid per unit. Output length is the dominant latency term, and it is dominant for a structural reason rather than an empirical one: it is the only term multiplied by a serial pass count.

Three things overturn that ranking, and each of them is a change to something the comparison assumed was held still. A prompt large enough that a single pass over it costs more than the whole run of decode steps puts prefill back on top, which is a question about how far apart the prompt and the output actually are rather than about the shape of the rule. A server where Request A queues behind other traffic and Request B does not adds waiting to A's clock that no count of its own forward passes predicts; concurrency belongs to topic 2.12. And an endpoint whose batching state differs between the calls is two serving contexts wearing one endpoint's name, since batching, which topic 1.17 owns, changes what a decode step costs in wall-clock terms. Inside a fixed serving context the ranking holds. Across contexts it is comparing two exchange rates as though they were one.

Streaming and time to first token#

The context-dependence of ranking comes from what each tier can observe in time. A decoding step is visible from inside the server, where every token appears the moment its pass finishes. A client sees only what the server chooses to hand over, and when.

Streaming, which topic 2.9 owns, and the total latency of a buffered response describe the same decoding loop: the same prompt, the same policy, the same number of steps, the same token at each step. What streaming changes is when a client is shown a token, not how many times the loop runs. Read the two modes across an 800-step reply:

Eight hundred steps in each column, in the same order, at the same cost apiece. Time to first token, a metric topic 1.9 owns, collapses under streaming because step 1's output leaves the server as soon as step 1 finishes instead of waiting for step 800. Total time is set by the number of serial steps, which the delivery point does not touch. A team that turns streaming on and then reports a latency win will find no total-duration measurement supporting it, and the win they actually got is real and sits in a different column.

With delivery separated from generation, the two components of a request's clock can be moved one at a time. Each comparison below holds the model, the hardware and the concurrency fixed, and moves one thing.

output pinned at 400 tokens

request             prompt tokens   startup                  per-token tail
new ticket          200             one pass over 200        400 decode steps
escalation thread   20,000          one pass over 20,000     400 decode steps

Time to first token moves between those rows, because the first token cannot be delivered until the pass over the prompt has finished and startup is where a prompt's cost lands. Inter-token latency, also topic 1.9's, stays roughly flat, because after the first token every step is doing the same work in each row.

prompt pinned at 200 tokens

request          output tokens   startup             per-token tail
terse verdict    50              one pass over 200   50 decode steps
drafted reply    2,000           one pass over 200   2,000 decode steps

Total time moves between those rows and time to first token stays roughly flat, which is the mirror image of the sweep above. Startup is identical because the prompt is identical; the tail is forty times longer in the second row because forty times as many steps run.

Those sweeps isolate startup from tail conceptually, and a real deployment can blur the split. KV cache state, which topic 1.10 owns, means a prompt whose prefix was seen recently may skip work another prompt of the same length pays for, so two prompts of equal length are not automatically equal startups. Batching means a decode step's wall-clock duration depends on how many other sequences share the pass. Neither of those breaks the mechanism; each breaks the assumption that the only thing differing between the requests being compared is the thing you varied, which is why the checklist above asks for those conditions to be held rather than assumed.

Cancelling a stream#

Given all that, the first thing a client watching tokens arrive will want to do with them is stop a generation that has gone wrong.

Stream cancellation, topic 2.9's, does exactly one thing to the loop: it prevents future decoding steps from running. It has no effect on steps that have already completed, because a forward pass that has run cannot be un-run. The drafted-reply request makes the split legible.

event                                            decode steps   output tokens
                                                 run            billed
steps 1-300 decoded and written to the client    300            300
client closes the connection
step 301, already in flight, runs to completion  1              1
steps 302-800 never scheduled                    0              0

Read the saving off the right-hand column. Providers bill for compute performed, so the request is charged for its full prompt plus the 301 output tokens whose forward passes actually ran. The 499 steps that never happened are never charged, and they are the whole of what cancellation buys. Cancelling aggressively on a stream is therefore a genuine cost control and never a refund.

The overshoot at step 301 is a property of the mechanism rather than a number to budget against. A cancel signal reaches a server that is in the middle of a step, that step finishes, and no further step is scheduled. How many tokens land between the client's decision and the server's stop depends on propagation and on where in a step the signal arrived, which is a measurement to take on a specific deployment rather than a constant.

A cancelled request also cannot be picked up where it left off, and causal token dependency is the whole of the reason. Sending a fresh request that asks directly for the passage that was never reached gives the model a different prefix: the abandoned run had 301 particular tokens sitting in front of step 302, and the new request has whatever its own prompt puts there. Every step of the new request conditions on that different sequence, so it is a different computation with a different output rather than a continuation of the first one. No path through the model produces token 750 without tokens 1 through 749 having been produced.

Given that the tokens already decoded are billed either way, letting a cancelled generation run to the end and caching its result is sometimes the better trade, and the condition on it is specific. Finishing is worth it when the output is likely to be asked for again and stable enough to reuse: a summary of a ticket thread that will not change, a risk report other agents in the same workflow will also want. The marginal cost is only the steps that had not yet run, which is small against a full re-run of prompt and answer from scratch, and prompt caching, topic 2.15's, makes the re-run cheaper without making it free.

Finishing is wrong when the output is personalised or one-shot, because a cache entry nobody reads is a bill with no return. It is also wrong under load, and that case is the one most easily missed. The remaining steps occupy a decode slot in the batch, so a request that has already been abandoned is holding capacity that a waiting request needs, and the money saved on a possible future call has been converted into someone else's latency now. Tokens are money and capacity is latency, and finishing abandoned work spends the second to save the first at exactly the times the second is worth most.

Trimming the output schema#

Which brings the engineer back to the question staging raised, now arriving from a product manager: can the endpoint return only risk_score, since reasoning is long and the response feels slow.

An output schema field, topic 5.3's, costs exactly the decoding steps spent generating its tokens — no more, and no less. The reason there is nothing else to count is the shape of the loop: no complete verdict object is computed somewhere and then serialised down to the requested fields. There is no object to filter. Every character in the response was produced by its own step, conditioned on the prefix in front of it, in the order it appears. A field is not a slice of a result; it is a stretch of the run.

That makes the product manager's request answerable, and it splits into two interventions that look interchangeable and are not. Deleting the field and instructing the model to reason silently each start from the reasoning-first schema exhibited earlier, and each ends with a response containing only a number.

"properties": { "risk_score": { "type": "number" } },
"required": ["risk_score"]

The same endpoint, the same ticket, the same model: only the schema's properties and required have changed, with reasoning gone from each.

The alternative keeps the intent of the paragraph and drops its output. The schema is the one directly above, and one line is added to the system prompt.

Work the ticket through carefully before you answer. Return only the score.

Set side by side, the two agree in every column that the loop can see, and differ only in what they appear to promise:

Neither row is a cheaper version of the reasoning-first run. Both are the shorter, different computation the schema pair already exhibited, and whether that computation's output is as good is a question for the endpoint's evaluation set rather than for its latency budget.

Topic 2.8 owns reasoning models, and they sharpen that same point to its edge. A reasoning model generates reasoning tokens — intermediate tokens produced before its visible answer and not returned to the caller. Those tokens are produced by the loop like every other token, one step each, and they enter the prefix that the visible answer is then conditioned on. Hiding a token from a response and never generating it are different events with different costs, and only the second one saves a step. A caller who sees a short answer from a reasoning model has not necessarily bought a short decode.

So the product manager's question was a quality decision wearing a latency question's clothes. Deleting the field and silencing the reasoning cost the same and produce different things, which is the comparison that matters and the one the framing hid. And the actual complaint — that the response feels slow — is a claim about time to first token, which streaming addresses without removing a token from anywhere.

That streaming works at all is a property of the decoding loop's structure. One loop, run once per output token, in which every step reads everything generated before it and nothing already emitted can be taken back. That is the whole of the machinery, and everything worked out of it above follows from those two properties: why a long prompt is cheap and a long answer is not, why field order changes an answer and not only its price, why a cancelled stream saves the future and not the past, and why a schema is a plan for a run rather than a filter on a result.

The heuristic that comes out of it has a fence around it, and the fence is part of the tool. Output length ranks two requests when model, hardware, batching and concurrency are the same for each of them, and it stops ranking them the moment any of those differ between the requests being compared — which is what the three overturning cases under the forward-pass budget were cases of. A short completion on a loaded, heavily batched endpoint can lose to a long one on an idle endpoint, and no amount of counting output tokens will see it coming.

Reason about one decode at a time, then. Where the serving context is not being held still, the checklist is the honest instrument, and a number that travels without the conditions it was taken under is a number somebody will misread.