Greedy decoding and floating-point noise

Non-determinism, in the sense this topic uses the word, means that one request sent twice to the same deployment — same text, same parameters, same model name — can come back with two different completions. The property people usually want instead is reproducibility: send the request again next week and get back the string you got last Tuesday, so that when an output does differ you know something real changed. Neither property is delivered or withheld by a setting, which is why the topic needs a chapter on where the variation physically comes from before any of the later ones can be useful. The rest of this topic is an elaboration of that one fact.

The behaviour exists as a side effect of how these models are served rather than as a property anyone chose to give them. A model runs inside a serving stack: a long-lived process that holds the weights in GPU memory, accepts requests over HTTP, and runs the arithmetic that turns your tokens into scores. The individual programs that arithmetic runs as on the GPU are called kernels, and a kernel is allowed to break one large sum into pieces, add the pieces in parallel across thousands of threads, and combine them at the end, because that is the only way to keep the hardware busy. The serving stack is also allowed to process many customers' requests in a single pass, for the same economic reason. Both freedoms are what make a million tokens cost cents instead of dollars, and both mean the same numbers can be added up in a different order on two different calls.

Temperature 0 deletes the sampler#

Topic 1.6, which covers the sampling step, established the sequence: logits go through a softmax, temperature divides the logits before that softmax, and a pseudo-random draw picks a token from the resulting distribution. A logit is the raw score the model produces for one candidate token — one number per entry in its vocabulary, unbounded, frequently negative, and meaningful only by comparison with the other scores in the same vector. The softmax is what turns that whole vector into probabilities that add up to one, and temperature is a divisor applied to the logits on the way in, which stretches or flattens the gaps between them before the conversion.

Setting temperature to 0 does not make that draw less random. It makes the draw disappear. Dividing by zero is undefined, so every serving implementation special-cases it, and the special case is argmax: take the index of the largest logit and emit that token. There is no distribution, no draw, and no random number generator involved in the decision.

import numpy as np

def next_token(logits: np.ndarray, temperature: float, rng: np.random.Generator) -> int:
    if temperature == 0.0:
        return int(np.argmax(logits))          # no RNG touched at all
    probs = softmax(logits / temperature)
    return int(rng.choice(len(probs), p=probs))

Read the greedy branch carefully. rng is a parameter of the function and the greedy path never reads it. Whatever you pass as a seed — 42, the request ID, the timestamp — flows into an object that is not consulted. That is the mechanical reason the "temperature 0 isn't really 0" explanation is wrong, and it matters practically rather than pedantically: the explanation sends people looking for a parameter to tighten, and there is no parameter. The decoding algorithm is already fully deterministic given its inputs.

Which means the inputs are not identical. argmax is a deterministic function of the logit vector; if two runs disagree about which index is largest, the two logit vectors differed. The question is how a model with frozen weights, given identical token IDs, produces two different vectors of 150,000-odd floats.

Addition on a GPU is not associative#

Real numbers obey (a + b) + c == a + (b + c). Floating-point numbers do not, because every intermediate result is rounded to the nearest representable value, and rounding depends on the magnitudes being combined. The classic demonstration takes three lines:

>>> (1e16 + 1.0) + -1e16
0.0
>>> 1e16 + (1.0 + -1e16)
1.0

Adding 1.0 to 10^16 changes nothing, because the gap between adjacent float64 values at that magnitude is 2.0 — the 1.0 is rounded away and lost. Reassociate the same three numbers and it survives.

Why the gap is 2.0 there rather than something smaller is the whole of the mechanism, and it is one property of the format. A floating-point number stores a fixed count of significant digits together with a separate exponent saying where the decimal point sits, which means the precision is fixed in relative terms and not in absolute ones. Near one, the smallest step the format can take is minute. At a magnitude of ten to the sixteenth, the same number of significant digits has all been spent before reaching the units column, so the smallest representable step there is larger than one — and any addition smaller than that step has nowhere to be recorded. It is not that the 1.0 was approximately lost. There is no representable value between the result and the value the machine returned, so the correct answer, rounded to the nearest thing the format can hold, is the number you started with.

Read the two lines with that in mind and the ordering does all the work. In the first, the tiny quantity meets the enormous one while the enormous one is still present, and it is annihilated; the subtraction that would have brought the magnitudes back down happens afterwards, too late. In the second, the tiny quantity meets a number of comparable size first, so the addition is representable and the result survives being combined with the large value. The error here is total, and the general shape holds at every magnitude: the order in which you accumulate a sum determines which low bits get rounded off, and different orders give different answers.

The general case is gentler than the demonstration and the demonstration is not misleading about it. Real logit computations combine numbers of broadly similar size, so nothing is annihilated outright and the discrepancy between two orders is a handful of low bits rather than an entire term. What carries across is that the discrepancy is not zero and is not a bug: both orders produce the correctly rounded answer for the sequence of operations they performed, and the sequences differed.

A transformer's forward pass — the single sweep through every layer that turns the tokens so far into one vector of scores over the vocabulary — is built almost entirely from sums of exactly this kind. A single logit is a dot product between a hidden state and a row of the output projection; the hidden state is the model's internal vector representing the position being predicted, the output projection is the weight matrix that turns such a vector into one score per vocabulary entry, and a dot product is the operation that multiplies two equally long lists position by position and adds the products. For a model with a 4,096-dimensional hidden state that is a sum of 4,096 products, and there are hundreds of such reductions per layer before you even reach the final one.

GPU kernels do not accumulate these left-to-right. They split the sum across threads, accumulate partial sums in parallel, and combine the partials in a tree. The split point is a function of how the kernel was configured for that launch, which is a function of the tensor shapes it was given. The reason they work this way rather than in order is that a left-to-right sum of 4,096 terms has 4,096 steps that must happen one after another, since each needs the running total from the step before it. A card with thousands of threads would run all but one of them idle. Splitting the sum lets every thread work at once and costs only a final combining step, which is the difference between using the hardware and owning it.

So two runs that use different splits produce logits that differ in the last few bits. The typical magnitude is small. The paper On the Structure of Floating-Point Noise in Batch-Invariant GPU Matrix Multiplication quantifies this for matrix multiplication specifically, and the useful summary for an engineer is that the perturbation lives around the relative precision of the accumulation format — on the order of 10^-7 relative error for float32 accumulation, and worse for bfloat16 inputs. Against logits that typically span a range of 20 or 30, an absolute wobble of a few times 10^-5 is nothing.

It is nothing right up until two candidate tokens are nearly tied. Consider a step where the model is choosing between the and a:

Every logit moved by about the same tiny amount. The third candidate is 3.3 logits behind and its wobble is irrelevant. But the top two were separated by 2 × 10⁻⁶, and the noise was twice that, so argmax returns the in run A and a in run B. From that token onward the two runs are decoding different sequences, and they will typically diverge further rather than reconverging, because each run now conditions on a different prefix.

That last clause is where the disproportion lives, and it is worth stating plainly because it is the thing readers most often get backwards. The natural expectation is that a tiny numerical difference produces a tiny textual difference — a shifted decimal somewhere, a word in a different place. What happens instead is that the difference passes through argmax, which is a discrete function: it does not return a slightly different token, it returns a different token or the same one. Below the gap between the top two candidates, any amount of noise produces identical output; above it, the output is a different word. And from the next step onward the model is not scoring continuations of the same text any more, because the text now differs. The two runs have become two different questions, and nothing in the loop pulls them back together. An error of a few parts in ten million therefore produces either nothing at all or two completely different paragraphs, with very little in between.

Near-ties are not rare. They are the normal state of affairs at any position where the text could reasonably continue several ways — after a comma, at the start of a sentence, on a filler word, on a list item that could be phrased two ways. What makes the behaviour confusing is that the divergence is bursty: a run of hundreds of confidently-decided tokens where every logit gap dwarfs the noise, and then one coin-flip position where it does not.

You can look at the gap directly rather than taking this on faith. Most APIs will return the top few log-probabilities per position if you ask — the logarithm of the probability the model assigned to each candidate token at that step, which is what the softmax produces — and a provider that returns log-probabilities is telling you the post-softmax values, so a gap of 10⁻⁶ in log-probability space corresponds to roughly the same order of magnitude in logit space for the top candidates. Request the top five at each step for a generation you have seen diverge, find the position where the two runs part company, and read the gap between the first and second candidate there. In practice you will find a handful of positions in any few-hundred-token generation where that gap is under 10⁻⁴, and the rest where it is measured in whole units.

The size of the noise also depends on how the numbers were stored on the way through. Contemporary serving stacks hold weights and activations in bfloat16, which has 8 bits of mantissa and a relative precision around 4 × 10⁻³ — far too coarse for a stable result if you accumulated in it. So the accumulator is float32, with about 6 × 10⁻⁸ relative precision, and the error you actually see comes from the accumulation order in that wider format rather than from the storage format.

Two formats for one calculation looks like an inconsistency and is a deliberate split, because the two roles are constrained by different things. The storage format decides how many bytes have to travel from memory to the compute units, and that traffic is what generation speed is limited by, so it is chosen as small as quality allows. The accumulator format decides how much precision survives adding thousands of terms together, and it costs nothing in bandwidth because the running total lives in a register rather than in memory. Storing narrow and accumulating wide takes the cheap option on the expensive axis and the expensive option on the free one. What it also means is that the relative precision figure to reason about is the accumulator's, which is why the wobble in the table above is measured in parts per ten million rather than in the parts per thousand bfloat16 would suggest.

Longer reductions accumulate more of it: a 4,096-element dot product has more opportunities to round than a 1,024-element one, and a model attending over 30,000 tokens of context is reducing over a much longer axis than one attending over 300. The direction of that follows from the mechanism rather than from measurement — every addition is one more chance for the running total to be rounded, so a longer sum has more roundings in it, and two different orderings of a longer sum have more places to disagree. The practical reading is that a long-context request is more exposed to this than a short one on the same model and the same hardware.

The clearest published measurement of this comes from Thinking Machines' Defeating Nondeterminism in LLM Inference. They sent the same prompt 1,000 times at temperature 0 to a large open-weight model on a standard serving stack and got 80 distinct completions, with the most common appearing 78 times. The first 102 tokens were identical across all 1,000 runs. Then position 103 was a near-tie, and the outputs fanned out.

Those four numbers are worth reading together rather than as a headline, because between them they rule out most of the explanations somebody will offer you. A hundred and two identical tokens across a thousand runs is not a system behaving randomly — it is a system behaving with perfect repeatability up to a specific position, which is what you would expect if the arithmetic is deterministic except where a decision is too close to call. Eighty distinct outcomes from one such position means the fan-out did not stop there either: each branch went on to meet near-ties of its own, and the branches multiplied. And the most common completion appearing seventy-eight times out of a thousand says no branch is the "real" answer that the others are deviations from. There is no privileged output being corrupted. There is a tree, and each run walks one path down it.

Measuring it on your own stack#

The claim "temperature 0 is not reproducible" is worth exactly nothing to you as received wisdom, and quite a lot as a number you measured yourself. Measure it before you reason about it: the experiment is twenty minutes of work.

import hashlib, collections, json, httpx

PROMPT = "Summarise the refund policy in exactly two sentences."
N = 20

client = httpx.Client(base_url="https://api.example-provider.com", timeout=60)
outputs = []

for i in range(N):
    r = client.post("/v1/chat/completions", json={
        "model": "some-model-v1",
        "temperature": 0,
        "seed": 42,                       # fixed; see chapter 3
        "max_tokens": 200,
        "messages": [{"role": "user", "content": PROMPT}],
    })
    body = r.json()
    outputs.append({
        "run": i,
        "text": body["choices"][0]["message"]["content"],
        "fingerprint": body.get("system_fingerprint"),
        "model": body["model"],
    })

counts = collections.Counter(o["text"] for o in outputs)
print(f"{len(counts)} distinct outputs across {N} runs")
for text, n in counts.most_common():
    print(f"  {n:2d}×  {hashlib.sha256(text.encode()).hexdigest()[:12]}  {text[:60]!r}")

json.dump(outputs, open("runs.json", "w"), indent=2)

Three details in that script matter more than they look. The seed is fixed even though chapter 3 will show it is inert here, because you want to have ruled it out rather than assumed it. The system_fingerprint and model fields are captured on every response, because if those change mid-experiment your twenty runs did not all hit the same build and the whole result is void — chapter 6 is about exactly that field. And the raw outputs are written to disk rather than only counted, because "three distinct outputs" is a weaker observation than the three actual strings, and you will want to look at where they part company.

Run this and you get one of two results. Twenty identical strings tells you your prompt is not passing through any near-ties on this model at this length, which is common for short, highly-constrained outputs and is a fact about that prompt rather than a property of the system. Anything less than twenty tells you the effect is live on your stack, and gives you a divergence rate you can quote when someone proposes an exact-match assertion in CI.

The explanations that do not survive the measurement#

The divergence is not residual randomness in the sampler, which is the explanation almost everyone reaches for and the reason so much time gets wasted. Nothing in the decode loop draws a random number when temperature is 0.

It is not the tokenizer. Encoding is a deterministic table-driven merge process on the CPU with no floating-point arithmetic in it at all; the same string always produces the same token IDs.

It is not the model being "updated under you", though that is a real failure mode with the same symptom, and chapter 6 of this topic comes back to it. The measurement above holds the model constant and still sees divergence, so a weight change cannot be the explanation for what you observe in a single afternoon.

And it is not a bug a well-run provider would have fixed. Batch-variant reduction order is a direct consequence of the optimisations that make serving economical. A vendor could give you bit-exact determinism and would charge you for it in throughput. Bit-exact reproducibility is purchasable rather than absent, and chapter 3 puts a price on it that runs from 1.6× to 30× the compute.

The most durable near-miss explanation, and the one you will meet in older forum threads, is that GPUs are concurrent and concurrency is nondeterministic — that some kernel is using atomic adds, which land in whatever order the scheduler happens to produce, and that this randomises the low bits. It is a reasonable guess and it is mostly wrong for the forward pass of a transformer. Atomic accumulation into a shared output does appear in some kernels, and where it does it genuinely is run-to-run nondeterministic, but the standard matrix-multiply and attention kernels in a serving stack do not work that way: given the same inputs and the same launch configuration they produce bit-identical output every time, because their reduction tree is fixed at launch. The variation does not come from threads racing. It comes from the launch configuration itself being chosen differently on different requests, which is a much more tractable problem and points at a much more specific culprit.

Where you meet all this is a short and specific list. A response cache whose hit rate is lower than the prompt distribution says it should be. An exact-match assertion in CI that fails roughly once a week and passes when re-run, which chapter 5 takes apart. A quality score that moves three tenths of a point over a weekend with no deploy behind it, which is chapter 6. A customer report of a bad answer that nobody on the team can reproduce, which is chapter 7. And, most expensively, two runs of one request that ask your code to perform the same action with different arguments, so that one of them refunds £41.50 and the other £41.05.

The mechanism under all of them is the one this chapter measured: greedy selection is exact, and the numbers it selects between are not. What the experiment deliberately does not tell you is why any particular pair of runs diverged, because it varies nothing on purpose. Twenty requests to a production endpoint differ in dozens of uncontrolled ways: which replica served them, what else was in flight at the time, whether the process had been restarted onto a different GPU. The obvious next question is what changes the reduction order in practice, and the answer is something you would never have guessed was an input to your request at all.