Senior · 7 chapters
Non-determinism in practice
Batching effects, why temperature=0 is usually not reproducible
If this is skipped, exact-match tests become flaky and incident reports lose the metadata needed to separate model changes from serving variance.
Read these first
What you will know
Run the same zero-temperature request at least 20 times with a fixed prompt, seed where supported, and recorded token outputs.
A tiny numerical change can alter the top-token decision even when sampling has been disabled.
Reproduce output divergence by changing batch membership while holding the request constant, then repeat with a second non-sampling variable such as device, kernel, or replica.
This distinguishes serving-path effects from randomness in the decoding algorithm.
Measure token agreement, the position of the first divergence, and throughput across batched, serialized, and deterministic execution modes.
The practical choice is usually a throughput and latency cost rather than a free reproducibility switch.
Write a test that tolerates it instead of flaking on it
Assert on properties and bounded variation, or record and replay a fixed response when exact text is genuinely required.
Reproduce a reported discrepancy from a captured prompt, model revision, parameters, seed, timestamp, and serving context, then classify the likely source of variation.
Without request and serving metadata, a one-off output cannot be meaningfully compared with a later run.
Chapters
- 1
Greedy decoding and floating-point noise
Temperature 0 removes the sampling step entirely, so selection is greedy and nothing in the decoding algorithm draws a random number. Send the identical request twenty times with a fixed seed anyway and the outputs will not all match, because floating-point addition is not associative and a shift in reduction order can flip a near-tie between the top two logits. This chapter traces the mechanism from the argmax down to the accumulator, and gives the twenty-minute experiment that measures it on your own stack.
- 2
Continuous batching and batch variance
A serving stack assembles whichever requests are ready into one forward pass, so the tensor your request is computed inside is a different shape on every decoding step. Kernels choose how to split their sums from those shapes, which makes reduction order — and therefore your output — a function of who else is calling at that moment. This chapter explains the mechanism, gives the controlled experiment that isolates it, and names the second variable you need before the result is a claim rather than an anecdote.
- 3
Seeds and batch-invariant kernels
A seed pins the pseudo-random draw a sampler makes, and at temperature 0 there is no draw to make, so setting it changes nothing about batch-driven divergence — the seed is not broken, it is inert. The controls that do work change reduction order itself: batch-size-one serving, batch-invariant kernels, and a pinned image, driver, accelerator and parallelism configuration. All three require running the model yourself, and determinism is bought in throughput at roughly 1.6× to 30×.
- 4
The first-divergence index
\"Different\" is not a measurement. The index of the first differing token is exactly defined, cheap to compute, and unlike a similarity score it partitions the suspects: an early divergence implicates the assembled prompt or the model revision, a late one implicates batch variance. This chapter defines the metric, works out how many runs a rate estimate needs, adapts the comparison for structured outputs, and prices batched, serialised and batch-invariant execution over the same set.
- 5
Regression tests on model output
An exact-match assertion on generated text fails roughly weekly and wrapping it in three retries is the wrong repair, because a partial regression that still emits the old string on one attempt now passes silently. This chapter works out the detection arithmetic, then covers the three assertions that survive non-determinism — properties and invariants, bounded variation, and recorded responses replayed from disk — and where a live model call still belongs once it is out of the merge path.
- 6
Model aliases and build identifiers
\"The model changed\" is only a diagnosis you can make if you captured a field that changes when the model does. The friendly name in your request body is usually an alias that a provider can repoint at a new build without your involvement, while the response carries the resolved build and a backend-configuration identifier. This chapter separates name from build from serving stack, logs the two that leave a fingerprint, and shows the query that turns a Friday-morning argument into three outcomes.
- 7
The prompt capture record
The string that reaches the model is assembled at request time from a system template, retrieved context, tool results, truncated history and interpolated runtime facts, all with independent lifecycles. Logging the user's question and the response captures none of them, so a report from last Tuesday cannot be replayed. This chapter specifies a capture record, defends its three storage decisions, and gives the classification procedure that separates a model change from a changed prompt from serving variance.
External resources
- Defeating Nondeterminism in LLM Inferencethinkingmachines.ai
The primary source: batch-size-dependent kernels, not just float non-associativity.
- On the Structure of Floating-Point Noise in Batch-Invariant GPU Matrix Multiplicationarxiv.org
Quantifies the numerical noise so you can reason about acceptable output drift.
- Setting the temperature to zero will make an LLM deterministic?zansara.dev
Short, experiment-backed answer to the temperature=0 assumption you probably hold.
- The Temperature=0 Myth: Why Your LLM Still Isn't Deterministicreliableai.substack.com
Frames what non-reproducibility means for caching, tests and eval pipelines.
- Determinism in the AI Tech Stack (LLMs): Temperature, Seeds, and Toolsyoutube.com
Specific mechanics on temperature, seeds, and tooling