Seeds and batch-invariant kernels

Providers expose a seed parameter and document it as the control for reproducible output, which makes it the first thing anyone reaches for after measuring the divergence of the previous two chapters. It is a real control over a real mechanism. The mechanism is the sampler's random draw, and at temperature 0 that draw does not happen, so the parameter is inert in the strict sense — not weakened, not partially effective, but accepted, validated and handed to code that never runs.

This chapter separates the parameter from the controls that would actually help. The first half is what a seed pins and where it stops, including at temperatures above zero where it is necessary and still insufficient. The second half is the three ways to fix reduction order itself, what each costs in throughput, and why every one of them requires being the party that runs the model. The chapter ends on what a deterministic deployment still does not give you, which is more than people expect.

What a seed pins#

A sampler at temperature above 0 needs a random number. It builds a probability distribution over the vocabulary, draws a uniform value in [0, 1), and walks the cumulative distribution until it passes that value. The uniform value comes from a pseudo-random generator, which is a deterministic function from an internal state to a number plus a new state. Seeding sets that initial state, so a seeded generator produces a fixed, known sequence of uniforms — the first draw is always 0.6394, the second always 0.0250, and so on for that seed.

Nothing in that generator is unpredictable, and the word "random" describes the statistical shape of the sequence rather than any ignorance about what comes next. Given the state, the next value is computed; given the seed, the whole sequence is fixed before a single token is generated. That is precisely what makes seeding useful, and it is also the boundary of what seeding can do: it determines which uniform value arrives at each step, and it has no opinion about the distribution that value is being used to index into.

Why walking a cumulative distribution produces the right frequencies is worth one sentence, because the rest of this section turns on it. Laying the tokens out end to end along the interval from zero to one, each occupying a stretch as wide as its probability, uses up the whole interval exactly, since the probabilities sum to one. A uniform draw has no preference for one part of that line over another, so it lands inside a given token's stretch as often as the stretch is wide — which is the token's probability. The walk through the cumulative sums is simply the search for which stretch the number fell in.

import numpy as np

def sample(logits, temperature, rng):
    probs = softmax(logits / temperature)
    return int(np.searchsorted(np.cumsum(probs), rng.random()))

logits_a = np.array([2.30000, 2.29999, 1.10])   # run A
logits_b = np.array([2.29999, 2.30000, 1.10])   # run B: last-bit difference

print(sample(logits_a, 0.7, np.random.default_rng(42)))  # 0
print(sample(logits_b, 0.7, np.random.default_rng(42)))  # 1

Both calls consume the identical uniform draw, because both generators were seeded with 42. They return different tokens anyway, because the cumulative distribution they are indexing into was built from logits that differ in the sixth decimal place, and the draw landed inside the sliver where the two boundaries crossed.

Picture the two lines side by side and the failure is obvious. In run A the first token's stretch begins at zero and ends somewhere near the middle, and the second token's stretch runs from there to nearly the end. In run B the two logits have swapped in their last decimal place, so the boundary between those two stretches has moved by an amount as small as the difference between the logits — a sliver. Everywhere else on the line the two runs agree about which token owns which stretch. The seed made certain that both runs draw the same uniform value, and that value happened to land inside the sliver, which is the one region where the two lines disagree about the answer. A fixed seed does not reduce the chance of that; it fixes where on the line you always look, and if that place is a sliver you will find it every time.

That is the honest description of what a seed buys you at temperature 0.7: it removes one source of variation and leaves the arithmetic source completely intact. Seeding is necessary for reproducible sampling and nowhere near sufficient.

At temperature 0 the picture is simpler still, because the function above is never called. The greedy branch of chapter 1's next_token returns without touching rng, so there is no draw to pin. The practical consequence is that a seed on a temperature-0 request is a comment: harmless to leave in, useful as documentation that you considered it, and worthless as a control.

Worth noticing, too, that the parameter is not universal. Anthropic's Messages API has never exposed a seed at all, and its most recent models go further by rejecting temperature, top_p and top_k outright with a 400 — the sampling surface is gone, not merely defaulted. A codebase that has "set the seed" as its reproducibility story will discover this the hard way at migration time. Whatever you plan to do about non-determinism has to survive a provider that offers no sampling knobs whatsoever.

The controls that do something#

If the variation comes from reduction order, the fix has to change reduction order, and there are three ways to do that. All of them require you to be the one running the model.

Serve at batch size one. The most direct reading of the previous chapter: if the batch never varies, the kernel never picks a different configuration for it. Force the scheduler to run a single sequence at a time and your request's arithmetic is identical on every run. This is trivially correct and economically ruinous — you have given up the amortisation that made serving affordable, and your cost per token goes up by roughly the batch factor you were previously enjoying. It is a legitimate choice for a small internal evaluation set and not one for a product.

Use batch-invariant kernels. The better answer, and the one Defeating Nondeterminism in LLM Inference argues for, is to make the kernel's reduction strategy independent of the batch dimension. Instead of letting the library pick a split based on how much work is available, you fix the split: the reduction over the hidden dimension is always decomposed the same way regardless of whether M is 1 or 512, so a given row's dot product accumulates in the same order every time. The results are still floating-point approximations with the same rounding error as before; they are the same approximation, every run, which is all reproducibility ever required.

The distinction in that last sentence is the one that makes the whole approach make sense, and it is easy to slide past. Accuracy is a claim about how close an answer is to the true value. Reproducibility is a claim about how close two answers are to each other. Batch-invariant kernels improve the second and leave the first exactly where it was — the logits are as wrong as they always were, and they are now wrong in the identical way on every run. That is why the price is paid in throughput rather than in quality, and why nobody should expect outputs to improve when the flag goes on.

The throughput is what it costs because fixing the split means declining to adapt. The default kernel chooses a decomposition that fills the device for the shape actually in front of it; a batch-invariant one commits to one decomposition and uses it whatever the shape turns out to be. At the batch sizes the fixed choice happens to suit, nothing is lost. At the others, the device runs less full than it could, and that gap is the bill.

Making that true across a whole model means auditing every operation that reduces over an axis, and there are three families of them. The normalisation layers reduce over the hidden dimension to compute a mean or a root-mean-square, and a naive implementation parallelises that reduction differently depending on how many rows it has to process. The matrix multiplies reduce over the contraction dimension, which is the case chapter 2 worked through. Attention reduces over the KV cache — the stored keys and values for the tokens already in the sequence — and its decomposition additionally depends on sequence length and chunking, which is why it is the last one to get fixed and the one whose deterministic version costs the most performance. A build that fixes the first two and leaves attention batch-variant will look reproducible on short prompts and stop being reproducible at length, which is a worse failure than not trying, because it passes the obvious test.

Recent versions of vLLM, one of the open-source serving stacks teams most often run a model on themselves, ship an opt-in mode that does this:

# throughput arm — default kernels, batch-variant
vllm serve Qwen/Qwen3-8B --port 8000

# deterministic arm — batch-invariant kernels, eager execution,
# and a pinned image so the kernels themselves cannot move underneath you
VLLM_BATCH_INVARIANT=1 \
vllm serve Qwen/Qwen3-8B --port 8001 --enforce-eager

The --enforce-eager flag in that second command turns off graph capture, the optimisation where the framework records the sequence of GPU operations once and replays it as a unit; a captured graph is free to fuse and reorder operations differently from the eager path, so leaving it on would reintroduce a source of variation the batch-invariant kernels have just removed.

Pin everything the kernels depend on. Batch invariance fixes the split for a given kernel build. It says nothing about a different kernel build, a different driver, a different GPU architecture, or a different tensor-parallel degree, all of which change the arithmetic for reasons that have nothing to do with the batch. Reproducibility across time therefore also means an immutable container digest rather than a floating tag, a pinned driver version on the node, a fixed accelerator model in the node selector, and a fixed parallelism configuration. Every one of those is a thing your platform team probably rolls forward automatically, which is why "we get bit-identical outputs" tends to be true on the day it is measured and false a month later.

What it costs#

The throughput number is the part people skip, and it is the part that decides whether any of this is available to you. The same source measured 1,000 completions of 1,000 tokens each from an 8B model on a single H100: roughly 26 seconds with default vLLM, roughly 55 seconds with an unoptimised batch-invariant build, and roughly 42 seconds once the attention kernel was tuned. So the honest price of determinism, with effort spent on it, is somewhere around 1.6× the wall-clock, and without that effort closer to 2×.

Both of those multipliers are one division against the same denominator. The tuned build's 42 seconds over the default's 26 is the 1.6, and the untuned build's 55 seconds over the same 26 is a little over two. The denominator is the figure to hold on to when quoting any of this, because a ratio is only a price if everyone agrees what it is a ratio to — and the default here is a production-grade batched serving stack rather than a naive baseline, which is what makes the number honest rather than flattering.

Batch-size-one serving is not on the same scale — it is worse by the batch factor, commonly 10× to 30×. That range is the previous chapter's amortisation argument read backwards. Batching bought a throughput multiple by spreading one sweep of the weights across many sequences; serialising gives every one of those multiples back, so the cost is roughly whatever batch size the deployment was achieving, bounded above by the point at which the amortisation had stopped paying anyway.

The sixth row is the one control on that list a hosted user actually has, and it is worth taking seriously even though it does not touch the mechanism. Divergence requires a near-tie, and near-ties are a property of the prompt as much as of the arithmetic. A request that asks for free-form prose has a near-tie every few dozen tokens because there are genuinely several equally good ways to say the next thing. A request constrained to a JSON schema, or to one of four category labels, or to a number, has almost none: at most positions exactly one continuation is legal and the logit gap is enormous.

A near-tie is not an accident of arithmetic, in other words — it is the model correctly reporting that two continuations are equally good. Free prose contains many such positions because English genuinely offers several equally acceptable next words after a comma or at the start of a clause, and a model that assigned a decisive winner there would be wrong about the language. Constrained output contains almost none because the format has already eliminated the alternatives, so one candidate leads by whole units and no plausible arithmetic perturbation reaches that far. Constraining the output does not make the arithmetic quieter; it removes the positions at which the arithmetic gets to decide anything.

Shortening the output helps for the same reason, since the chance of hitting at least one coin-flip position grows with length. That growth is the arithmetic of "at least one": a generation is stable only if every position stays stable, so the chance of a clean run is the per-position chance of stability multiplied by itself once for each token emitted. A number slightly below one, multiplied by itself enough times, falls away — which is why a long generation on a model with a low per-position divergence rate can still be unstable most of the time, and why quoting a per-token figure without the length it applies over says nothing usable. None of this makes the deployment deterministic — it lowers the rate at which the non-determinism becomes visible, which for a classification endpoint can be the difference between one flip in four hundred and none you ever observe. It is a mitigation with a real cost in what the endpoint can express, and it should be described that way rather than as a fix.

The right-hand column is the uncomfortable one. A hosted API gives you a model name and a request body. It does not give you the batch your request landed in, the kernel version that served it, the GPU it ran on, or any way to constrain them, and it would not be able to sell competitive pricing if it did. That is not a gap in anyone's documentation; it is the shape of the product. The constraint is stronger than a business preference, because the batch your request lands in is shared. Serving you a fixed batch composition means deciding what everyone else on that replica gets too, and running batch-invariant kernels for you means running them for every tenant on the hardware, at the throughput cost measured above. There is no per-request version of the setting to expose. Which means that if a comparison table in an incident report has a "deterministic" column, that column was produced on infrastructure you control, and the numbers in it do not transfer to your hosted production path.

The limits of a deterministic arm#

Batch-invariant serving gives you run-to-run reproducibility on one pinned stack. It does not give you agreement between two stacks. Two sets of machines with the same weights, the same kernels and different GPU generations will still disagree, because the hardware's fused-multiply-add behaviour and its reduction hardware differ. Anyone expecting a deterministic build to reproduce last quarter's outputs on this quarter's fleet is expecting something the technique does not offer.

It also does nothing about the model itself changing, which is a separate cause with an identical symptom and is the subject of chapter 6.

And determinism is not accuracy. A batch-invariant deployment reproduces its answers exactly, including its wrong ones. The value is diagnostic rather than qualitative: with a reproducible arm you can change one thing and attribute the difference to it, which is the precondition for evaluating a prompt change, bisecting a regression, or trusting a cache. Reinforcement-learning training setups care about this for a sharper reason — when the sampler that generates rollouts and the trainer that scores them compute slightly different logits for the same tokens, an algorithm you believed was on-policy — learning only from samples drawn by the very version of the model being updated — quietly is not.

The seed, then, is a control over sampling and reproducibility is a property of execution, and the two only meet at temperatures above zero where you need both. Everything that would give you the second is a change to how the model is run: the kernels, the batch policy, the image, the hardware. Each is purchasable, each has a price in throughput that can be quoted to a budget holder, and none is on sale from a hosted endpoint at any price.

So you now have up to three execution modes worth comparing: production batched serving, a serialised arm at batch size one, and a batch-invariant arm. Comparing them requires saying what "the same" means, and "the outputs were different" is not a measurement. Two outputs can differ in a trailing space, in one word out of four hundred, or from the third token onward, and those three observations point at completely different causes.