Continuous batching and batch variance

The previous chapter established that a shift in reduction order can flip a near-tie between the top two logits and send two runs of one request down different continuations. What it left open is what changes the reduction order in practice, on a service where nothing about the request has changed at all. The answer is that your request is not computed on its own. It is computed as one row of a matrix whose other rows belong to whoever else was calling at that instant.

That sounds as though it should be impossible, since the requests are logically independent — nothing about somebody else's input should inform the answer to yours. And logically it does not. The mechanism is narrower and stranger than data leakage: the presence of other requests changes the shape of the tensors your request is computed inside, and the shape changes how the kernel splits its sums.

One forward pass serves many sequences#

A serving stack that ran one request at a time on an 80 GB accelerator would waste almost all of it. Generating one token requires reading every weight in the model out of memory and doing comparatively little arithmetic with each one, so a batch of one is bound by memory bandwidth and leaves the compute units mostly idle. Batching amortises that read: load the weights once, and multiply them against 64 hidden states instead of 1. The arithmetic is 64× but the weight traffic is unchanged, so throughput rises by something close to an order of magnitude at a modest latency cost.

Follow that argument one step further and the qualifier at the end of it stops being a hedge. If a step's duration is set by moving the weights, and the weights moved are identical however many sequences are riding on them, the step takes about the same time whether it produces one token or 64. Sixty-four tokens in the time one used to take would be a sixty-fourfold improvement, and the reason the real figure is closer to an order of magnitude is that the argument stops holding partway up. Somewhere on the way to 64 the arithmetic, which does scale with the batch, grows large enough to become the thing you are waiting for instead of the memory traffic. Past that point each extra sequence costs real time and the curve flattens. The latency cost is the same fact seen from one sequence's point of view: it is now waiting for a step that is doing everyone else's work as well as its own.

Modern servers do this continuously rather than in fixed rounds. A scheduler maintains a set of in-flight sequences, and on every decoding step it assembles whichever sequences are ready into a single batch, runs one forward pass, and scatters the resulting logits back to their owners. Sequences join when they arrive and leave when they emit a stop token, so the batch is a different size and a different composition on essentially every step. The arrangement is called continuous batching, in contrast with the older static scheme where a batch was assembled once and every sequence in it had to finish before the next batch could start.

Every matrix multiply in that forward pass now has a batch dimension. Your hidden state is one row of a matrix that is 37 rows tall this step and might be 4 rows tall on the next. Mathematically each output row depends only on its own input row, and the model is not mixing your data with anyone else's. The kernel, though, does not see 37 independent problems. It sees one matrix multiply with M = 37, and it chooses how to decompose the work — how many thread blocks, which are groups of threads scheduled together onto one compute unit; how large a tile of the output each block takes; whether to split the reduction dimension, meaning the axis being summed over, across multiple blocks and combine partial results afterwards — based on those dimensions and on how much of the hardware it can keep busy.

That last choice is the one that reaches your output. When the batch is small there is not enough parallel work to fill the device by tiling over rows and columns alone, so the kernel splits the reduction dimension too: block 0 accumulates elements 0–1023 of the dot product, block 1 accumulates 1024–2047, and their partial sums are added at the end. When the batch is large there is plenty of work, so the kernel keeps each dot product whole inside one thread block and accumulates straight through. Both compute the same mathematical quantity, and the previous chapter's point about associativity says they will not compute the same float.

The arithmetic behind that choice is worth making concrete, and it is small enough to do in full. Take the output projection of an 8B model: a hidden state of 4,096 values multiplied by a weight matrix of 4,096 × 152,064. With a batch of 1, the natural decomposition assigns each thread block a tile of output columns — a tile being the rectangle of the output one block is made responsible for — and has it walk the full 4,096-element reduction. On an accelerator with 132 streaming multiprocessors — the independent compute units the chip schedules blocks onto — and a tile width of 128 columns, that gives 1,188 blocks, enough to fill the device, so no split is needed.

Each number in that sentence comes from the one before it. There are 152,064 output columns to cover and each block covers 128 of them, so the count of blocks is 152,064 divided by 128, which is the 1,188. The device has 132 places to run a block, and 1,188 is nearly nine times that, so every compute unit has work queued and the kernel has no reason to look for more parallelism. Whether the device is full is the only question the kernel is asking, and here the answer arrives from the output width alone.

Now consider a mid-layer projection with a much narrower output, where the same tiling yields 32 blocks for a batch of 1. Three quarters of the hardware would sit idle, so the kernel splits the 4,096-element reduction across four blocks each handling 1,024 elements, and adds the four partial sums at the end. That factor of four is not arbitrary either: 32 blocks against 132 compute units leaves most of the chip with nothing to do, and cutting each block's reduction into four pieces multiplies the block count by four, to 128 — which is as close to filling 132 units as this decomposition can get. The kernel reached for the reduction axis because it had run out of output columns to divide.

Raise the batch to 64 and there is suddenly 64× more row-parallelism available, the device fills without splitting, and the same dot product accumulates straight through in one pass. Same inputs, same mathematics, two different summation orders. The chain that produced the difference is worth reading backwards, because it is the chain the whole chapter rests on: strangers arrived → the batch dimension changed → the amount of parallel work available changed → the kernel's choice of whether to split the reduction changed → the order the additions happened in changed → the low bits of every logit changed.

Attention has its own version of this, and it is the harder half to fix. The reduction there runs over the KV cache — the keys and values the model computed for every token already in the sequence and keeps around so it does not have to recompute them at each step — rather than over a weight matrix, so its length grows with the sequence and its decomposition depends on how the cache is chunked, on whether the request is in its prefill phase, where the whole prompt is pushed through at once, or its decode phase, where tokens come out one at a time, and — under chunked prefill, where a long prompt is processed in pieces interleaved with other sequences' decode steps — on where the chunk boundaries happened to fall. Two runs of the same 3,000-token prompt can be split into chunks differently depending on what else the scheduler was juggling, which changes the accumulation order in attention even before the batch dimension is considered.

Chunked prefill is the part of this most worth understanding, because it is the one that makes an input you fully control behave as though you did not. The scheduler has a budget for how much work it will put into a single forward pass, and a long prompt exceeds it, so the prompt is cut into pieces and the pieces are processed across several passes with other sequences' decode steps interleaved between them. Where the cuts fall depends on how much of the budget the other sequences were already consuming when each pass was assembled — which is to say, on arrivals you cannot see. Different cuts mean the attention reduction is broken into different segments and its partial results combined in a different order. Your prompt was byte-identical on both runs, and it was nevertheless processed as a different sequence of operations.

Your logits therefore depend on the batch size, and the batch size depends on strangers. Under a continuous-batching scheduler your request will sit in a batch of 4 for its first few tokens, a batch of 51 in the middle, and a batch of 12 at the end, with the boundaries decided by arrival times you have no visibility into. Kernels that behave this way are called batch-variant, and batch variance rather than raw float error is the primary source of run-to-run divergence in production serving — the central argument of Defeating Nondeterminism in LLM Inference, and the reason a fix has to change kernels rather than change flags.

Reproducing it on purpose#

The experiment is to hold the request byte-for-byte constant and vary only what surrounds it. That means serving the model yourself, because a hosted endpoint gives you no control over the batch, and running two conditions:

import asyncio, collections, httpx

TARGET = {
    "model": "qwen3-8b",
    "temperature": 0,
    "seed": 42,
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "Explain a write-ahead log."}],
}
FILLER = dict(TARGET, messages=[{"role": "user", "content": "Count from 1 to 400."}])

async def one_trial(client, n_filler: int) -> str:
    """Fire the target request with n_filler concurrent neighbours."""
    tasks = [client.post("/v1/chat/completions", json=TARGET)]
    tasks += [client.post("/v1/chat/completions", json=FILLER) for _ in range(n_filler)]
    responses = await asyncio.gather(*tasks)
    return responses[0].json()["choices"][0]["message"]["content"]

async def main():
    async with httpx.AsyncClient(base_url="http://localhost:8000", timeout=300) as c:
        for n in (0, 63):
            seen = collections.Counter(
                [await one_trial(c, n) for _ in range(20)]
            )
            print(f"filler={n:2d}  distinct={len(seen)}  top={seen.most_common(1)[0][1]}/20")

asyncio.run(main())

Two things about that script are load-bearing. The target request body is a single module-level constant reused across every trial, so there is no chance of a stray whitespace difference between conditions. And the filler request asks for a long, boring generation, because a filler that finishes in five tokens leaves the batch almost immediately and you end up measuring nothing.

On an 8B model with a 4,096-dimensional hidden state, a typical result looks like the table below. The exact numbers depend on the model, the kernel library and the prompt, and yours will not match these; the shape is what transfers.

The first row is the one people find surprising: served strictly alone, on a warm process, on one GPU, the same request really does reproduce. Determinism is available; it is not what a shared endpoint sells you. The second row is the effect under test. The third row is the control described next.

The second variable, and why you need one#

A single experiment that varies batch membership and observes divergence is suggestive rather than conclusive, because concurrency changes several things at once. Adding 63 requests also raises memory pressure, changes which KV-cache blocks get allocated where, and may push the scheduler into preempting and recomputing a sequence. To claim that the serving path rather than the decoding algorithm is responsible, vary a second thing that has nothing to do with sampling and check that it produces the same class of effect.

The reason a second arm is worth the extra afternoon is that the first one on its own is compatible with several stories. Divergence under load is what you would see if reduction order depended on batch composition, and it is also what you would see if the scheduler were preempting sequences and recomputing them slightly differently, or if memory pressure were pushing the stack down some other path. All three predict the same observation, so the observation discriminates between none of them. A second variable that shares only the reduction-order pathway — a different tensor-parallel degree, a different kernel build — breaks the tie: it changes nothing about load, preemption or memory pressure, so if it produces the same bursty divergence then reduction order is the thing the two arms have in common. One arm supports a plausible story; two arms support a claim, and a claim is what an incident review asks for.

Three good candidates, in increasing order of how much infrastructure they require. Replica: pin the same request to two pods of the same deployment, one at a time, each otherwise idle, with the same weights, container image and flags — if their outputs diverge, the difference is hardware or scheduling state, since nothing about the request or the algorithm changed. Device: run the same container on two different accelerator models, or on the same model with a different tensor-parallel degree. Tensor parallelism — splitting each layer's weight matrices across several GPUs so that each one computes part of every result and the parts are then summed across devices in a collective operation called an all-reduce — is a particularly clean probe, because splitting a layer across 2 GPUs instead of 4 changes how many partial sums are combined in that all-reduce, so the reduction order changes by construction and the outputs typically diverge in exactly the same bursty way. Kernel version: hold the hardware fixed and change only the inference library or its attention backend. Bumping a serving framework by one minor version, or switching between two attention implementations, routinely changes the low bits of every activation in the model.

The replica arm is the cheapest to run and the easiest to get wrong. Sending twenty requests at a service address and hoping the load balancer spreads them is not a controlled experiment, because you do not know how they were distributed and you have reintroduced the concurrency you were trying to remove. Port-forward to one pod, run the twenty requests against it while nothing else is pointed at it, record the pod name alongside the outputs, then repeat against a second pod. If you cannot take a pod out of rotation, the next best thing is to include the serving pod's identity in the response — many gateways will echo an upstream hostname header — so that at least you can partition the results after the fact rather than averaging over an unknown mixture.

Running any two of the first four is enough. What you have then is not an anecdote but a claim with a control: the output moves when non-sampling parts of the serving path move, and stays put when they are held still.

What is not going on#

Nothing is leaking between requests. Each row of the batch reads only its own hidden state and its own KV cache, and a bug that mixed them would produce garbled text rather than one flipped near-tie 88 tokens in. The guarantee is structural rather than a matter of care: in a matrix multiply, the value at output row i is computed from input row i and the shared weights, and no term in it involves row j. Batching stacks independent problems into one tensor for the hardware's convenience and does not connect them. What the batch dimension does reach is the kernel's decision about how to organise the work, and that decision applies to every row at once — which is how another request can affect your low bits without any of its data being anywhere near your arithmetic. If you are seeing another tenant's content in your output, you have a much more serious problem than this chapter describes.

The scheduler is not truncating you under load either. Load-shedding, timeouts and max_tokens cutoffs all produce visibly incomplete output with a stop reason that says so; the divergence here produces complete, fluent, plausible alternative outputs.

And the effect is not proportional to load in any usable way. There is no threshold of concurrent requests below which you are safe. A batch of 2 and a batch of 3 can select different kernel configurations just as a batch of 1 and a batch of 64 can, so "we only see it at peak" describes when near-ties happen to get flipped, not when the mechanism switches on. The classification service above saw its complaints cluster in the morning peak for exactly that reason.

The distinction there is between a mechanism and an observation rate, and keeping them apart is what stops a team drawing the wrong safety line. The mechanism is armed at every batch size, because any two different batch sizes can select different decompositions. What the morning peak changes is how many requests pass through it, and how many distinct batch compositions each of them meets on its way through — more requests, more chances that some request's near-tie is decided one way in the morning and the other way at midnight. A service that sees no divergence overnight has not switched the mechanism off. It has taken a smaller sample.

What the mechanism gives you, in exchange for the confusion, is a diagnosis with a shape. Divergence caused by batch variance appears late in a generation rather than at its start, appears at the same position across many pairs of runs, and disappears entirely when the request is served alone on a pinned replica. A team that has checked those three signatures can say which of its unstable outputs are the serving path and which are something else.

The natural response at this point is to reach for the control that exists specifically for making generation repeatable. Providers expose a seed parameter, the documentation says it makes output reproducible, and the request bodies in both experiments so far have been quietly setting it to 42 without comment. It is worth being precise about what that parameter does, because it does something real — just not here.