Low-temperature replay and rate estimation

A replay is a re-run of a frozen trace: the stored request sent again, to the same model, with the parameters it was sent with the first time. It is the first thing anybody does with a preserved bad output and the first place a single-trace investigation goes wrong, because "can we reproduce it" is two questions wearing one sentence. One asks whether your own stack, today, can turn the recorded bytes into a sane answer. The other asks how often, at the settings production actually runs at, this prompt produces the bad one.

The two need different replays, and confusing them is expensive in both directions: a clean run at temperature 0 gets written on a ticket as "cannot reproduce", and a single failing run gets treated as a rate. Both mistakes have one root. A replay answers a question only when it holds fixed everything the question is not about, and the two questions above are not about the same thing. The number that comes out of the second design also fixes the exchange rate for the rest of this topic, since it decides whether a single edited re-run counts as evidence at all.

What a low-temperature replay is for#

Between the run that produced the bad output and any run you make now, two families of things can differ. One is the pipeline: the rendered request bytes, the model snapshot behind the alias, the decoding parameters and the harness that assembles them. The other is the draw — which candidate the picking step took from the scores the model returned, a step that is allowed to be random and is random by default in production. An experiment that lets both vary produces an observation with two candidate explanations and no way to choose between them.

That is why the plumbing check runs with sampling turned off. Holding the draw fixed leaves the pipeline as the only remaining source of difference, so an answer to a question nobody asked is attributable to the reconstruction. At production sampling the same wrong-looking output means either that you rebuilt the wrong request or that this run landed in the tail of a correct one, and a single response separates neither, since a tail sample and a mis-rendered request both arrive as a model answering badly.

The first question, then, is whether your own stack can produce the recorded output from the recorded bytes. Until the answer is yes, every experiment you run afterwards is measuring your harness. This is where pinning belongs: same rendered request byte for byte, same resolved model snapshot rather than the alias, same top_p, same max_tokens, same tool definitions, a fixed seed where the provider supports one, and temperature at zero to remove sampling noise while you check the plumbing.

What you are looking for is not a perfect match of the original bad output — at temperature 0 you will usually get the clean answer, because the original was a tail sample. You are looking for the pinned replay to be stable across three runs and for the output to be a sane response to this prompt. If those three runs disagree with each other, or if the model answers as though it were given a different question, the request you reconstructed is not the request that was sent, and the correct next move is back to the rendering checks rather than forward into experiments.

Three runs, not one, because temperature 0 is not bitwise deterministic on any serious serving stack, and the platform team's claim that it is deserves a specific answer. Floating-point addition is not associative: every intermediate sum is rounded to a fixed number of bits, and which values were rounded together depends on the order they were added in, so one set of numbers summed two ways can land on two different totals. The reduction order inside a batched matrix multiply depends on how the request was batched with other traffic, on which CUDA kernel the autotuner selected, and on the tensor-parallel split. None of those is a property of your request, so your input can be identical while the order of the additions is not.

Mixture-of-experts models, which send each token through a small subset of the network chosen at runtime, add routing that can depend on the other sequences sharing the batch. The resulting logit differences are tiny, on the order of 1e-6, and they are irrelevant except at a near-tie between the top two tokens, where they flip the argmax. Greedy decoding is a comparison between the two leading scores, and a comparison only flips when the gap between them is smaller than the perturbation, so flips land exactly where the model was close to indifferent. One flipped token in the first sentence can send the rest of the generation somewhere else entirely, since every later step conditions on the text chosen so far, which is why the observed effect is bimodal: identical outputs most of the time, and occasionally a completely different answer.

Teams that measure this on hosted endpoints typically see a few percent of greedy replays differ. So "temperature 0 is deterministic, one replay is enough" is wrong as a fact about the hardware, and it would be beside the point even if it were true, because the production system does not run at temperature 0.

Reading a rate off thirty runs#

The second question is the one a ticket usually needs: at production settings, how often does this prompt produce the bad output? That is estimated by replaying the stored request twenty to thirty times at the production sampling settings, with the seed for each run recorded so that any interesting sample can be re-materialised on its own.

import concurrent.futures as cf

def has_duplicate_refund(resp) -> bool:
    ids = [c.input["order_id"] for c in resp.content
           if c.type == "tool_use" and c.name == "refund_order"]
    return len(ids) != len(set(ids))

def replay(seed: int):
    resp = client.messages.create(
        model=trace["model"]["resolved_snapshot"],   # pinned snapshot, not the alias
        system=trace["system"],
        messages=trace["messages"],
        tools=trace["tools"],
        temperature=0.8, top_p=0.95, max_tokens=1024,
        extra_body={"seed": seed},
    )
    return seed, has_duplicate_refund(resp), resp

with cf.ThreadPoolExecutor(max_workers=8) as pool:
    runs = list(pool.map(replay, range(1000, 1030)))

bad = [seed for seed, is_bad, _ in runs if is_bad]
print(f"{len(bad)}/{len(runs)} duplicated; seeds={bad}")

The classifier matters as much as the replay. Thirty outputs is thirty careful reads if you judge by eye, which is enough friction that people stop at five and call it clean. A deterministic predicate over the structured tool calls — a plain function that reads the response and returns true or false, with no model in the loop — costs six lines, runs in microseconds, and gives the same verdict every time, and it exists only because the failure is expressible as a property of the output rather than a matter of taste. When the failure mode is "the tone was wrong", this whole approach gets much more expensive, which is a good reason to start these investigations on failures with crisp predicates.

Suppose the run prints 1/30 duplicated; seeds=[1017]. One replay in thirty duplicated the refund, and the obvious guess at how often this prompt does that is the fraction that did, one divided by thirty, or 3.3%. That fraction is an estimate rather than a measurement, and the difference between those two words is the whole of the care needed here.

Write p for the probability that a single replay of this request produces the duplicate: unknown, because nobody reads a distribution off directly, and fixed, because the prompt, the weights and the sampling settings that determine it did not change between runs. Write n for the number of replays, thirty here, and k for how many came back bad, one here. The estimate is k divided by n, and the reason that particular fraction is the right single guess is an argument rather than a convention: among all the values p could take, the one that makes an outcome of exactly one bad run in thirty most likely is p equal to one in thirty, which is the only sense in which that fraction is the answer.

That argument needs the thirty runs to be independent draws from one distribution, a condition to check rather than assume. The model carries no state between requests, so its scores are a function of the input text and the weights alone, and nothing produced by the seventeenth run reaches the eighteenth; each seed is used once, so no draw is counted twice. It breaks when a provider rolls a new build partway through the batch or an alias resolves to a different snapshot, and a count pooled across two distributions estimates neither.

A count of one in thirty is compatible with a great many rates. A prompt that fails one time in a hundred will still show a single failure in a batch of thirty reasonably often, and so will a prompt that fails one time in ten. An interval collects the rates under which a count like this one would not be surprising, and reports the range of them. The 95% attached to it describes the procedure rather than this batch: draw fresh batches of thirty over and over, build the interval the same way each time, and about ninety-five in a hundred of those intervals contain the true rate. The true rate is one fixed number that does not move, and the interval is what varies from batch to batch.

The interval most people were taught steps out the same distance either side of the observed fraction, a couple of standard errors wide, the standard error being the square root of the observed fraction times one minus it, divided by the square root of the number of runs. At one in thirty that puts the lower end below zero, and a negative rate is not a rate. At zero bad runs in thirty it fails worse: the observed fraction is zero, so the quantity under the square root is zero, the width is zero, and the method reports a true rate of exactly zero with no uncertainty attached — a claim that the failure is impossible, made on the strength of not having seen it. Thirty bad runs in thirty breaks the same way at the other end. Both failures have one cause: the recipe approximates the count by a symmetric bell, while the real distribution of counts at a small rate is pinned against zero on one side and has a long tail on the other.

The 95% Wilson interval — a confidence interval for a proportion that stays sensible when the sample is small and the rate is near zero — asks a different question. It takes each candidate rate in turn and asks whether a batch of thirty drawn at that rate would produce a count like this one often enough to be unremarkable, keeping the rates that pass. Values below zero and above one are never candidates, so the endpoints cannot leave the range a proportion may occupy, and the surviving set is under no obligation to sit symmetrically around the estimate.

For one duplicate in thirty that interval runs from about 0.6% to 16.7%, which is wide because thirty is a small sample. Read against the 3.3% in the middle, the upper end sits more than thirteen points above the point estimate while the lower end is under three points below it, which is the asymmetry the naive recipe could not produce.

Put that against traffic: at 8,000 refund-eligible conversations a day, multiplying each end by the day's conversations gives 48 to 1,340 double refunds a day, with a central estimate near 270, and at an average refund of €62 that is somewhere between €3,000 and €83,000 a day. The multiplication carries an assumption the interval does not, that the day's conversations behave like this one. If this prompt shape is typical the exposure is that range; if the trace is an oddity, the range describes the oddity, and the step a reviewer should demand is a rate measured across a sample of refund conversations rather than generations from one of them.

What the interval does not license belongs beside it. It is a statement about a rate and not about the next run, so it predicts nothing about a replay done once more in a meeting. It covers sampling variability alone, and says nothing about the risk that the thirty runs were drawn under conditions production does not share. And it is computed from k, so a classifier that mislabels one output moves both ends of it: an interval computed on a wrong count is precise about the wrong quantity, which is the second reason the predicate deserved care.

A failure that fires one time in thirty is a systematic defect with a low probability. It is not evidence of absence, and the word that does the damage is "fluke". The output that duplicated the refund was drawn from a distribution over continuations fixed by the prompt and the weights before any randomness enters, since the picking step is the only random part of the loop and it chooses among candidates it did not create. An outcome that occurred therefore held a non-zero share of that distribution, and it held that share on every request carrying this input, including all the ones that came back clean.

A genuine fluke would need a mechanism outside the distribution — a corrupted byte, a truncated tool result, a harness that sent the request twice — and each leaves a mark in the frozen trace. When the trace shows the request was the request, what remains is a distribution containing bad continuations, which is a property of the configuration rather than of one occasion. Whether a rate that size is worth fixing is an economic question, separate from whether the defect is real.

The belief that reading displaces is worth stating in the form people hold it: a failure that does not reproduce did not really happen. It is tempting for a good reason. A single re-run is the cheapest experiment available, one command and a few seconds, and its result reads like a verdict. It is also correct practice nearly everywhere else, where identical input producing correct output is genuine evidence that the input you have is not the input that failed. What breaks the habit here is the picking step: identical input does not imply identical output when the code around the model draws from a distribution instead of taking the maximum, so a clean re-run is one more sample rather than a check.

The twin belief costs as much in the other direction: that a failure which reproduced once is deterministic and can be debugged by editing and re-running. One bad run out of one says only that the rate is not near zero, since the rates under which a single failure would be unsurprising stretch from a few per cent up to near certainty. A single reproduction carries almost no information about frequency.

The mirror case is worth having a number for as well. If thirty runs come back clean, the rule of three — with zero failures observed in n runs, the true rate can still be about 3/n — gives an upper bound of roughly 3/30, so the true rate could still be as high as 10%, and 10% of 8,000 is 800 duplicate refunds a day.

That rule is short enough to derive, and deriving it stops it being a folk constant. If the true rate is p, one run comes back clean with probability one minus p, and n independent runs all come back clean with that quantity multiplied by itself n times. Ask which rates survive an entirely clean batch: the ones for which that product is not too small, conventionally the ones above the one-in-twenty a 95% bound allows. Taking logarithms turns the product into a sum, and for a small p the logarithm of one minus p is very close to minus p, so the equation becomes n times p equalling about three — the logarithm of one in twenty being about minus three. Dividing by n leaves the bound, and the approximation holds precisely where the rule gets used, which is where p is small.

Thirty clean runs do not clear a ticket; they bound the rate at something that is still commercially serious. The same rule read backwards sizes the next batch: three divided by the number of runs only drops under one in a hundred once the runs pass three hundred. Getting a useful upper bound under 1% needs about 300 runs, which is still cheap: at a 30,000-token prompt and four seconds a call, 300 runs at concurrency 8 is about ten minutes and roughly 9 million input tokens.

Two of those figures do not sit together, and it is better to say so. Three hundred runs eight at a time, four seconds each, is under forty waves and a couple of minutes of model time — a floor rather than a schedule — while the ten minutes is wall clock, and the gap is what a provider's throughput limits do to a burst of nine million input tokens.

Seed 1017 is the most valuable artifact the run produced, and it is the one people throw away. Recording it means the failing generation can be re-materialised on demand instead of waited for, which turns an intermittent bug into something closer to a fixture. That holds where the provider treats a seed as reproducible, and only against the same snapshot: the arithmetic that stops temperature 0 being bitwise deterministic can spoil a seeded replay too, so a recorded seed is a strong lead rather than a guarantee.

Diffing that generation against a clean sibling from the same batch — the same prompt, a different seed — usually shows the two continuations agreeing for several hundred tokens and then parting at one specific position, and the token where they part is frequently informative on its own: a tool call opening where the clean run wrote a sentence, or a restatement of an order id that the clean run did not produce. That is not a cause, but it narrows what you are looking for before any edit is made.

Turning the temperature down is not the fix#

The instinct after seeing 3.3% at temperature 0.8 is to set production to 0.2 and move on. Temperature acts at one point in the loop, and being exact about which point settles the question. The model produces a score for every candidate next token — a logit, one unbounded number per candidate — and the softmax turns that column into probabilities by making every entry positive in an order-preserving way and dividing each by the total. Temperature divides every logit by the same positive number before that happens.

Dividing a column by a positive constant cannot reorder it, so the candidate that led still leads and the candidate that opens a second refund_order call keeps its exact rank. What moves is the spread: a smaller divisor stretches the gaps, and after the division by the total the leader takes a larger share while everything behind it takes less. Temperature never touches the scores themselves, which come out of the weights applied to the prompt, so a defect that lives in those scores — the model rating a second refund call as a plausible continuation — is as present at 0.2 as at 0.8, and narrowing which candidate gets picked makes it rarer without making it impossible.

The distribution does not have a cliff at 0.2. Temperature rescales the logits before the softmax, so a continuation carrying 3% of the mass at 0.8 might carry 0.4% at 0.2 — which at 8,000 conversations a day is 32 double refunds a day instead of 270. The change has made the failure rare enough to stop appearing in a thirty-run test and left it frequent enough to cost real money, and it has quietly changed the behaviour of every other feature sharing that model configuration, including the ones whose output quality depended on some sampling diversity. The same mechanism explains why the plumbing replay was silent about the rate: greedy decoding reports the leader, and the defect never was the leader.

The engineering fix for a duplicate side effect is an idempotency key on the refund call, so that a second call to refund_order with the same order id and the same request id is a no-op that returns the first result. The model may still emit that second call at whatever rate the distribution gives it, and the number of refunds paid stops being a property of the draw. That converts a probabilistic model behaviour into a bounded one, and it holds at any temperature, including the ones a future config change introduces. Sampling is stochastic by design; a correctness property that requires the model never to sample badly is not a correctness property.

The idempotency key bounds the side effect rather than the output: the response still contains two tool calls, so anything downstream that counts them still sees a duplicate. Prompt work can still reduce the rate afterwards, and it should, but it does so on top of a system whose worst case is bounded rather than underneath one where it is not.

Which replay makes which experiment admissible#

The reason this chapter comes before the ablation and bisection work is that the rate you just measured determines what a single re-run is worth. The mechanism is a comparison between how surprising an observation is under each of the two accounts you are separating: the edit changed the rate, or it did not. Run the edited prompt once and get a clean output. Under the account where the edit did nothing, a clean output was already the outcome twenty-nine times in thirty; under the account where it fixed things, a clean output is the outcome every time. Those expectations are barely different, and an observation both accounts expected leaves you believing what you believed before the run.

An informative run is the opposite arrangement: common under one account, rare under the other. That is why the base rate decides admissibility rather than the quality of the edit. If the failure fires on nearly every run, a single clean run afterwards is close to impossible under the account where the edit did nothing, and one run per condition genuinely separates the two. At one in thirty no amount of care rescues a single run, because the unedited configuration produces a clean output almost always by itself.

The last row carries the practical sting. Detecting a drop from 10% to 3% with 80% power, meaning an 80% chance of seeing the change when it is genuinely there, needs roughly 300 runs per arm, and detecting a drop from 3% to 1% needs several thousand. Nobody runs several thousand replays per hypothesis while debugging, which means intermittent failures cannot be debugged by editing and re-running once, however tempting the fast loop is.

What makes the next two chapters practical is finding a configuration where the failure is deterministic. Both edit one part of the request and re-run — the substitution ablation of the next chapter, where a retrieved block is swapped for an inert passage of the same length, and the instruction bisection after it, where a system prompt is halved and each half tested against the failure — so every step of either is a comparison priced by the table above.

Sometimes the frozen trace already is one: the failure fires on 30 of 30 runs, and then a single run per condition is admissible evidence and you can afford dozens of edits. Sometimes you can manufacture one by taking the seed that failed — 1017 above — and pinning it, so that each edited variant is compared against the same sampling path. That is weaker, because an edit changes the token sequence and therefore what that seed produces downstream, but it is far better than comparing two unpinned single runs. And when neither is available, the honest position is that every experiment in this investigation costs 60 runs and reports a difference in rates, with an interval attached.

The limits of a rate measured this way#

The 3.3% is conditional on one input. It is the failure rate of this prompt, with these retrieved blocks, this conversation history and this order id, and it says nothing about the rate across the population of refund conversations — which could be far lower, if this trace is unusual, or far higher, if the escalation channel surfaced the first instance of a common pattern. Estimating the population rate is aggregate work — topic 3.6, where a quality metric is computed over a sample of inputs rather than over generations from one input.

Temperature is also not the only knob that moves the distribution. Nucleus sampling with top_p at 0.95 truncates the tail before temperature ever applies, so the same prompt at 0.8/1.0 and 0.8/0.95 are different experiments, and presence or frequency penalties change the dynamics of repeated tool calls specifically. Pinning means pinning all of them, along with the snapshot, because a provider updating the model underneath a stable alias will change your rate without changing your code.

What the two replays leave you with is a pair of statements you can defend: that the recorded bytes produce a sane answer on your own stack today, and that this prompt fails at some rate with an interval around it. Neither is a diagnosis, and the second is the one to quote before anybody starts editing, since it decides whether an edit followed by one clean run is a finding or a coincidence.

With a reproduction you trust and a rate you can quote, the investigation can finally start changing things. The first edit most people reach for on a retrieval-augmented generation (RAG) failure — one where the answer was meant to come out of the passages fetched at request time, and where chapter 3 already checked that the answer was present in those passages — is to delete the retrieved block that looks guilty and re-run, and that edit changes three things at once.