The first-divergence index

Two runs of one request come back and somebody has to say how different they are. The obvious answers are all bad in the same way: a character-level similarity ratio is high for any two paragraphs of English about the same subject, a distinct-output count says nothing about severity, and reading the two strings side by side produces an adjective rather than a number. None of them tells anybody where to look next.

The measurement this chapter settles on is the first-divergence index: the number of leading tokens on which two generations agree exactly. It costs nothing to compute, it is unambiguous, and it carries diagnostic information rather than only severity, because the causes discussed so far in this topic produce divergences at characteristically different positions. The chapter defines it alongside the two weaker metrics it replaces, works out how many runs an estimate of a divergence rate actually needs, adapts the comparison for structured outputs where token-level equality is the wrong test, and ends with the three execution modes of the previous chapter priced side by side on one set of prompts.

The three numbers worth recording#

Comparing N generations of the same request is a small enough problem that it is worth being exact about, because the obvious approaches are subtly wrong.

Distinct-output count is the cheapest and the least informative. Hash each output, count the distinct hashes, and report the size of the largest group. Twenty runs producing three distinct strings with the most common appearing fourteen times is a real fact about your stack that fits in a sentence. It tells you nothing about severity, because one flipped near-tie on a filler word and a completely different answer both register as "distinct".

Positional token agreement is the number people reach for second, and it needs care. Comparing token 5 of run A against token 5 of run B is meaningful right up until the two sequences diverge; after that, the sequences are describing different things and often at different lengths, so position 200 of A and position 200 of B have no relationship at all. An agreement rate computed positionally over the whole output therefore reports a number somewhere between "the true prefix agreement" and "the base rate at which two English sentences share a token in the same slot", and drifts between those depending on how the two continuations happen to line up. If you want a similarity measure over the full outputs, use an edit-distance-based one and call it that; do not present a positional rate as though it means agreement.

First-divergence index is the one that earns its place. It is the length of the longest common prefix in tokens, it is exactly defined, it is cheap, and unlike the other two it carries diagnostic information. Record it per pair, and report the distribution rather than the mean, because the distribution is bimodal in a way the mean will hide. The two humps are the two causes the next section separates: an upstream difference shows up in the first handful of tokens, and an arithmetic near-tie shows up after a long identical run, with very little between them. A mean over those two populations lands in the gap where neither cause lives and points at nothing.

from itertools import combinations
from statistics import median

def first_divergence(a: list[int], b: list[int]) -> int | None:
    """Index of the first differing token, or None if one is a prefix of the other
    and they are equal as far as the shorter one goes."""
    for i, (x, y) in enumerate(zip(a, b)):
        if x != y:
            return i
    return None if len(a) == len(b) else min(len(a), len(b))

def summarise(runs: list[list[int]]) -> dict:
    pairs = list(combinations(runs, 2))
    divergences = [d for a, b in pairs if (d := first_divergence(a, b)) is not None]
    return {
        "n_runs": len(runs),
        "distinct": len({tuple(r) for r in runs}),
        "largest_group": max(sum(r == s for s in runs) for r in runs),
        "pairs_diverging": f"{len(divergences)}/{len(pairs)}",
        "first_div_min": min(divergences, default=None),
        "first_div_median": median(divergences) if divergences else None,
        "mean_len": sum(len(r) for r in runs) / len(runs),
    }

Two implementation notes. Compare token IDs, not strings: a difference in a trailing space or in how two adjacent tokens merge is invisible in the decoded text but real in the sequence the model produced, and comparing strings will hide it. The reason such a thing is possible at all is that decoding a token sequence into text is a one-way function that loses information. Several different sequences can spell the same string, because the same characters can be covered by one token or by two adjacent ones. So two runs that genuinely made different decisions can decode identically, and a string comparison will report agreement where the model diverged. Going the other way is safe — identical token IDs always decode to identical text — which is why the sequence is the thing to compare and the text is the thing to read. If your provider returns only text, either re-encode the outputs with that model's tokenizer or request per-token log-probabilities, which return the token sequence directly. Second, note that first_divergence returns None only when the sequences are genuinely equal, and returns the shorter length when one is a strict prefix of the other — a truncated output is a divergence at the truncation point, not a match.

How many runs you need#

Twenty runs is the number chapter 1 suggested because it is cheap and it either shows you the effect or does not. It is not the number you need to estimate a rate. If the true probability that two runs of a request disagree is p, the chance that twenty runs are all identical is roughly (1 − p)^19 for the pairwise-to-first comparison, so a 2% divergence rate produces twenty identical outputs about 68% of the time.

That exponent is worth deriving rather than trusting, because it is where the weakness of a twenty-run experiment becomes visible. Take the first run as the reference. Every one of the remaining runs then has to match it for the experiment to come back clean, and each matches with probability one minus the divergence rate. The runs do not influence one another, so the chance that they all match is that quantity multiplied by itself once for each of them — nineteen times, not twenty, because the reference is not being compared with itself. At a 2% rate that is a number just under one multiplied by itself nineteen times, which comes out around two thirds. Two experimenters running this on the same unstable prompt would get one clean result and one dirty one, and would draw opposite conclusions from a system behaving identically for both.

The same expression says what a clean run of twenty does establish, which is worth knowing so as not to overclaim in the other direction. At a rate around 15% the chance of twenty matching runs falls to a few per cent, so seeing them would be a genuine surprise and the rate is fairly excluded. At the low rates that actually cause trouble in production, a clean twenty is what you would expect to see most of the time whether the problem exists or not. A clean run of twenty is therefore weak evidence of anything: it rules out rates above roughly 15% and says almost nothing below that.

Getting a usable rate means more runs, and the useful framing is how small a rate you need to detect. To be reasonably confident of observing at least one divergence at a true rate of 1%, you want on the order of 300 runs; at 0.1%, on the order of 3,000. Both of those come from the same shortcut, which is worth carrying around: to have a good chance of seeing at least one event whose probability is p, run about three divided by p trials. Three over one per cent is the 300, and three over a tenth of a per cent is the 3,000. The relationship is a straight reciprocal, so each further decimal place of sensitivity costs ten times the runs, and that is what makes chasing a very low divergence rate against a metered endpoint an expensive way to learn something. Those same run counts are trivial against a local endpoint, which is another reason the deterministic and serialised arms are usually measured on infrastructure you own. In practice the pragmatic split is twenty runs across many different prompts to find out which prompts are unstable at all, then several hundred runs on the two or three that are, to get a rate you can put in a table.

Prompt choice matters as much as run count. Twenty runs of a prompt that produces four tokens of JSON will show you nothing regardless of your stack, and reporting that as "we are deterministic" is how a team convinces itself the problem does not apply to it. Pick the longest, most open-ended generation in your production traffic as the stress case and the most constrained one as the control, and report both.

Early and late divergences have different causes#

The reason the index is worth more than the rate is that it partitions your suspects.

A divergence at token 0 to 5 is usually not the phenomenon from chapters 1 and 2. Reduction-order noise flips a decision only where two logits are within a whisker of each other, and the first few tokens of a well-specified generation are typically the most confident in the whole sequence — the model is committing to a format, an opening word, a JSON brace.

Why the opening is the confident part is worth one step, because the whole diagnostic hangs off it. At position zero the model has been given the entire prompt and has produced nothing, so everything it is conditioning on came from you, and a well-written instruction leaves very little room about how a reply begins. A response that must be JSON starts with a brace and nothing else is legal. A summary asked for in a fixed register starts one of two or three ways. The logit gap at such a position is measured in whole units, and an arithmetic wobble measured in millionths cannot reach across it. So an early divergence is evidence that the conditioning itself differed — which is to say that something upstream of the model handed the two runs different text.

When those opening tokens flip, look upstream instead: the assembled prompt was not identical (a timestamp in the system block, a re-ranked retrieval set — the documents your system searched for and pasted in ahead of the question, coming back this time in a different order — a truncated history), the model revision changed, or a sampling parameter is set differently between the two runs. Chapters 6 and 7 deal with both of those.

A divergence at token 50 to 400 with a long identical prefix is the signature of the batch-variant mechanism. The model was confidently reproducing itself, hit a position where the top two candidates were separated by less than the arithmetic noise, and the two runs took different branches. In the published 1,000-completion experiment the entire fan-out began at token 103 — 102 tokens of perfect agreement, then a coin flip. That shape is diagnostic.

A cluster of divergence indices at the same position across many pairs is worth pursuing directly. It means one specific decision point in this prompt is a near-tie, and you can go look at it: request the top log-probabilities at that step and read the gap. A gap of 10^-5 confirms the diagnosis and, incidentally, tells you the prompt is genuinely ambiguous at that point in a way a small prompt change might resolve.

Structured outputs need a different comparison#

Everything above assumes the output is a token sequence you compare as a sequence, which is right for prose and wrong for a JSON payload or a tool call, the structured function name and argument object a model emits when it wants your code to perform an action and hand the result back. Two responses that differ only in key order, or in whitespace between fields, or in 1.0 versus 1, are identical as far as every consumer of that output is concerned, and a token-level comparison will report them as divergent and send you looking for a cause that does not matter. Parse first, canonicalise, then compare on the parsed structure — and keep the token-level first-divergence index alongside it as the diagnostic number rather than the pass/fail one.

The severity ordering also inverts. A prose divergence at token 74 is usually cosmetic; a tool-call divergence is usually not. If two runs call the same tool with the same arguments and merely narrate it differently, nothing downstream changes. If they call different tools, or pass amount=41.50 and amount=41.05, you have a behavioural difference that a similarity score will report as 0.99 and that will cost somebody money. Comparing structured outputs field by field, with the fields that drive side effects compared exactly, is the version of this measurement worth alerting on.

Pricing the three modes over the same set#

With a metric defined, the comparison from chapter 3 becomes a table you can actually produce. Run the same set of prompts through each execution mode and record both the reproducibility numbers and the throughput, because a reproducibility number without its cost is not a decision input.

Those figures come from one 8B model on one node and are illustrative of the shape rather than transferable; what matters is what the three rows say relative to each other. Reproducibility is not a flag you switch on, it is a purchase, and the two ways to buy it differ by roughly an order of magnitude in what they cost. Serialising is a 25× throughput cut for a result that batch-invariant kernels get you at around a 1.6× cut.

Both multipliers are divisions along the fourth column, and the fifth column is the fourth one turned upside down. Production runs at 3,180 tokens a second and the serialised arm at 128, and the first divided by the second is the 25. Production against the batch-invariant arm's 1,970 is the 1.6. The cost column follows automatically rather than being measured separately, because the hardware is rented by the hour and the hour costs the same however many tokens come out of it: a mode producing a twenty-fifth of the tokens spreads the same rent over a twenty-fifth as many of them, so $0.42 becomes $10.40. Reading the two columns as one number stated twice is the quickest way to see that this table has no free options in it. If you produce this table for your own stack and the middle row is the only deterministic option you have, that is a finding worth escalating rather than a configuration to adopt.

The last column matters because it converts the argument from an engineering preference into a budget line. "We could have reproducible evaluations for an extra $0.26 per million output tokens on the evaluation fleet only" is a proposal someone can approve. "We should use deterministic kernels" is not. That $0.26 is the batch-invariant row's $0.68 less the production row's $0.42, and quoting the difference rather than the multiple is deliberate: a 1.6× increase sounds like a large fraction of something, and on an evaluation fleet running for part of a night it is a small number of dollars. The same multiple applied to production traffic is not, which is exactly why the proposal names the fleet it applies to.

What a divergence rate cannot tell you#

It does not tell you which output is correct. Two runs diverging at token 74 might both be right, both be wrong, or differ in a way that matters enormously — an amount, a date, a yes instead of a no. Divergence rate and answer quality are independent axes, and a deployment that reproduces itself perfectly while being reliably wrong scores 100/100 on the table above. If you need to know whether the answers are good, that is an evaluation set, and it is a different instrument.

It also does not give you a safety threshold. A 3% divergence rate on a summarisation endpoint is probably fine and a 3% divergence rate on a tool-calling endpoint that decides whether to issue a refund is not, and no property of the measurement distinguishes them. What the number does support is a frequency argument: at a 3% per-call divergence rate, a CI job that makes eight model calls per run and asserts exact equality on each has roughly a 1 − 0.97⁸ ≈ 22% chance of failing per run, which on a repository merging thirty times a day is about seven red builds a day from nothing at all.

Both steps of that are worth doing slowly, because the result surprises people who are comfortable with the 3%. A CI run is green only if every one of its eight assertions holds, and each holds with probability one minus the divergence rate. The calls do not influence one another, so the chance of a wholly green run is that quantity multiplied by itself eight times, which is a little under four fifths — leaving the 22% for at least one failure. Multiplying that by the thirty merges a day gives the expected number of red builds, and the multiplication is legitimate because an expected count adds up across independent trials whether or not the trials are related in any other way. The lesson generalises past this table: a per-call rate small enough to ignore becomes a per-run rate that blocks people, because assertions multiply and merges accumulate.

And the measurement says nothing about stability over time. Every number in that table is a property of one image on one hardware generation on one day. Re-run it after a base image bump and the deterministic rows can quietly stop being deterministic, which is why the table is a thing to regenerate on a schedule rather than a fact to record in a wiki.

An index, then, rather than a percentage. The first-divergence index costs a handful of lines, survives being put in a table next to a throughput figure, and answers the only question worth asking of a pair of differing outputs, which is where to look. Its distribution across many pairs tells you which mechanism you are dealing with; its median next to a cost per million tokens tells you what removing that mechanism would cost; and its absence tells you nothing at all about correctness, which is a different instrument's job.

That frequency calculation is where the topic stops being a curiosity and starts costing people their afternoons, because the divergence does not stay in a notebook. It shows up as a red build on a test somebody wrote in good faith, on a Tuesday, in a pull request that changed nothing relevant — and the first repair anyone reaches for makes the situation considerably worse than the flake it removes.