Sampling temperature and answer equivalence

Chapter 5 left a self-consistency run with two unspecified knobs. The score is the size of the largest agreeing group divided by the number of samples, which presupposes that the samples differ from one another when the model is unsure, and that there is a rule for deciding when two answers count as the same. Neither follows from the code that computes the fraction, and both are easy to leave to whatever the surrounding system already happened to be configured with.

The first knob is the sampling temperature, and its failure is silent: a scorer wrapped around a client configured for deterministic output reports maximum confidence on everything, forever, without erroring. The second knob has no default at all, which is worse than a bad default, because whoever writes the comparison picks one implicitly and it never gets written down. This chapter measures what the temperature buys, works through four definitions of agreement on one set of samples, and ends with the division of labour between confidence scores and deterministic checks.

Sampling has to be genuinely on, and the temperature sets the ceiling#

Topic 1.6 established why greedy decoding produces the same continuation every time, and the mechanism is short enough to restate here, because everything in this chapter rests on it. A model does not emit text directly. At each position it produces one score for every entry in its vocabulary, and a decoding rule turns that vector of scores into a single choice. Greedy decoding is the rule that takes the highest-scoring entry, and providers expose it as temperature=0.

The scores at a position are a function of two things only: the model's weights, and the text that precedes the position. Neither of those changes between two identical requests to the same model version. So the highest-scoring entry is the same entry, the token appended is the same token, and the text preceding the next position is therefore also the same. That argument repeats at every position, and each step is what licenses the one after it, which is why two completions come out identical rather than merely similar.

Self-consistency measures the spread of a distribution by drawing from it, and a distribution you are not drawing from has no observable spread. The consequence for the score is arithmetic rather than approximate. The score is the size of the largest agreeing group divided by the number of samples; all five samples are the same string, so they form a single group of five, and five divided by five is 1.00. The same thing happens on every case after it.

A score that is wrong on some inputs is still a measurement, and a measurement can be recalibrated or thresholded differently. A score that returns the same constant on every input measures nothing, because its value does not depend on the input, so every case lands on the same side of every cutoff and no threshold separates anything.

The prerequisite is not merely "temperature above zero" either: a fixed seed pins the draws in the same way, so five seeded requests with the same seed are also one answer five times, and providers that accept an n parameter differ on whether the n completions share a seed. The step underneath that is worth spelling out. Drawing from a distribution means generating a pseudo-random value and using it to pick a candidate, a seed initialises the generator that produces those values, and a generator started from the same seed produces the same sequence every time. Same values, same picks, same text — so a seed reproduces the temperature-zero failure with sampling nominally switched on. What the method needs is therefore not a setting but a property of the output, that the five samples actually differ. Both are worth verifying by sending one request and printing the five strings before anything is built on top.

The wiring deserves a test rather than an inspection, because the failure is silent by construction and the code that produces it looks correct. The test is three lines and belongs in CI: draw N samples for a fixed prompt, assert that the set of distinct strings has more than one member, and assert that the observed agreement over a small fixture of known-ambiguous cases is below 1.0. Both assertions fail the moment someone points the scorer at a temperature-0 client, changes a shared default, or adds a seed for reproducibility in an unrelated debugging session. A monitoring alert on the production distribution catches the same class of problem after the fact — if the agreement score has a variance of zero across a day's traffic, something upstream is not sampling — and the two together are considerably cheaper than a review queue that sits empty for a week.

The belief that makes this failure common is worth naming, because it is a reasonable belief rather than a careless one. A scorer is usually written as a wrapper around the client that already talks to the model, and that client was configured by somebody who knew what they were doing for the job it was originally written to do. Inheriting its settings therefore looks like inheriting a decision instead of making one. The trap is that the setting which was correct for the original job — deterministic output on a drafting path — is the exact setting that empties this one of meaning, so the more carefully the original client was configured, the more confidently the wrong value arrives.

The second belief is about the observable rather than the code. An empty review queue is normally evidence that whatever feeds it found little to send, and that reading is correct in almost every system anybody has worked on. It is wrong here because the queue is fed by the score, and a constant score sends nothing regardless of what the model is doing. An observable that stays silent for the same reason whether the system is healthy or broken cannot tell the two apart, which is why the check that works is on the score's variance rather than on the queue's depth.

Once sampling is genuinely on, the temperature you choose sets how much disagreement the method is capable of showing, and there is an optimum that has to be measured rather than assumed. On 200 labelled replies from a policy reply drafter, scored at five samples each:

The third column needs its definition before the rows can be read. AUROC is the probability that a randomly chosen correct answer scores higher than a randomly chosen incorrect one, so 0.50 is a coin flip and 1.00 is perfect separation. It measures ordering rather than accuracy: it asks whether the score puts the wrong answers at the bottom, and says nothing about what any particular value means.

The curve has a peak because the two ends fail for opposite reasons. Too cold and every case scores 1.00, so the score carries no information; too hot and cases disagree because the sampler is reaching into low-probability territory, which produces disagreement unrelated to whether the model knows the answer. At 1.2 the mean agreement is genuinely lower, but the ordering degrades: correct answers start disagreeing with themselves as readily as incorrect ones do. Somewhere in the middle the disagreement is tracking real competition between candidate answers, and where that is depends on the model and the prompt.

The cold end fails at the selection rule. A low temperature stretches the gaps between the leading candidate and the rest, so the leader holds almost all of the probability at almost every position and a draw returns it almost every time. Two samples can only differ if at least one position went a different way, which needs a position where two candidates sit close enough for the second to be picked with any regularity. Very few qualify at 0.3, which is why mean agreement there is 0.94 and why the handful of cases that fall below 1.00 cannot order a queue.

The hot end fails at what the disagreement is caused by. A high temperature compresses the gaps, the tail of the distribution then holds enough mass that draws land in it regularly, and a draw from the tail changes the answer for a reason that has nothing to do with whether the model knew it. Cases the model is sure about and cases it is unsure about now both disagree with themselves, so the two stop being separable — which is what the AUROC column reports when it falls from 0.74 to 0.66 while mean agreement is still going down. The peak sits where most of the observed disagreement is caused by genuine competition between candidate answers rather than by the sampler's reach.

Agree has to be defined before anything can be counted#

Here are five samples from that reply drafter, all responding to the same question about the returns window:

1. Refunds are available within 30 days of delivery.
2. You can request a refund up to thirty days after the item arrives.
3. The refund window is 30 days from delivery.
4. Refunds are available within 30 days of purchase.
5. Refunds must be requested within 14 days.

Four definitions of agreement, four different scores from the same five strings:

All four rules run the same three steps and differ in exactly one of them. Split the five samples into groups, putting two in the same group when the rule says they agree; count the members of the largest group; divide that count by five. The scores below differ only because the groupings differ, which makes the spread from 0.20 to 1.00 a fact about the definitions rather than about the model.

Exact string match reports 0.20 and is simply broken; five fluent English sentences will essentially never be byte-identical above temperature 0, so this definition returns near-zero on every case and a threshold tuned against it is a threshold on nothing. Trace it on the five above and every sample is a different sequence of bytes from the other four, so each one forms a group of its own, the largest group has a single member, and one divided by five is 0.20.

The next rule is worth walking key by key, because the fraction in the table is the only evidence that the extractor did what was intended. Samples 1, 3 and 4 yield 30 directly, sample 2 yields it once the normaliser maps thirty onto the digits, and sample 5 yields 14. That is a group of four and a group of one, so the largest group has four members and four divided by five is 0.80.

Extracting the number of days and normalising gives 0.80, treating the first four as equivalent and flagging the fifth. That is closer to right, and it hides something important: sample 4 anchors the window to purchase rather than delivery, which for a retailer with two-week shipping is a materially different policy. The extraction saw 30 in all four and stopped looking. This is the general hazard of field extraction — it measures agreement on the field, and everything outside the field is invisible.

Embedding cosine at a 0.85 cutoff reports 1.00, which is the worst answer of the four, and it fails in a way that catches teams repeatedly. Sentence embeddings are fixed-length vectors positioned so that texts with similar meanings point in similar directions, compared by the cosine of the angle between them, and they encode topic and register far more strongly than they encode polarity or magnitude. Refunds must be requested within 14 days and Refunds are available within 30 days of delivery sit at cosine 0.91 because they are both short English sentences about refund deadlines. The same effect makes The customer is eligible for a refund and The customer is not eligible for a refund land around 0.93. Any agreement metric built on embedding similarity will call a claim and its negation the same answer, which is precisely the disagreement you most needed to see. Traced on the five samples, every pair clears the cutoff, so all five land in one group and five divided by five is 1.00.

The reason the vectors behave that way is in how they were fitted, and it decides whether a better cutoff would rescue the method. An embedding model is trained to place texts a reader would treat as being about the same thing close together, and the pairs it learns from differ overwhelmingly in subject and register, because subject and register are what most pairs of texts differ in. Polarity and magnitude are carried by a handful of characters — a not, a 14 where a 30 stood — and changing them leaves the vocabulary, the sentence shape and the topic untouched. The geometry is therefore a geometry of what a text is about, and two answers that contradict each other are about the same thing.

That makes the failure structural rather than a badly chosen number. The pair that disagrees about the deadline sits at 0.91, which is above the 0.85 cutoff that was already grouping the genuine rephrasings together, and the negation pair sits higher still at 0.93. Any cutoff low enough to keep the rephrasings in one group also keeps the contradiction in it, and any cutoff high enough to expel the contradiction expels rephrasings on the way. The metric does not order the two kinds of difference the way the task needs them ordered, so there is no value of the threshold at which it does.

Bidirectional entailment reports 0.60 and is the most defensible of the four here: it groups 1, 2 and 3, separates 4 on the purchase-versus-delivery distinction, and separates 5. That grouping is three, one and one, so the largest group has three members and three divided by five is 0.60. It costs 20 calls to a small NLI model — a natural-language-inference classifier, trained to say whether one sentence implies another, contradicts it, or neither — for five samples, because you need both directions on every unordered pair, and it degrades on long answers where one clause differs and the rest match: the entailment verdict is all-or-nothing over the whole text, so a five-claim answer with one wrong claim is simply "not equivalent", which is true but uninformative about which claim moved.

The call count follows from the relation rather than from the implementation. Entailment is directional: Refunds are available within 30 days of delivery implies that refunds are available at all, while the bare claim that refunds are available implies nothing about a deadline. Two answers are equivalent only when each implies the other, so every pair has to be asked twice, once in each direction. Five samples give 5 × 4 ordered pairs of distinct samples, which is 20, and that is the figure in the table. The count grows with the square of the sample size, so a method that is comfortable at five samples is not comfortable at fifty.

The all-or-nothing limit has the same source. Entailment relates two complete statements, and a reply making five claims is one statement for that purpose, so the classifier is asked whether the whole of one text follows from the whole of another. One differing clause makes the honest answer "no", and the verdict carries no channel for saying which clause did it — a limit on what the method can report rather than an error in what it reports.

A judge model — a second model prompted with the samples and asked to say which of them mean the same thing — reports 0.80 and inherits everything a judge model inherits: it is another generation, non-deterministic in the same way, biased toward answers phrased like its own, and it is the most expensive option per case. It also has a property the other three lack, which is that its verdict changes when you rerun it — asked to group the same five samples ten times, a judge at temperature 0.7 returned three different groupings, so the confidence score for this case is itself a random variable. There is a real use for a judge, which is when the equivalence relation is genuinely subtle and you can write down what "the same answer" means in a paragraph but not in a regex. There is a much more common non-use, which is reaching for one because deciding what equivalence means is hard and delegating the decision feels like solving it. The judge does not decide it; it puts the decision somewhere you cannot read.

The 0.80 in that row shows what "somewhere you cannot read" costs. The score says the largest group had four members and one sample stood apart; it does not say which four, because a fraction is a count and the count survives any grouping of that shape. Under the extracted-field rule the grouping is recoverable by rerunning the regular expression, and under entailment it is recoverable from the pairwise verdicts. A judge returns the number without the grouping that produced it, so two runs both reporting 0.80 may have disagreed about which sample was the odd one out.

The temperature you sample the judge at is a separate setting from the one you sample the answers at, and it should be zero. The variance you want to measure is the variance in the answers; variance in the grader is noise added on top, and a judge at temperature 0 is at least reproducible, even if it is still opaque.

For most production systems the right starting point is the second row of the table, made explicit and made narrow:

import re
from collections import Counter

def extract(answer: str) -> str | None:
    """Pull the decision-bearing fact: window length AND its anchor event."""
    m = re.search(r"(\d+|thirty|fourteen|sixty)\s*days?\s*"
                  r"(?:of|from|after)\s+(\w+)", answer, re.I)
    if not m:
        return None
    words = {"thirty": "30", "fourteen": "14", "sixty": "60"}
    n = words.get(m.group(1).lower(), m.group(1))
    anchor = {"purchase": "purchase", "order": "purchase",
              "delivery": "delivery", "arrives": "delivery",
              "arrival": "delivery"}.get(m.group(2).lower(), m.group(2).lower())
    return f"{n}d/{anchor}"

def agreement(samples: list[str]) -> tuple[str | None, float]:
    keys = [extract(s) for s in samples]
    counts = Counter(k for k in keys if k is not None)
    if not counts:
        return None, 0.0
    top, n = counts.most_common(1)[0]
    return top, n / len(samples)          # unparseable samples count against you

Extracting the anchor alongside the number moves the score for those five samples from 0.80 to 0.60 and puts the case in front of a human, which is the correct outcome. Counting unparseable samples in the denominator rather than dropping them is the other detail worth copying: a sample the extractor could not read is a sample that disagreed about the answer's shape, and silently discarding it inflates every score.

Which method each decision is entitled to#

Two routing decisions from the same product make the split concrete. In the first, the model outputs one of five labels and the code branches on it. That is a label decision, the whole answer is one token, and chapter 2's renormalised probability is a legitimate confidence score obtained for free. In the second, the model outputs a three-sentence reply with a citation. Averaging the logprobs over that reply gives a number dominated by ordinary grammatical tokens, so it mostly measures how conventionally the sentence is phrased. The available methods there are agreement across samples, or the conversion move from chapter 5 that turns the citation into a label decision and reads its logprob.

One part of the second case is outside the reach of both. Whether the cited clause exists is a database lookup against your policy documents, costing a millisecond and returning a definite answer. No confidence signal derived from the model should be used where a deterministic check is available, and a surprising amount of the uncertainty in citation-bearing outputs dissolves once the citation is validated rather than scored.

There is a design consequence in that pairing worth stating plainly. The confidence machinery and the correctness machinery are different tools and should be built in that order: validate everything that can be validated deterministically, and reserve the confidence signal for what remains. A system that scores citation-bearing replies at 0.8 agreement while never checking whether the cited clause is in the policy database has spent five inferences approximating something a hash lookup answers exactly. In practice the deterministic layer usually removes a third to a half of the cases that would otherwise have gone to review, and it removes them with certainty rather than with a score.

The ordering runs that way because of what the two kinds of check are about. A deterministic lookup consults a record — the policy database, a case list, a schema — so its verdict is a statement about the world, and asking twice returns the same verdict twice. An agreement score consults the model's own output several times, so its verdict is a statement about the text, and it is a random variable with the standard error chapter 5 measured. A statement about the world settles a question; a statement about the text only narrows one. Every case the deterministic layer settles is a case no score has to be trusted on.

The failure that survives both checks is chapter 4's. A fabrication the model produces confidently is produced again on every sample, so five drafts leaning on clause 4.2 agree perfectly, and if clause 4.2 exists the lookup passes as well. Neither check has looked at whether the clause says what the reply claims it says. The agreement score sits awkwardly next to that failure, because it is reporting something real — that the answer is not an artefact of one unlucky draw — in a place where a reader will take it for something else.

Agreement over an extracted field is also, strictly, agreement about that field. Five samples can agree perfectly on 30d/delivery while three of them attach it to the wrong product category, and the score will read 1.00. Define agreement over what the downstream system consumes, and know that everything you did not extract is unmeasured.

Both settings are choices, and neither announces itself when it is wrong. A temperature inherited from a client tuned for deterministic output produces a score that is constant; an equivalence rule chosen by whoever wrote the comparison produces a score that means whatever that rule happened to measure, which on the five samples above ranges from 0.20 to 1.00 for the same five strings. Writing both down next to the threshold, and testing the first in CI, is what turns the fraction into a measurement.

With sampling on and equivalence defined, the score means something. Turning it into a routing decision requires one more number — the cutoff — and that number is neither transferable between models nor stable across versions of the same one.