Five samples agreeing measures the model's peak, not the passages

Self-consistency is the practice of generating an answer several times at a non-zero temperature and using the spread between the samples as a measure of uncertainty. It is the signal teams reach for the moment somebody establishes that a model's stated confidence is not worth cutting on, and it has three genuine advantages: it needs no special API support, it works behind any model that returns text, and unlike a number the model writes about itself it is computed from behaviour rather than from self-report.

What it measures is narrower than "uncertainty", and getting the narrowness right is the whole chapter. Drawing samples estimates a property of the distribution the samples were drawn from. That distribution is a distribution over what the model would say, conditioned on the prompt. It is not a distribution over what is true, and nothing in the sampling procedure gives it any contact with a document that was not retrieved.

Agreement is an estimate of a peak#

Sampling N answers and looking at their spread is a Monte Carlo estimate — an estimate of a quantity built by drawing random samples and counting what comes back — of how concentrated the output distribution is. The arithmetic is short enough to do in full, and doing it in full is what shows the limits of the signal.

Let p be the probability of the modal answer under the sampling distribution: the single answer the model is most likely to produce. If the five draws are independent, the chance that all five land on that answer is p multiplied by itself five times. At p equal to 0.92 that is a little under two thirds — 66% — and at p equal to 0.75 it is under a quarter, 24%. Two things follow. First, the agreement rate is a fairly sharp instrument for reading p off, since a drop of seventeen hundredths in p cuts the five-of-five rate by more than half. Second, and more importantly, p is the only thing it reads. The exponent is a statement about the sampler and there is no term anywhere in it for the world.

That is the stumble in this chapter, and it is a belief about a defence rather than a slip in arithmetic. Sampling feels like a second opinion, because the answers come back separately and you compare them. It is a second draw from the same distribution. A distribution can be peaked because the retrieved passage states the answer in one sentence, and it can be peaked because the model absorbed a plausible fact during pretraining and has no competing hypothesis. Both look identical from the outside, and the quantity that stays high across both is agreement while the dangerous property — whether the peak came from a passage or from the weights — varies freely underneath it.

The diagram is the objection in one picture: two branches that need to be distinguished arrive at the same observation, because the observer is the sampler and the sampler sees one distribution. Nothing downstream can undo a collapse that happened before the measurement was taken.

Temperature is the parameter people reach for when the agreement rate looks wrong, and it moves the signal without improving it. At T = 0.5 the 5/5 agreement rate on a full corpus rises from 58% to 79%, which compresses the score into a narrow band at the top and destroys what resolution it had; at T = 1.2 it falls to 34%, and much of the new disagreement is wording noise that the entailment clustering then has to absorb. Neither of those movements touches the branch structure above. What they do is change p, and changing p changes the estimate of p.

The temperature that maximises AUROC on a given evaluation set is a quantity you can find, and finding it is a design decision that deserves naming as one. Sweeping temperature and taking the best value is fitting one hyperparameter to a few hundred graded items, on a metric computed from those same items, which is how a signal that measures 0.66 honestly comes to be reported as 0.71. The reported figure is then a property of the sweep rather than of the signal, and it will not survive contact with a different question mix.

Topic 4.7 introduced this machinery for a different purpose — taking the majority vote to improve the answer, which works because on questions where reasoning paths diverge, the modal path is more often right than a single sampled path. Reusing the spread as a confidence score changes the objective from "pick the better answer" to "predict whether the answer is correct", and the two objectives have different failure profiles on the same peaked-and-wrong case. Majority voting is untroubled by it; it returns the wrong answer it would have returned anyway, and nothing is worse than it was. Using the peak as a confidence signal takes the same input and actively asserts that the wrong answer is trustworthy, which is worse than saying nothing.

Measuring spread without measuring wording#

Before the numbers mean anything, agreement has to be defined on meaning rather than on strings, since five samples asserting one thing will be five distinct strings. Three approaches are in common use and they are not interchangeable.

The rows are not a quality ladder, and reading them as one hides what each step buys. Each rule is sensitive to a particular kind of difference and blind to the rest, and the gaps say which kind.

Exact match treats every rewording as a disagreement, which on free-form answers means it reports disagreement almost always, and it is still what a first implementation usually does. Embedding similarity fixes precisely that and introduces a new blindness in the same move: "three retries" and "five retries" embed close together, because the sentences differ in one token and the vector is an average over all of them, so the one token carrying the entire claim is diluted by everything around it. What the embedding gained on paraphrase it lost on the difference that decides correctness. Bidirectional entailment, the clustering used in the semantic-entropy line of work — where the entropy is computed over clusters of meaning rather than over distinct strings — is the version that behaves: two samples are in the same cluster when each entails the other, and an entailment model is trained to be sensitive to exactly the numeric and negation differences embeddings smooth over. It costs an NLI model, which is the same component chapter 5 uses directly, so a team that adopts this clustering has already paid for the signal that chapter recommends instead.

import math
from collections import Counter

def semantic_entropy(samples, entails):
    """Cluster samples by bidirectional entailment, return entropy in nats."""
    clusters = []
    for s in samples:
        for c in clusters:
            if entails(c[0], s) and entails(s, c[0]):
                c.append(s)
                break
        else:
            clusters.append([s])
    n = len(samples)
    probs = [len(c) / n for c in clusters]
    return -sum(p * math.log(p) for p in probs), clusters

score, clusters = semantic_entropy(five_samples, nli_entails)
# score = 0.0, clusters = [[all five samples]]  -> maximum confidence, wrong answer

When every sample falls into one cluster, that cluster's share of the samples is one, and the logarithm of one is zero, so the entropy is zero. Zero is the smallest value the statistic can take, which means five identical claims produce the highest-confidence value the signal is capable of emitting — whatever those claims say.

What five generations cost#

The signal is not free, and the cost is multiplicative on the most expensive part of the pipeline. Measured on a runbook assistant with a 3,400-token prompt and 180-token answers, against a mid-tier hosted model:

The cost column is exactly proportional to the sample count — each row is its N multiplied by the single-generation cost of $1.90 — which is worth knowing because it means the column is arithmetic rather than a measurement, and it carries no caching discount at all. In practice prompt caching absorbs a large share of the input tokens when the samples share a prefix, so realised spend comes in below this column; the figures here are the undiscounted ceiling, and how far below depends on your provider's caching terms rather than on anything in the table.

Issuing the extra generations in parallel keeps the latency penalty, quoted here at p50 and so describing the median request rather than the worst one, to the difference between the slowest and the median completion rather than N times the median, so the wall-clock cost is much smaller than the token cost. Returns flatten quickly past five, and that follows from the exponent rather than from any property of these particular models: additional samples refine an estimate of p that is already well estimated, and the marginal information in the sixth draw about a quantity five draws have already pinned down is small.

The latency figure is the one that tends to decide deployments. Two extra seconds on a support-desk lookup is tolerable; on an interactive assistant where the answer streams token by token it is not, because the confidence score is only available once every sample has finished, so either the stream is held back until the slowest generation lands or the answer is shown and then retracted. Teams that want both usually stream the primary answer and run the sampling asynchronously, then attach a warning banner a second or two later, which is a product decision about whether a user who has already read an answer will believe a late caveat.

Why averaging it with the stated score makes things worse#

The proposal to combine the two signals — average them, or take the minimum, or fit a two-feature logistic regression — assumes they carry independent information about correctness. On this corpus they correlate at r = 0.41 overall, and the correlation is not the problem by itself; the problem is where the correlation is strongest.

Before drawing anything from that table, check it against the pooled figures the topic has been quoting, because the check is available and it is exact. Chapter 3 established the split: of the two thousand graded answers, 514 had no gold passage in the retrieved set and the remaining fourteen hundred and eighty-six did. Weight the stated-confidence column by those sizes and it lands on 0.52, which is chapter 1's figure for that signal. Weight the agreement column the same way and it lands on 0.66, the figure quoted for five-sample agreement. Weight the last column and it lands on 0.61. Three signals, one weighting, three hits.

What that licenses is a reading of every headline AUROC in this topic: each is a fact about a retrieval hit rate of roughly three in four as much as it is a fact about the signal. Improve retrieval and the pooled figures rise without anything about the signals changing; let retrieval degrade and they fall. What it does not license is treating AUROC as generally decomposable — it is not a size-weighted average of within-stratum values in general, and the agreement here is an identity that holds on these numbers rather than a theorem to lean on elsewhere.

Now the substance. Both signals reach their maximum on the retrieval-gap subset, where they should be at their minimum. Averaging two scores that are jointly wrong in the same direction produces a composite that is confidently wrong with less variance, and the overall AUROC of the average, 0.61, is worse than agreement alone at 0.66.

That the composite is worse rather than merely no better is worth a step, because the intuition from ensembling says otherwise. Ensembling helps when the component errors are uncorrelated, so that averaging cancels them. Here the two components are functions of the same conditional distribution, so their errors share a cause and do not cancel. On the retrieved half the stated score is near chance, so mixing it into a signal at 0.71 dilutes a good ordering with a random one. On the absent half both are near chance and agree, so the average inherits the error with reduced variance and no correction. The weaker signal contributes noise where the stronger one was working and agreeing error where it was not, and there is no subset where it contributes anything else.

Set against each other, the two signals do have failures the other avoids, and naming them precisely is what makes the combination question decidable rather than a matter of taste. The stated number is a single self-report produced in the same breath as the answer, it lands on four round values so it has almost no resolution, and it moves when the prompt is rephrased or the schema changes, none of which is true of an agreement rate computed over five draws. The agreement rate costs four extra generations and two seconds of latency, needs a clustering rule that is itself a modelling decision, and reports maximum confidence on a single peaked belief, none of which is true of a number the model writes for free. What they share is the one property that decides the RAG case, and sharing it is what makes the average a worse instrument than the better half. The Stop Asking LLMs for Confidence Scores screencast makes the first half of this argument well and stops before the second half, which is roughly where most teams that adopt self-consistency stop too.

That comparison also decides which signal to validate when there is time for one. The retrospective check on stated confidence is not really a sprint's work at all, because it needs no new inference — every logged answer already carries its number, so the cost is the grading, and the grading is required for either validation. The sprint goes to the sampling signal, and the validation consists of: a few hundred questions drawn from real traffic and graded, stratified by whether the gold passage appeared in the retrieved set, AUROC computed within each stratum rather than pooled, a reliability curve on the agreement score, and a measured latency and cost delta at N = 5. The stratification is the part that is usually skipped, and it is the part that separates a 0.71 from a 0.52 in the same signal; pooled over the whole corpus that pair reports 0.66 and reads like a success.

Where sampling genuinely earns its cost#

None of this makes self-consistency spread a bad signal in general, and it is the best of the three measured so far. It is genuinely informative on questions where the model has to combine two retrieved facts, or interpret an ambiguous phrase in the question, or choose which of two passages applies — cases where the answer distribution really is spread and the spread really does track difficulty. On the subset of one runbook corpus requiring a two-passage synthesis, it reaches AUROC 0.74. It is also the only signal here that works with any model behind any API, including one that exposes nothing but text.

The condition those cases share is that the competing hypotheses are all present in the model's distribution, which is what the sampler can see. When the correct answer is absent from that distribution because the passage carrying it was never retrieved, there is nothing for the samples to disagree about, and no number of draws recovers what the distribution was never conditioned on. That limit is structural rather than a tuning problem, and it does not move with N, with temperature, or with a better clustering rule.

A signal that can notice a retrieval gap has to take the passages and the claim as two separate inputs and ask whether the first supports the second. That is a different kind of model entirely — it reads the evidence rather than the generator — and it brings a different, more specific set of failures with it, which are the subject of the next chapter.