Self-consistency, majority voting over N samples, best-of-N with a judge

Read these first

• Chain-of-thought, scratchpads
• Sampling

What you will know

• Sample the same prompt N times and aggregate the answers

  Agreement only helps when samples make meaningfully different errors; changing temperature may still leave failures highly correlated.

• Show accuracy improving with N, and where it stops improving

  Plot task-level accuracy and variance against N on a held-out set; gains often flatten before the added latency and spend become acceptable.

• Calculate what the improvement cost per point

  Report marginal dollars and latency per percentage-point gain, including judge calls, retries, and any failed or discarded samples.

• Use the spread across samples as an uncertainty signal

  Disagreement is poorly calibrated when every sample shares a model or prompt bias, so compare the signal with observed correctness before routing on it.

• Benchmark one call to a stronger model against five weaker-model calls and a judge under matched quality, latency, and budget constraints

  A judge can add substantial cost and amplify shared stylistic bias, so compare end-to-end success and p95 latency rather than vote agreement alone.

Chapters

1. 1

   Self-consistency and shared errors

   Five samples at temperature 0.8 with a majority vote lifts overall accuracy two points and changes almost nothing on the slice that was already worst. On that slice the samples share a model and a prompt, so the error comes from a systematic misconception or an ambiguous instruction rather than from sampling: the vote comes back unanimous and confirms the wrong answer. Covers what the vote estimates, the difference between dispersion and displacement error, and why agreement has to be checked per slice.

   Read chapter
2. 2

   Sampling diversity and error correlation

   Self-consistency assumes you can produce N answers that fail differently, and nothing about the method delivers that on its own. Covers where the samples come from: why greedy decoding returns the same answer N times, how raising temperature trades per-sample accuracy for diversity, what else you can vary — prompt phrasing, example order, a second model family — when temperature leaves the failures correlated, and how to check that N samples decorrelated rather than assuming they did.

   Read chapter
3. 3

   Aggregating free-form answers

   Self-consistency counts votes, which requires an answer space with countable things in it. Over free-form paragraphs every sample is a different string and every count is one, so the proposal quietly becomes best-of-N: you need a canonicaliser, an equivalence function or a judge model, and the judge brings a sixth call, its own cost and its own preferences. Covers the line between the two methods, where embedding similarity fails as an equivalence relation, and which output shapes each suits.

   Read chapter
4. 4

   The accuracy-versus-N curve

   With repeat spread under half a point, accuracy can run 84.0, 88.1, 89.0, 89.3 and 89.4 for N of 1, 3, 5, 9 and 15 — so going from 5 to 15 triples generation spend to buy four tenths of a point the measurement cannot confirm exists. Covers drawing the curve once by subsampling stored draws rather than running each setting separately, reading the plateau as an estimate of displacement error, and why the accuracy figure has to travel with cost and latency beside it.

   Read chapter
5. 5

   Marginal cost per accuracy point

   Five points of accuracy for a 5x bill on 600,000 calls a day is not a trade anyone can evaluate until somebody prices an error. Covers computing marginal cost and latency per percentage point from tokens rather than from the multiplier, counting judge calls, retries and discarded samples, adaptive N with an early exit, and the prior question: if a downstream check or a human already catches the failures, N buys only a small reduction in rework.

   Read chapter
6. 6

   Judge bias in best-of-N

   Generating five candidates and asking the same model with a rubric to pick the best introduces a selector whose preferences correlate with the generator's, so it reliably picks the most fluent answer rather than the most correct one. Covers scoring a judge only on the discriminative subset against a random-pick baseline, the position-swap flip rate, why high judge agreement is not evidence of judge accuracy, and what an independent signal would have to look like instead.

   Read chapter
7. 7

   Agreement as a routing signal

   Routing every non-unanimous case to a human can settle at 8% of traffic while accuracy on the other 92% barely moves, because unanimous cases carry most of the errors: confidently wrong cases produce agreement and stay in the auto path, while disagreement mostly selects hard cases the model often got right. Covers the calibration test you run before routing on anything, the capture curve and what its shape tells you, reviewer lift on the escalated population, and how to set a threshold once the signal passes.

   Read chapter
8. 8

   Independent uncertainty signals

   When agreement turns out not to track correctness, no threshold on it will, so this chapter covers what you build in its place: a verifier or objective check whose errors are independent of the generator, an ensemble decorrelated across model families or prompts rather than across samples of one prompt, a router trained on input features you can observe before the model is called at all, and spending the escalation budget on the slice you already know is worst. Each is measured on the same capture curve, and catching an error is priced separately from repairing it.

   Read chapter
9. 9

   End-to-end success under a deadline

   One call to a strong model at 1.9s p95 against five cheap calls plus a judge at 2.9s and two points better offline, compared on the axes an offline harness has no column for: partial failures across six network calls, a p95 set by the slowest sample rather than the average, a judge advantage that shrinks on traffic postdating the eval set, and the caller's timeout. The design that wins offline loses 3.3 points of end-to-end success, and would have won had the deadline been four seconds.

   Read chapter