Sample size for a human evaluation

A diagnosis that rests on human ratings which did not move only holds if those ratings were capable of moving. That is a claim about the instrument, and it has two independent parts. The first is statistical power — the probability that a study detects a difference of a given size when that difference is genuinely there — which is arithmetic, decided by the sample size and the effect you care about. The second is design: which prompts the answers were generated from, whether raters could tell the variants apart, and what question they were asked. The second part decides what the number means, and no amount of the first compensates for getting it wrong.

This chapter builds the comparison in that order, because the order is where teams go astray. The instinct when a human study produces an unwelcome answer is to argue about its size, and size is the cheapest thing about it to check.

Sizing against the difference you would refuse to miss#

The sample size question has a well-posed form, and it is not "how many ratings can we afford". It is: what is the smallest difference in user-visible quality that would change a decision, and how many ratings does it take to detect that difference reliably? For a support assistant where a five-point movement in answer quality is inside the noise of a quarter's product changes and a ten-point movement would justify shipping, ten points is the number to power for.

A comparison of two proportions around 0.68, at the conventional 5% two-sided significance and 80% power, needs roughly

from statistics import NormalDist

z_a = NormalDist().inv_cdf(0.975)   # 1.960
z_b = NormalDist().inv_cdf(0.800)   # 0.842

def n_per_arm(p1, p2):
    var = p1 * (1 - p1) + p2 * (1 - p2)
    return (z_a + z_b) ** 2 * var / (p2 - p1) ** 2

n_per_arm(0.68, 0.78)   # 305  -> 300 ratings per arm detects ~10 points
n_per_arm(0.68, 0.91)   #  44  -> a 23-point effect needs ~44 per arm

Four lines of that are doing all the work, and a reader who cannot rebuild them cannot tell a study that was underpowered from one that was fine. Take them in turn, starting from what the study is trying to see. Two arms each produce a win rate. The rates differ a little even when the two systems are identical, because each rate is computed from a finite number of ratings and a finite sample lands somewhere near the truth rather than on it. So the question the arithmetic settles is how much scatter that produces, and whether the difference you care about is bigger than the scatter by enough to be noticed.

var is the scatter. For a proportion, the variance of a single rating is p * (1 - p) — a rating is a coin that comes up "win" with probability p, and that expression is the variance of one such coin. Two arms contribute one each, and the two are added rather than averaged because the quantity being tested is their difference, and the difference of two independent quantities carries the uncertainty of both. Notice what the expression implies about where studies are cheap: p * (1 - p) is largest at one half and shrinks toward the ends, so a comparison of two systems that both win about half the time needs more ratings than a comparison of two that both win nine times in ten.

(z_a + z_b) is where the two error rates you chose enter. z_a is the number of standard errors at which a two-sided test at 5% stops calling a difference chance, and the percentile the code asks for looks wrong until you remember the test is two-sided: a 5% error budget split between an upper tail and a lower one leaves half of it in each, so the cut has to sit above all but that half rather than above all but the whole 5%. z_b is the number of standard errors of headroom you need beyond that cut for a real effect to land past it 80% of the time, which is the percentile matching the power. They add because the true difference has to carry the estimate across the cut and leave enough margin that four draws in five arrive on the far side. Squaring the sum converts a distance measured in standard errors into a count of ratings, because the standard error of a mean shrinks with the square root of the sample size: halving the uncertainty costs four times the data, and that square is where the cost comes from.

Dividing by (p2 - p1) ** 2 is the same fact seen from the other end. The smaller the difference you insist on detecting, the more ratings it takes, and the relationship is quadratic rather than linear — asking to detect five points instead of ten does not double the study, it quadruples it. That single property is why the sizing question has to be asked as "what would change a decision" rather than "what would be nice to know", since a decision that turns on a five-point difference costs four times as much evidence as one that turns on ten.

Put the numbers through it once to see the parts land. At 0.68 against 0.78 the variance term is a shade under four tenths, the squared multiplier for 5% and 80% is a little under eight, and the difference squared is one hundredth. Eight times four tenths is about three; three divided by one hundredth is about three hundred, which is the 305 the code prints. Do it again for 0.68 against 0.91 and two of the three pieces move. The difference squared goes from one hundredth to a little over five hundredths, which is more than five times larger and divides the answer by that much. The variance term shrinks as well, because 0.91 is nearer the end of the range than 0.78 was, and it takes off roughly another fifth. Three hundred divided by five and a bit, then reduced again by that fifth, lands in the mid forties, which is the 44 in the second comment.

The third column is worth more than the second. On a null result at 300 per arm, the 95% confidence interval on the difference runs from about −7.5 to +7.5 points. That figure is the same arithmetic run backwards and is worth checking against the sizing line above it, because the two have to agree. The standard error of the difference is the square root of the variance term divided by the sample size — the same p * (1 - p) from each arm, added, over 300 — which comes to a little under four points. A 95% interval reaches z_a standard errors either side of the estimate, and that multiplier — just under two — times a little under four is about seven and a half. Compare that with the row's second column, which says the same sample detects a ten-point effect: the interval is narrower than the detectable effect, and it has to be, because detecting an effect means the interval around it excludes zero and that needs the effect to be the wider of the two. Reporting the interval rather than a p-value converts "we did not find a difference" into "any difference is smaller than 7.5 points", which is a claim with content. The formal version of that argument is an equivalence test — two one-sided tests against a bound you declare in advance — and for a decision like this one the confidence interval carries the same information in a form nobody has to be talked through.

The before-measurement is the part teams skip, and skipping it is what makes the argument unwinnable later. Rating only the new variant, against an absolute standard — "was this answer good?" — yields a number like 74% that has nothing to be compared against, and the discussion collapses into whether 74% is a good score. A baseline that was pre-registered, meaning written down as a comparison before any answer was rated, turns the human evaluation into a difference rather than a level, and differences are what decisions are made on. It also costs half of nothing, since the baseline answers already exist.

Pairing tightens it further and costs nothing. Rating the two variants' answers to the same 300 held-out prompts, rather than two independent samples, removes prompt difficulty from the variance, and on this workload the paired interval came in at ±5.4 points.

The mechanism there is worth a sentence, because "pairing reduces variance" is the kind of phrase that gets repeated without being held. Some prompts are harder than others, and a hard prompt drags down whichever variant is asked it. With two independent samples, the two arms get two different draws of prompts, so a difference between the arms mixes together a difference between the systems and a difference between the two prompt draws — and the second of those is noise you paid for and did not want. Ask both variants the same prompt and that term cancels: whatever the prompt's difficulty contributed, it contributed to both sides of the comparison, so it is gone from the difference before any statistic is computed. What is left is the variation in which answer a rater preferred, which is the thing being measured. Going from ±7.5 to ±5.4 is therefore not a better test of the same data; it is the same test on a quantity with one source of noise removed, bought by a change to which prompts get rated.

Two further design choices interact with the sizing and are worth settling before the table is consulted. Ratings can be absolute — score this answer on a scale — or pairwise, which is what the table above assumes; pairwise ratings are faster, agree between raters more often, and produce a difference directly rather than requiring two levels to be subtracted afterwards. And ties have to be allowed or forbidden explicitly. Forbidding them forces a rater who genuinely cannot separate two answers into a coin flip, which adds variance without adding information; allowing them and reporting the tie rate alongside the win rate tells you how much of the comparison the panel could actually resolve. That is the same argument that applies to a judge asked the same question, and the two tie rates are worth reading next to each other.

Blinding, hold-out and the panel#

A large sample of bad ratings is a precise measurement of the wrong thing, and the failures are independent of the sample size.

The prompts must come from a set the optimisation loop never scored against. If the loop selected on 400 prompts and the human study rates answers to those same 400, then any edit that exploited the specific prompts is measured on its home turf, and the human number will move for reasons that will not reproduce in production. Holding out is not about avoiding memorisation by a model that was never trained; it is about the loop having had no opportunity to fit the evaluation cases.

Raters must not be able to tell which variant produced which answer, which is harder than it sounds when one variant emits ## Summary / ## Details / ## Sources on every answer and the other does not. Blinding by hiding a label is insufficient when the format is the tell, because a rater who notices that the structured answers are the "new" ones will drift toward or away from them depending on what they believe the team wants. The step people skip is the one that makes this specific rather than general anxiety about bias: the optimised prompt's very first accepted edit was an instruction to use those three headings, so the variant under test announces itself in the first line of every answer it produces. Blinding failed here not because raters are unreliable but because the treatment is visible, and a treatment you can see is not blinded by withholding its name. The workable version normalises presentation before rating — render both answers through the same markdown pipeline, or strip headings to bold lines — and randomises which answer appears first in each pair, then records the order so that a preference for position can be measured and, if present, subtracted.

pairs = []
for prompt in holdout_300:
    a, b = answers["baseline"][prompt], answers["optimised"][prompt]
    left_is_baseline = rng.random() < 0.5
    pairs.append({
        "pair_id": uuid4().hex,
        "prompt": prompt,
        "left":  normalise(a if left_is_baseline else b),
        "right": normalise(b if left_is_baseline else a),
        "left_is_baseline": left_is_baseline,   # stored, never shown
    })

Each pair needs more than one rater if you want to know whether the raters agree with each other, and disagreement rates matter: if two raters pick different winners on 40% of pairs, the panel is measuring something noisier than the effect you are hunting, and the effective sample size is well below the nominal one. Three raters with majority adjudication on this workload produced pairwise agreement of 0.71, which is unremarkable for open-ended quality judgements and is the reason the interval is ±5.4 points rather than ±3.

Who the raters are changes what the number means, and the choice is a trade between throughput and domain knowledge. A crowd panel will produce 300 paired ratings in an afternoon for a few hundred dollars and will judge answers on readability and apparent confidence, because that is all they can assess about a question involving your product's refund policy. Support agents from the team that answers these questions for a living will catch that an answer cites a superseded policy version, and they cost roughly six times as much per hour and cannot be scheduled at short notice. The compromise that works is a small expert panel — twenty to forty pairs, adjudicated carefully — used to validate that the large crowd panel is ranking the same way; when the two disagree, the crowd number is measuring fluency and should be reported as such.

Rater calibration is worth an hour before the study rather than an argument after it. Give every rater the same fifteen practice pairs with known answers, discuss the disagreements, and only then start collecting. On this workload that step moved pairwise agreement from 0.62 to 0.71, which is the difference between a ±7-point interval and a ±5.4-point one at the same cost — cheaper than buying the equivalent precision with 150 more pairs.

The rating question itself has to be about the user's outcome rather than about the rubric. Asking raters "which answer is more accurate and complete" imports the judge's own vocabulary into the human instrument, which is exactly the surface the optimiser learned to satisfy. Asking "if you had sent this question to support, which reply would have resolved it" measures something the loop had no access to.

An instrument that did not move is not an instrument too weak to move#

The distinction the objection elides is between a null result and a flat point estimate. A study that returns 68% against 71% with a wide interval genuinely has failed to resolve the question, and asking for more data is the right response. A study that returns 68% against 68% has produced an estimate of zero, with an interval that excludes everything anyone claimed. More data would narrow ±5.4 to ±3; it would not create twenty-three points that two paired measurements agree are absent.

Being specific about what would change the conclusion keeps this from being stubbornness, which is why the three named results above matter more than the power calculation. Each is cheap, each is falsifiable, and naming them in advance means the next disagreement is settled by running one rather than by arguing about instruments. By the time the power objection was raised, the ablations had already shown 17 of the 23 points recovered by formatting alone, so two independent lines of evidence pointed the same way and neither of them depended on the human study at all.

The panel has its own preferences#

The comparison built here is the acceptance test, and it is not ground truth. Human raters have their own systematic preferences, and they overlap uncomfortably with the judge's: raters prefer longer answers, prefer confident phrasing, and prefer visible structure, all of which is why preference-trained models produce those things. A generator optimised long enough against a human panel would find the panel's preferences the same way it found the judge's, at perhaps a hundred times the cost per iteration, which is the only reason the failure is rarer.

The panel's advantage is independence from the specific scorer being gamed, and independence is a property of the setup rather than of humans as such. A second judge held out of the loop buys some of the same independence for a thousandth of the price, which is why a held-out judge column and a human panel belong in the same report — they fail in different ways, and agreeing when both are flat is worth more than either alone. Where the product allows it, the instrument worth more than both is an outcome measure: escalation rate, repeat-contact rate within 24 hours, or resolution confirmed by the user, none of which can be improved by adopting a rubric's vocabulary.

A human comparison earns its authority twice over: once from being sized against a difference somebody committed to caring about, and once from a design in which the loop had no opportunity to fit the cases and the raters had no way to tell the variants apart. Get the second right and 300 paired ratings will carry a diagnosis. Get it wrong and two thousand will not, because the number will be measuring format recognition or prompt difficulty rather than answer quality.

All of this assumed a planned comparison, with a before and an after, both variants available to generate from, and someone who knew which prompt produced which answers. The next time a number moves it will not be planned — a pass rate will be nine points higher on Tuesday than it was on Monday, with a dozen changes in between, and what you can establish about it will depend entirely on what the harness wrote to disk before anyone suspected anything.