Building a private benchmark

A selection benchmark is a set of cases built to decide between candidate models, as distinct from a monitoring set built to describe how a shipped system is doing. The two look similar and are allocated on opposite principles. A monitoring set should mirror traffic, because its question is "what is happening to users". A selection set should not, because its question is "which of these candidates is better", and a case that every candidate answers correctly carries no information about that question no matter how often it occurs.

The step from each of those questions to its own allocation rule is short enough to walk. "What is happening to users" asks for an average over a population of encounters, and such an average is estimated honestly only by a sample in which each kind of case appears in the share it holds there; weighting by frequency is not realism, it is the arithmetic of that question. "Which of these candidates is better" asks for a difference between two systems, and a difference is settled entirely by the cases on which the two behave differently. One question wants the mix production has and the other wants the mix on which candidates disagree, and one artefact cannot be both.

A private selection benchmark is the instrument that everything in the previous six chapters was arguing for and, simultaneously, the instrument that inherits all of their problems. It has its own slice question, its own scoring protocol, its own way of going stale, and its own capacity to be optimised against until it stops predicting anything. The difference is that you control it, which means you can build it so that it survives the audit.

Allocating the set by stratum#

The belief standing in the way deserves naming, because it is held for good reasons. A team that draws its evaluation cases uniformly at random from last month's traffic has done what a statistics course teaches: they did not go looking for examples that flatter a favourite model, every case is one a real user really sent, and the sample is an unbiased estimate of how the candidates perform in production. All of that is true, and none of it defends the set as a selection instrument. Unbiasedness is a relationship between a sample and one particular quantity, and the quantity a uniform draw estimates well is production accuracy. What the same draw does to the difference between two candidates, it does without introducing any bias at all, which is why the defect survives every check a careful team thinks to run.

The mechanism is worth taking slowly, because everything else in this chapter rests on it. Scoring two candidates on the same set gives each a total that is the sum of its per-case scores. Take any case on which both produce the same verdict, whether both find the clause or both miss it. It adds the same amount to the first total as it adds to the second, so subtracting one total from the other — the arithmetic of "which is better" — subtracts those two identical contributions to nothing. The case is present in both totals and absent from the difference, and it would have been just as absent had it appeared a thousand times.

That would be a curiosity if the cases that cancel were scattered evenly through production. They are not, and the reason is what makes frequency the wrong allocation rule rather than an unhelpful one. A case is frequent because it is routine — a standard template, a common phrasing, a document shaped the way most of its kind are shaped — and routine is what every candidate has seen most of and has already learned to handle. So, with exceptions worth hunting for rather than assuming away, the chance that two candidates agree on a case rises with the same property that makes the case common. Production frequency is close to a direct measure of a case's tendency to cancel.

The consequence for a headline score follows in one more step. If most of the set comes from strata that cancel, most of it contributes to both totals and to neither difference, while the few separating cases contribute a difference that is then divided by the size of the whole set. Two candidates far apart on the work the project exists to get right come out looking nearly identical. The set has not lied: it spent its budget resolving a quantity nobody was deciding on, and left the quantity in question to whatever noise sits in a handful of items.

Six weeks and real budget buys roughly 400 annotated documents for a compliance-extraction task, which is enough if they are allocated deliberately. The allocation is written as strata, meaning named groups of items that each get a fixed quota of the set rather than a share proportional to how often they turn up in production, and the resulting mix looks nothing like the traffic.

The first row is the cancellation argument with a price on it. Routine documents are about seventy-two percent of what arrives and twenty percent of what gets annotated, because seven documents in every ten spent on template work every candidate handles would contribute the same amount to every total and nothing to any difference between them.

Two more rows deserve comment. Harvested past failures is the highest-value stratum per annotated document and the one teams skip, because it requires that someone has been logging production errors in a form you can retrieve. If that log does not exist, starting it is the first task of the six weeks, and the eval built in week six will be better than the one you could have built in week one for exactly that reason. Negative controls are what make fabrication measurable at all: a set in which every document contains the clause cannot distinguish a model that finds clauses from a model that always claims to.

"Cannot distinguish" is doing real work there, and the reason it is literally true rather than merely difficult takes one step. Consider two behaviours: a model that reports the indemnity clause when the document contains one, and a model that reports an indemnity clause whatever the document says. On a document containing the clause they produce the same output, so on a set where every document contains the clause they produce the same output on every item. Every score anyone could compute — recall, expected cost, span overlap — is a function of those outputs, and two behaviours whose outputs agree everywhere cannot be told apart by any function of them. Documents that genuinely lack the clause are what make the two differ, because there the second reports something and the first reports nothing. A model can only be caught claiming what is not there if it is given occasions on which nothing is there.

The consequence of enriching is that the aggregate over the whole set is no longer an estimate of production quality — it is roughly four times as hard as production. That is fine as long as nobody reports it as an expected accuracy. Report per-stratum scores as the primary output, and compute a frequency-reweighted headline only when someone needs a single number for a slide, with the weights written next to it.

That aggregate is not an accuracy, and the reason is the cancellation argument read backwards. Each stratum's share was chosen for how much it separates candidates rather than for how often it occurs, so the set over-represents the hard strata and under-represents the routine ones by design. An average over that mix is an average over a population of documents no user sends and no month produces — a serviceable basis for ranking two candidates against each other, and a false thing to write beside the word accuracy. A per-stratum score has no such problem, because the routine documents really are the routine documents, and the reweighted headline recovers the production question by putting the traffic shares back in as weights.

The oracle, and how stable it is#

Extraction is one of the fortunate tasks where a real oracle is available — a source of ground truth that settles every item, fixed in advance and independent of whichever model is being scored. A human can mark the exact span of a contract that constitutes the indemnity clause, and a model's output either overlaps it or does not. That is much better than a rubric, because it removes the judgement step at scoring time.

It does not remove the judgement step; it moves it to annotation. Two paralegals annotating the same fifty documents will not agree perfectly. One marks the operative sentence and the other marks that sentence together with the definitions it depends on; one treats a carve-out sitting in a schedule as part of the clause and the other treats it as a separate provision. Neither is being careless. The documents are genuinely ambiguous at the edges, and the guideline the annotators were handed did not rule on the case in front of them. The share of items on which two independent readers land on the same answer is their annotator agreement, and the degree to which they disagree is a hard ceiling on what any model score means. Measure it: have two annotators independently label an overlapping subset, compute agreement per clause type, and adjudicate the disagreements with a third reader.

Why that number is a ceiling rather than a caveat takes one step. No model is ever scored against the truth; it is scored against one annotator's rendering of the truth, because that rendering is the only version the scoring code can see. On this project, agreement on payment-terms clauses was 0.94 and agreement on change-of-control clauses was 0.61. On that second clause type two trained readers landed on the same answer about three times in five, so on close to two items in five the recorded label is one of two defensible answers and the other reader would have recorded the other. There the label is near enough the verdict of a coin, in the sense that which answer got written down turned on which reader was handed the document. A model producing the right span is marked wrong on some of those items and a model producing the wrong one is marked right on others, so a clause type where two trained humans agree 61% of the time cannot support a claim that one model is four points better than another on it.

The response is to sharpen the annotation guideline for that clause type and re-annotate, not to average the disagreement away. Averaging yields one label per item and a set that looks clean, but blending two readings does not take the noise out of the labels — it takes the noise out of sight, and the score computed on top inherits all of it while reporting none. Sharpening attacks the cause, since most disagreement on a clause type traces back to a situation the guideline never ruled on, and a rule saying which reading to write down removes the disagreement rather than splitting it. Re-annotating is what makes the items labelled before the change comparable with those labelled after.

Recording the annotation guideline version alongside the items matters for the same reason chapter 2's provenance block mattered. A score is a function of the labels, and the labels are a function of a document someone wrote.

Sizing, from the smallest gap you would act on#

The question "how many items do we need" has no answer until you say what difference you intend to act on. Working backwards from that is straightforward using the paired bootstrap from the previous chapter: run a pilot of 80 items, compute the per-item variance of the paired difference, and solve for the count that gives an interval narrower than your minimum detectable effect, which is the smallest gap between two models you have committed in advance to acting on.

import math

def required_items(pilot_a: list[float], pilot_b: list[float], mde: float) -> int:
    """Items needed for a 95% paired interval narrower than +/- mde/2,
    estimated from a pilot run of both models on the same items."""
    diffs = [a - b for a, b in zip(pilot_a, pilot_b)]
    n = len(diffs)
    mean_d = sum(diffs) / n
    var = sum((d - mean_d) ** 2 for d in diffs) / (n - 1)
    # half-width = 1.96 * sqrt(var / n)  ->  solve for n
    return math.ceil(var * (2 * 1.96 / mde) ** 2)

# pilot: 80 documents, per-item score in [0, 1], paired
print(required_items(pilot_a, pilot_b, mde=5.0))   # -> 264
print(required_items(pilot_a, pilot_b, mde=2.0))   # -> 1648

The single line of arithmetic inside that function is worth reading rather than trusting. The half-width of a 95% interval around the mean of n paired differences is 1.96 * sqrt(var / n), where var is the per-item variance the pilot measured and 1.96 is the multiplier that makes the interval a 95% one. Everything there is fixed by the pilot except n, so the count follows from deciding how narrow the interval must be, and an interval no wider than the gap you intend to act on has a half-width of half that gap. Set 1.96 * sqrt(var / n) equal to mde / 2, divide both sides by 1.96, and square both sides to clear the root: var / n on one side, (mde / (2 * 1.96)) ** 2 on the other. Multiplying up gives n as var * (2 * 1.96 / mde) ** 2, the line the function returns, with math.ceil outside it because an item count is a whole number.

Set the constants aside and one relationship is left. The half-width falls as the square root of the item count, so the count must rise as the square of any tightening asked of it: halving an interval's width costs four times the items, because four is what a count must be multiplied by before its square root doubles. Tightening the target gap from five points to two is not a halving but a shrink by a factor of two and a half, and the price is that factor multiplied by itself, a bit over six. Six times two hundred and sixty-four is a little under sixteen hundred, and the extra quarter in the factor carries the rest of the distance to the 1,648 the second call reports.

The jump from 264 to 1,648 as the target gap tightens from five points to two is the whole lesson about sizing. Resolving small differences is not a little more expensive, it is quadratically more expensive, and at 400 annotated documents you can settle a five-point question and cannot settle a two-point one. Knowing that in advance is what stops a team from spending six weeks and then arguing about a difference their instrument was never able to see.

One assumption holds that whole calculation up, and the allocation this chapter argues for breaks it. It takes the per-item variance measured on eighty pilot items and treats it as the variance the full set will have, which is legitimate only if the full set is drawn the way the pilot was — variance is a property of the mix of items rather than of the metric computed over them. An enriched allocation is not drawn that way: the routine stratum has a small per-item variance precisely because almost every item in it scores the same for both candidates, and the rare-clause stratum has a large one precisely because that is where candidates come apart. One blended pilot variance fed into one overall count therefore sizes the set for a mix it will not have.

Sizing applies per stratum, not just overall. A rare-clause stratum with twelve items cannot support a per-stratum conclusion no matter how large the total set is, which is the arithmetic reason the enriched allocation exists: thirty percent of 400 is 120 items on the clause types that matter, against the eight documents uniform sampling would have given.

A quota is easier to judge as a count than as a percentage. Three tenths of four hundred is where the hundred and twenty comes from, and a hundred and twenty annotated items on the clause types carrying the regulatory exposure is enough to put an interval around a per-stratum score and to see a gap of the size this project committed to acting on. With a dozen items the interval around a score is wider than the range the candidates could plausibly fall in, so every interval overlaps every other and the ranking read off them is the order the items happened to land in. An instrument sized below what its question needs does not merely fail to find a gap; it amplifies the noise around one into a number somebody will present as a finding.

Scoring when the errors are not the same error#

A missed clause is a compliance gap that a human reviewer might still catch downstream. A fabricated clause is a false statement entered into a regulatory record, which downstream review is much less likely to catch because it looks like a finding rather than an absence. Treating both as "one wrong answer" is a modelling decision, and it is the wrong one here.

from dataclasses import dataclass

COSTS = {"hit": 0.0, "miss": 1.0, "fabrication": 8.0, "wrong_span": 2.0}

@dataclass
class ClauseOutcome:
    clause_type: str
    kind: str        # hit | miss | fabrication | wrong_span

def cost_and_counts(outcomes: list[ClauseOutcome]) -> dict:
    """Report the scalar AND the vector. The scalar ranks; the vector is
    what a human actually has to sign off on."""
    counts = {k: 0 for k in COSTS}
    for o in outcomes:
        counts[o.kind] += 1
    total_cost = sum(COSTS[k] * v for k, v in counts.items())
    found = counts["hit"]
    return {
        "expected_cost_per_doc": total_cost / len({id(o) for o in outcomes} or {1}),
        "recall": found / max(found + counts["miss"], 1),
        "precision": found / max(found + counts["fabrication"] + counts["wrong_span"], 1),
        "counts": counts,
    }

That function returns three readings of the same run: an expected cost per document, recall, which is the share of the clauses genuinely present that the model found, and precision, which is the share of the clauses it reported that were really there. The eight-to-one ratio in COSTS is not measured, derived or defensible on technical grounds. It is a policy input, and the right way to obtain it is to ask the person who owns the regulatory risk what ratio they would accept, write their answer down with their name next to it, and re-run the ranking at a couple of nearby ratios to see whether the winner changes. If the ranking is stable from four-to-one through twenty-to-one, the exact number does not matter and you can stop arguing about it. If the ranking flips between six and eight, you have discovered that the decision is being made by an assumption rather than by the data, which is worth knowing before you ship.

That re-run is a test whose two outcomes mean different things. The ranking is a function of two inputs: the outcomes the models produced, and the exchange rate somebody chose. Sweeping the rate across the band the risk owner would have accepted, with the outcomes held fixed, asks which of the two is producing the answer. A winner that holds across the band means no defensible choice of the number would have changed the decision, so the data is deciding. A winner that changes inside it means the data decided nothing and the assumption decided everything, which is a finding to report rather than a wobble to smooth over.

Keeping it from going stale#

An eval set decays in three distinct ways. Contract templates and regulations change, so items become unrepresentative. The set gets used for prompt iteration, so the system becomes tuned to it and its scores stop predicting anything — the same overfitting chapter 4 described, now happening to your own instrument. And candidates improve until everyone scores above 95%, at which point it has saturated and no longer separates anything, exactly as GLUE and MMLU did.

The third of those is this chapter's own argument arriving from the other direction: a set saturates when the candidates improve until nearly every item contributes the same amount to every total and drops out of every difference, which is the state a frequency-allocated set starts in.

The three defences map onto that loop. Rotating twenty percent per quarter, drawn from recent incidents, keeps the items current and slowly retires the ones every candidate now passes. Keeping a dev split that absorbs all prompt iteration and an eval split that is never tuned against preserves the eval split's meaning; when the eval split starts leaking into iteration, retire it and promote a fresh one. And versioning the set, with old versions kept runnable, is what lets you distinguish instrument drift from model drift — when this quarter's score drops, re-running last quarter's frozen version answers whether the system got worse or the test got harder.

The last of those three defences has the least obvious mechanism. A quarterly score is produced by a system and a test together, so a drop in it has two candidate causes that one number cannot separate: the system got worse, or the rotation swapped in harder items. One measurement, two unknowns. Re-running the frozen previous version holds one of them still, because that version is what it was to the byte, so a drop that shows up on it as well is a regression in the system, and a drop that vanishes there is a system standing where it was against a test that got harder. Neither reading is available without the archive, which is why retired items are versioned rather than deleted.

The argument that no benchmark should decide this#

Here is the case, stated as strongly as it deserves. The two errors this product can make are not commensurable: a missed clause is a failure to find something, and a fabricated clause is an assertion of a fact that is not true, entered into a record a regulator may read. Ranking two models by a single scalar requires an exchange rate between those errors, and choosing that exchange rate is a judgement about how much regulatory risk the organisation will accept. That is not a measurement. No amount of annotation makes it one, and a benchmark that reports a single number has quietly made the judgement on the organisation's behalf and hidden it inside a weight.

The case has two further parts. An eval set can only contain failure modes someone anticipated, and the tail that produces regulatory exposure is by construction the part nobody has seen yet — a set built from last year's incidents is well defended against last year. And the eval measures a model while the thing that ships is a system: retrieval, chunking, the review interface, the escalation path and the humans using it. A model that scores two points lower inside a system with a good escalation path can produce far better outcomes than the winner inside a bad one.

I believe the argument, and I believe it is narrower than it sounds. What it establishes is that the final selection is an accountable decision rather than an arithmetic one: the eval narrows the field to the candidates that clear the bar, a named person reads a sample of each finalist's outputs on the rare-clause and negative-control strata, and that person signs the choice off and owns it. The signature is about accountability, not about the number being untrustworthy.

Everywhere below that line, the number is exactly the right thing to believe. Whether last week's prompt change helped, whether a retrieval tweak raised recall without raising fabrication, whether a vendor's point release regressed anything, whether a cheaper model is close enough on the routine stratum to serve most traffic — those are all questions the set answers well, and answers cheaply, every time you ask.

That is what six weeks buys, and it is now the instrument on the weekly dashboard: run it on every prompt change before merge, run it when a vendor ships, run it when a new model tops the leaderboard and you want to know whether the shortlist should re-open. The leaderboard tells you when to look. This tells you what to do.