Rubric weights and abstention

A reversal is a specific and diagnosable event: two models that a public board separates in one direction come out separated in the other direction on a set of your own cases, run with the same prompts, the same data and the same scoring. The first hypothesis should always be a bug — the wrong model string, a cached response, a rubric applied asymmetrically — and chapter 6, on auditing your own instrument, is the procedure that rules those out before anyone acts on a number.

This chapter is about the case where the audit comes back clean and the reversal is real, which is common. The explanation for it is almost always the same shape, and it is not that the benchmark was contaminated or badly built. It is that an aggregate scored over items with a guaranteed correct answer cannot see a behaviour that only appears when there is no correct answer, and that behaviour can be worth more to a product than everything the benchmark did measure.

Decomposing the reversal#

Both instruments hand you a single number per model, and a single number is the one form in which a disagreement cannot be examined. An aggregate is several separately measured behaviours brought together into one figure, and the bringing-together is where the information goes: the figure that comes out does not say which of those behaviours moved it.

An aggregate becomes diagnosable the moment its components are scored separately. Here is a support-ticket rubric with four components, weighted by what a bad outcome costs the support organisation, applied to two finalists on 300 real tickets. A rubric, here, is the list of behaviours being scored; a weight is how much one of those behaviours counts relative to the others, so the row carrying 0.30 counts three times as much as the row carrying 0.10.

The last column is the only one in the table that is not a measurement. Each of its entries is arithmetic on the two columns to its left: take the difference between the two models on that row, multiply it by that row's weight, and label the result with whichever model the difference favours. Work the abstention row to see it happen. Model A abstains correctly on 28.9% of the unanswerable tickets and model B on 88.1%, so the two are a bit under sixty points apart on that behaviour. Three-tenths of a bit under sixty is a little under eighteen, which is the B +17.8 the table prints. Scaling the difference by the weight is allowed because the weight is applied to each model's component score before anything else happens to it, and taking three-tenths of each score and then subtracting gives the same answer as subtracting first and taking three-tenths of the difference.

Running the same two steps on the lightest row shows the operation at the other end of its range. On domain vocabulary the two models score 73.5% and 79.8%, a difference of just over six points, and that row carries a weight of 0.10. A tenth of just over six points is a little over six-tenths of a point, which is the B +0.6 in the last column. What the pair of calculations says in plain terms is that a component moves the standings by its weight multiplied by its gap, and that neither factor decides anything on its own. The heaviest row in this rubric is answer correctness at 0.40, and its entry in the last column is smaller than every row's but the lightest, because the two models are barely separated on the behaviour it scores. A small weight on a large difference outranks a large weight on a small one, which is exactly what happened here, and it is why the reversal stops being mysterious as soon as the components are visible.

Model A is genuinely better at the thing the benchmark measures. On the 234 tickets whose answer is in the knowledge base it wins by 2.6 points, which is roughly its four-point public lead arriving as expected. It loses the eval on the other 66.

Those two counts are a split rather than two separate facts. Two hundred and thirty-four answerable tickets and sixty-six unanswerable ones together are the whole of the three hundred, and every ticket sits in exactly one of the two groups, because what puts it there is a single label: whether the knowledge base contains what the answer needs. The correctness row then follows from the same two-step as the rows above. The models are 2.6 points apart on the answerable group, four-tenths of 2.6 is a little over a point, and that is the A +1.0 in the table.

What the split buys is a precise statement about where the two instruments agree. Every item on the public benchmark has a correct answer, which makes each of them an item of the same kind as one of the two hundred and thirty-four, and none of them an item of the same kind as one of the sixty-six. So on the part of the eval the benchmark is capable of seeing, the benchmark and the eval do not disagree at all: the public lead arrives, in the right direction and at roughly the right size. The whole of the disagreement is produced by the part of the eval for which the benchmark has no items, and an instrument that agrees with you everywhere it is able to look is not a broken instrument.

Those 66 tickets are the ones where the knowledge base has nothing: a customer asking about an unreleased feature, a billing case that requires account-specific data the retriever — the search step that pulls knowledge-base passages into the prompt — cannot reach, a question whose premise is wrong. The correct behaviour is to say so and escalate. Model A produces a confident, fluent, plausible answer on 71.1% of them. Model B declines on 88.1%. In production those two behaviours give you two different systems: one that occasionally routes a hard ticket to a human, and one that tells customers things about the product that are not true.

Model A's 71.1% is not a fifth measurement. It is the abstention row read the other way round. A model handed an unanswerable ticket either declines it or answers it, and the scoring recognises no third outcome, so the two proportions have to account for the whole of the sixty-six between them. Model A declines a bit under three tickets in ten, which leaves a bit over seven in ten on which it produces an answer — and an answer to a question whose answer is not in the knowledge base is an answer the model made up. Model B's fabrication rate comes off its own abstention figure the same way, and it is a bit under one ticket in eight.

No widely cited public benchmark scores abstention well, and the reason is structural rather than an oversight. A four-way multiple-choice item has a correct answer by construction, so there is nothing to abstain from; a model that replies "I cannot determine this from the given information" is scored wrong. Benchmarks with unanswerable items exist, and they are a minority of what appears on general leaderboards. So a model tuned to always produce an answer is rewarded by the benchmark's design and punished by yours, and the sign of that effect is large enough to swamp a four-point capability difference.

The chain from that construction to a model's behaviour is worth walking a link at a time, because it is what makes the reversal predictable rather than surprising. An item on a forced-choice benchmark is written together with its key, and it is only usable as an item if exactly one of its options is the correct one. That makes "none of these can be determined from what I was given" a response the scoring has no way to mark right, whatever the item happens to be about. A model that produces it is recorded as having failed — and recorded as having failed in exactly the same way as a model that picked a wrong option with complete confidence. Aggregate enough items scored like that and the measurement has no channel left in which a model that knows it does not know can look different from one that does not.

The material a model is trained on has the same shape. Preference data is usually collected by showing people two candidate responses and asking which is better, and a refusal is rarely rated above a fluent answer that appears complete, since the rater cannot check the claim any more easily than the reader of a support ticket can. The cost of the default all of that selects for is the thing to hold on to before the capability that avoids it gets a name: a system that always produces an answer will produce one when it has nothing to produce it from, and what comes out is specific, confident and unconnected to anything the system actually holds.

Model A is not defective here. Its behaviour is what its training rewarded: forced-choice evaluations, helpfulness-oriented preference data, and a long history of benchmarks in which declining to answer is indistinguishable from failing to answer all push in the same direction. Calibration — knowing when you do not know — is a capability that has to be trained and measured deliberately, and a model whose developer prioritised it will look slightly worse on the boards and considerably better on a support queue. Two models can differ on this axis by sixty points while differing by four on everything a leaderboard reports.

The practical consequence is that abstention is cheap to check during the screen, and almost nobody does. Twenty unanswerable items, hand-written in an hour, run against every shortlisted model, will separate the candidates on this axis before any integration work starts. The items do not need to be sophisticated: a question about a feature that does not exist, a request that requires data the system cannot see, a question with a false premise. What you are measuring is whether the model's default under uncertainty is to produce or to decline, and that default is remarkably stable within a model and remarkably variable between models. What twenty items report is the default at whatever prompt and settings you ran them with, not a ceiling: whether an instruction to decline, or a schema with an "insufficient information" branch in it, can move that default is a separate experiment, and one worth running on any model you would otherwise drop.

The other two components are the same story with smaller numbers. Schema validity was invisible to the lenient extractor that scored the public benchmark, as chapter 1's slice table already showed. Domain vocabulary — product names, plan tiers, internal status codes — appears nowhere in a general knowledge benchmark, and a model that renders "Enterprise Plus" as "Enterprise+" breaks a downstream string match.

Writing the weights down makes the disagreement legible#

The weights column is doing the real work, and it is the part teams most often leave implicit. Making it explicit converts a clash of instruments into a claim about priorities that someone can argue with on the merits.

from dataclasses import dataclass

@dataclass
class TicketResult:
    answerable: bool          # does the KB contain what is needed
    abstained: bool           # did the model decline / escalate
    answer_correct: bool      # judged against the reference answer
    schema_valid: bool
    vocab_correct: bool

def score_ticket(r: TicketResult) -> dict[str, float | None]:
    """Components are None where they do not apply, so the aggregate
    averages each one over its own denominator rather than over 300."""
    return {
        "correctness": (r.answer_correct and not r.abstained)
                       if r.answerable else None,
        "abstention": r.abstained if not r.answerable else None,
        "schema": r.schema_valid,
        "vocab": r.vocab_correct,
    }

WEIGHTS = {"correctness": 0.40, "abstention": 0.30, "schema": 0.20, "vocab": 0.10}

def aggregate(results: list[TicketResult]) -> float:
    scored = [score_ticket(r) for r in results]
    total = 0.0
    for name, weight in WEIGHTS.items():
        vals = [s[name] for s in scored if s[name] is not None]
        total += weight * 100 * sum(vals) / len(vals)
    return total

Two design choices in that function are worth defending. Correctness and abstention are scored on disjoint subsets, each with its own denominator, so a model cannot raise its correctness score by abstaining on hard answerable tickets — the abstention would simply not count as correct. And a confident wrong answer on an unanswerable ticket scores zero on abstention rather than being ignored, which is the minimum needed to make fabrication visible. Chapter 7, on building the eval set, goes further and argues that on some products a fabrication should score negative, because a miss and an invention are not errors of the same kind.

Both of those choices are floors rather than refinements, and what they are floors above is visible in what an eval without them would report. Take the denominators first. Suppose correctness were scored across all three hundred tickets, with a ticket counted correct when the model produced the reference answer, incorrect when it produced something else, and passed over when the model declined to answer at all. A model facing an answerable ticket it is likely to get wrong then has a move available to it: decline. The declined ticket is passed over rather than counted wrong, so the average is taken across the tickets that remain, and the model's correctness score rises on the strength of having answered less. Splitting the denominators closes that route. Correctness is averaged over the 234 answerable tickets, where a decline is scored as a failure to get the answer right, and abstention is averaged over the 66 unanswerable ones, where declining is the behaviour being credited and is credited nowhere else.

Scoring a confident wrong answer as a zero on abstention is a floor in the same sense. The lenient alternative is not a harsher rule but no rule: dropping unanswerable tickets from the scoring altogether on the grounds that there is no reference answer to compare an output against. That is a defensible-sounding decision, and it deletes the entire behaviour from the eval, leaving you with a private instrument carrying the same blind spot as the public one you built it to escape. A zero is the least a rubric can charge for a fabrication and still have fabrication appear in the number at all.

Why three more public benchmarks do not break the tie#

The natural proposal when two instruments disagree is to get a third opinion: run both models on three more public benchmarks and go with the majority. It is a reasonable instinct and it does not work, for two separate reasons.

The first is that additional public benchmarks add evidence about other people's tasks. The disagreement is between your distribution and theirs. Resolving it by collecting more of theirs is answering a different question more loudly — and if the tie-break comes out 3–1 for model A, the only thing established is that A is better on four distributions that are not yours, which was never in dispute.

The second is that four public benchmarks are much less than four measurements. Popular general benchmarks share their source material (web text, textbooks, academic exams), their item format (short, self-contained, single-turn), their scoring philosophy (a gold answer exists and the model must produce it), and their contamination history (published on GitHub, mirrored, discussed in tutorials, folded into instruction mixes). Model scores across them correlate strongly — pairwise correlations in the 0.8 to 0.95 range are typical across a field of current models — which means the fourth benchmark's result is mostly predictable from the first three before you run it.

The reason count is the wrong thing to look at, and independence the right one, is a mechanism rather than a convention. Two measurements of the same quantity beat one when each can go wrong in a way the other cannot: the first may be thrown by an item somebody worded ambiguously, the second by a scorer that mishandles a formatting variant, and because those two failures have unrelated causes, agreement between the two results is unlikely to be a coincidence and therefore tells you something. Every step of that argument rests on the causes being unrelated. Instruments built from the same source material, cut to the same item format, marked by the same scoring philosophy and exposed to the same training corpora do not have unrelated causes of error. They have one set of causes, wired into all of them.

A correlation in the range those boards sit in has a plain-words reading, and it is stronger than most people expect. Line a field of models up by their score on one board and then by their score on another, and the two orderings come out nearly the same ordering, with most of the small disagreement falling among models that were close together to begin with. Given three such boards, a well-informed guess at where a model lands on the fourth is close to right before anybody runs the fourth. A result you could have written down in advance carries no information when it arrives, since what makes a measurement worth taking is the chance that it comes back other than expected.

Correlated evidence still counts for something — it is not zero — but treating benchmark agreement as a vote assumes the voters are independent, and these voters read the same newspaper. If you want a genuine second opinion on the disagreement, the productive move is a second private measurement that varies something you control: a different sample of tickets, a different rubric author, a human preference comparison on fifty outputs. Each of those varies a source of error your eval might have. Another leaderboard varies none of them.

Each of the three is aimed at a different source. A different sample of tickets varies which cases you happened to draw, so it tests whether the gap belongs to your traffic or to the particular 300 tickets somebody pulled out of last quarter. A different rubric author varies the judgement that turned behaviours into components and weights, so it tests whether the finding survives someone else reading the cost of a bad outcome differently. A human preference comparison varies the scoring machinery itself, replacing the rubric with people looking at outputs, so it tests whether the rubric measures what its author meant it to.

The newspaper is a reframe and not a proof, and it is worth saying where it stops. Voters who read the same paper still disagree with one another, and these boards do disagree at the margins — a model can place a little higher on one than on another. What the shared source removes is not their disagreement but the particular thing you were hoping to buy with a fourth reading: the chance that the fourth board is wrong about your two models in a way the first three were not. Nothing in the picture says any of the four boards is inaccurate. A newspaper can be entirely truthful and four readers of it will still not give you four independent opinions.

The sample-size objection#

The stakeholder version is: their benchmark has 14,042 items and yours has 300, so theirs is obviously the more reliable measurement.

It is the objection a reader is most likely to arrive at unprompted, and it is tempting for a good reason: the rule it appeals to is a real rule. Given two estimates of the same quantity, the one built from more items is the better estimate, and that holds regardless of what is being estimated. The words doing the work are of the same quantity, and they are the words the objection drops. Both instruments here return something that looks like a score out of a hundred, and the shared units are what make the comparison feel available; units are not a statement about what was measured.

Sample size and relevance are different axes, and the objection conflates them. A very large sample gives a very precise estimate of the quantity it measures. If that quantity is accuracy on short multiple-choice questions and your quantity is weighted quality on 8,000-token support tickets with abstention, then their precision buys nothing about yours, because the two quantities are not the same quantity. Precision about the wrong thing does not become accuracy about the right thing by getting more precise.

Precision is worth defining, since it is the whole of what the extra items buy. An estimate is precise when repeating the measurement would land in much the same place: draw another fourteen thousand items of the same kind, score the same models on them, and the answer barely moves. Nothing in that sentence says anything about what is being measured. Precision is a property of how much an answer would wobble on a second run, and relevance is a property of whether the quantity being answered is the one your decision turns on. Different things set them: precision is set by how many items were scored, relevance by what the items are of. Adding items to a very precise measurement of the wrong quantity makes it a more confident statement of something you were not asking.

That said, the objection is half right, and the half it gets right is worth conceding. Three hundred items is a small sample, the nine-point gap has a confidence interval attached — a range of values the true gap plausibly takes given how few items you measured it on — and if that interval spans zero the reversal is a story rather than a finding. The honest response is not to dismiss the concern but to answer it with numbers: here is the run-to-run variance, here is the bootstrap interval on the gap, here is why nine points clears it.

Both of those are measurements of your instrument rather than of the models. Run-to-run variance is what you get by scoring the same models on the same tickets more than once and watching how far the totals move while nothing about either model has changed; if that movement is itself several points, the gap is inside the instrument's own noise and there is nothing further to discuss. A bootstrap interval is built by drawing a new set of 300 tickets at random from the ones you already have, allowing the same ticket to be drawn more than once, rescoring both models on it, and repeating that many times to see the range the gap falls in — which turns "three hundred is not many" from a worry into a printed interval. Chapter 6, the audit, is where both of those get produced, and the scepticism that prompts them is better aimed at your own eval than reserved for the leaderboard.

A clean benchmark measuring something else#

A reversal is not evidence that the public benchmark is contaminated or badly built. It is fully compatible with the benchmark being a clean, honest measurement of a construct that is not the one you are buying. The perturbation and completion probes from chapter 3 exist to test contamination specifically; a disagreement between two instruments measuring different things is not one of those probes.

It is also not a general claim that model B is better than model A. On answerable tickets A won. If your product's failure mode were "the assistant refuses too often" rather than "the assistant invents policy", the same two runs would rank them the other way with no change to the data. The rubric weights are a statement about your costs, and a different product with the same models and the same tickets can correctly reach the opposite conclusion.

And it is not permission to trust the nine points yet. The private eval is an instrument you built, and every source of error you have spent four chapters finding in other people's instruments is available in yours. The prompt was written by someone, the rubric was written by someone, the sample was drawn by someone, and each of those is a place a nine-point gap can come from without either model being better.

What the decomposition buys, before any of that is settled, is a shift in what the disagreement is about. Two instruments returning different orders is not a puzzle to be resolved by finding the more authoritative instrument; it is two different questions returning two different answers, and the weights column is where you say which question you are asking. A team that can point at 0.30 and defend it is having an argument about product priorities, which is the argument worth having. A team that cannot is having an argument about whose number is bigger.