Reading a benchmark score

A public benchmark is a fixed set of test items — questions, coding problems, documents — released openly, each paired with an answer somebody has certified as correct, together with a stated rule for deciding whether a model's response counts as that answer. Anyone can download the items, run any model over them, and compute a percentage; a leaderboard is the table that collects those percentages for many models at once. Benchmarks exist because a language model arrives with no specification you can write tests against, which leaves two alternatives: try every candidate on your own work, at a cost of weeks per model in a field that ships new ones most months, or take a vendor's word about its own product. A shared set of items scored by a shared rule made models comparable to one another for the price of an afternoon of inference, and that comparability is the whole of what a benchmark sells.

You meet the same instrument in four places — a vendor's release post, a model card or documentation page listing scores against named competitors, a community leaderboard that re-runs dozens of models under one pipeline, and the procurement deck someone assembled out of the first three — and the names recur across all of them, a small roster that turns over as benchmarks saturate and get replaced. MMLU is Massive Multitask Language Understanding, GSM8K is Grade School Math with roughly eight thousand problems, GPQA is Graduate-level Google-Proof Question Answering, HumanEval is a set of hand-written Python exercises, and SWE-bench is a software-engineering benchmark built from real GitHub issues. Each of those names is attached to a percentage that gets quoted, compared and put into slides far from the machinery that produced it, and the failure this chapter is about is a reading failure rather than an instrument failure: an aggregate answers the question "how do these models compare on this distribution under this protocol", and it is routinely asked to answer "which one should we ship".

The three things a number is made of#

Ordinary software can be tested because its interface can be stated case by case. A function that parses a date has a nameable set of behaviours: this string yields that timestamp, this malformed one raises, this ambiguous one resolves by a rule somebody wrote down. Those statements are the specification, and a test suite is that specification made executable. A language model has no such interface. Its input is any string a user might type and its output is any string the model might produce, over a space of situations nobody can enumerate, so there is no finite list of behaviours to assert and nothing to point a test suite at.

What survives is a much weaker notion of correct, and it is the only one available: agreement with an answer somebody certified in advance. Fix a question, have a person or a trusted process decide once what the right answer is, and call the model correct there when its response agrees with that answer. This needs no specification, only a judgement made once and reused, which is exactly why it works on a thing that has none. A benchmark is a set of those judgements held still — items, their certified answers, and a rule for deciding when a response agrees — and the word carrying the weight is fixed. A fixed set of items is a sample, and a sample supports claims about the population it was drawn from and no claims at all about a population it was not.

Three decisions therefore get made before any model is run, and every number a benchmark produces is downstream of all three: what the items are meant to be about, which items go in, and what counts as agreement. Those are the capability, the slice and the protocol.

The capability is what the benchmark's authors set out to measure — the construct. MMLU measures recall and light reasoning over textbook knowledge across 57 subjects. GSM8K measures multi-step arithmetic word problems at roughly grade-school difficulty. HumanEval measures whether a model can write a short, self-contained Python function from a docstring. SWE-bench measures whether a patch generated against a real repository issue makes that repository's own tests pass. These are not interchangeable, and none of them is "intelligence". The distance between the construct a benchmark names and the thing it actually samples is what psychometrics calls construct validity, and it is the first place a leaderboard reading goes wrong.

The slice is the concrete distribution of items. MMLU's test set is 14,042 four-way multiple-choice questions, most of them one to three sentences long. GPQA Diamond is 198 graduate-level science questions. HumanEval is 164 problems whose reference solutions are usually under fifteen lines. Every one of those sets was assembled by people who had a budget, a source of items, and an idea of what a fair sample looked like — and the sample is what got measured, not the population you care about. Those counts describe a particular released version of each set. A benchmark can gain items, retire ones later found to be mislabelled, or publish a filtered subset that becomes the thing everybody quotes, and when that happens the same name is attached to a different sample.

The scoring protocol is the function from model output to a score. For multiple choice it might be exact match on a letter, or log-likelihood ranking, which scores each of the four options by how probable the model considers it and picks the highest without generating any text at all, or a substring search for the correct option text anywhere in the response. For code it is a test suite. For free text it is an n-gram overlap metric, counting how many short word sequences the response shares with the reference, or a model-as-judge, which is a second model prompted to grade the first one's answer. The protocol is where the largest surprises live, and chapter 2, on the evaluation harness, is about exactly that.

An aggregate score collapses all three into one number, which is what makes it convenient and what makes it lossy. The loss is not uniform: it hides the slices where models differ most, because those slices are usually small.

There is a pattern in how these three drift apart over time, and it is worth knowing because it explains why the top of a leaderboard is the least informative part of it. GLUE, the sentence-understanding benchmark that anchored the field around 2018, was superseded by SuperGLUE within a year because models had reached human baseline on it; SuperGLUE saturated in turn, and MMLU became the general-knowledge yardstick, and then frontier models crossed 85% on MMLU and its remaining headroom was mostly its own mislabelled items — a thing that only became checkable once people went back through the set item by item, which is itself something that happens to a benchmark near the end of its life. Each replacement was built because the previous one had stopped separating candidates at the top, which is the honest admission that a benchmark's useful range is bounded. When the leading scores sit within a few points of each other and within a few points of the ceiling, the ordering among them is being decided by the residue — ambiguous items, annotation errors, formatting luck — rather than by capability.

That sequence is dated by construction, and the dating matters more than the names in it. Whichever benchmark is the field's general yardstick while you read this has not yet saturated, and it will be replaced the same way for the same reason. Where a benchmark sits in that life cycle is therefore a condition on any reading of it: the same set separates candidates well early and badly late, which is not a defect in the set but what happens to a fixed sample when the things being sampled keep improving.

What the released examples tell you that the leaderboard does not#

Most reputable benchmarks publish their items, though which ones do is itself a fact with a date on it: a set that is open today may be withheld tomorrow, often precisely because it became popular enough to leak into training data, which is the pressure chapter 3 is about. Where the items are available, fetching them and running two candidates costs an afternoon and a few dollars of inference, and it converts a rank into evidence. The important part is not the total you compute — that comes in the next chapter — but the per-item record you keep on the way.

import json, pathlib
from statistics import mean

def evaluate(model_id: str, items: list[dict], generate) -> list[dict]:
    """Run one model over the benchmark's released items, keeping every field
    we might want to slice on later. `generate` wraps whichever provider SDK
    you are using and returns the raw response text."""
    records = []
    for item in items:
        prompt = build_prompt(item)          # the benchmark's own template
        output = generate(model_id, prompt)
        records.append({
            "item_id": item["id"],
            "subject": item["subject"],
            "gold": item["answer"],
            "raw_output": output,
            "prompt_tokens": count_tokens(prompt),
            "correct_exact": output.strip() == item["answer"],
            "correct_contains": item["answer"] in output,
            "parsed_as_json": try_parse(output) is not None,
        })
    return records

items = [json.loads(line) for line in pathlib.Path("test.jsonl").read_text().splitlines()]
for model in ("candidate-a", "candidate-b"):
    records = evaluate(model, items, generate)
    pathlib.Path(f"runs/{model}.jsonl").write_text(
        "\n".join(json.dumps(r) for r in records)
    )
    print(model, round(100 * mean(r["correct_exact"] for r in records), 1))

Three details in that loop matter more than they look. Storing raw_output rather than only the verdict means every slicing question you think of next week is answerable without paying for inference again. Storing prompt_tokens per item means you can ask the length question, which is the one that usually decides whether the benchmark tells you anything about your traffic. And computing two verdicts in the same pass — strict equality and substring containment — costs nothing and gives you a first reading on how sensitive the score is to the grading rule, which turns out to be the difference between a leaderboard number you can reproduce and one you cannot.

Here is what the slices look like for a public-benchmark run on two candidate models for a contract-review feature, using the benchmark's own items and its own prompt template.

The headline gap is real and it is also almost entirely earned on the short items, which are two-thirds of the set. On the forty-five longest items the measured ranking inverts by thirteen points, and on the rare label it inverts by sixteen. If your production traffic is short single-fact questions with a common label, the leaderboard is giving you good advice. If it is long documents where the interesting cases are rare, the leaderboard is giving you advice about somebody else's traffic and calling it a ranking.

Both readings come out of the table by arithmetic, and the arithmetic is worth doing rather than trusting, because it shows how a three-point headline and a thirteen-point inversion sit inside one set of items without either being wrong. The three length rows hold 806, 389 and 45 items, and those add to 1,240, the count on the headline row. Every item falls in exactly one of them and none falls in two, so the length rows partition the set rather than giving three overlapping views of it. That property is what the next step needs and it is not automatic: the rare-label row overlaps the length rows, since a rare-label item also has a length, and the schema-validity row carries no item count at all.

An accuracy is a count of correct items divided by a count of items, so a slice's correct count is its accuracy multiplied by its size. Because the length slices partition the set, adding those three correct counts gives the correct count for the whole set, and dividing by 1,240 gives the headline accuracy. Written that way, the headline is each slice's accuracy weighted by that slice's share of the items. The same holds for the other candidate with the same weights, because the slices are defined by prompt length, a property of the item and not of the model: both candidates were measured on the same 806 short items, the same 389 middling ones and the same 45 long ones. Subtracting one weighted sum from the other, weight by weight, leaves a weighted sum of the differences. The headline gap is each slice's gap scaled by that slice's share, and nothing else.

Putting the numbers in is a matter of sizes rather than decimals. The short slice is 806 of 1,240, about sixty-five items in every hundred, and five and a half points of gap at that share contributes a little over three and a half points. The middle slice is a little under a third of the set, and half a point of gap there contributes about a sixth of a point. The long slice is under four items in every hundred, so thirteen points running the other way subtracts just under half a point. Three and a half up, a sixth up, half a point down comes to about three and a quarter, which is the +3.3 on the headline row to within the rounding carried by the published percentages.

Said plainly, the headline is the short-item gap, shaved because a third of the items are not short, and then nudged by the rest. The long-item inversion is not being concealed by the aggregate, and nothing about the aggregate is dishonest. It is being outvoted. An aggregate counts every item once, so a slice's influence on the total is exactly its size, and a regime that is rare in the test set loses that vote however badly it goes. That is the non-uniform loss named earlier with a mechanism under it: the slices where models differ most are usually the small ones, and small is the same thing as outvoted.

A careful reader arrives here and writes "candidate A is the better model", and the sentence is worth pulling apart at the moment it forms. The headline row is the only row computed over everything: it leaves no item out, and completeness feels like generality, since a number that saw all of the evidence sounds like a number that covers all of the cases. It is complete over one distribution of items, and the row directly beneath it shows the same two models trading places when that distribution moves, on the very same responses. What the table supports comes with conditions attached, and they are the three decisions above: which capability, which items, which rule.

The two inverted rows need a condition of their own, and it cuts against reading either as a finding. On forty-five items a single item is worth more than two points of that slice's accuracy, since moving one from wrong to right moves the count by a forty-fifth. The thirteen-point gap is therefore six items — six long documents where candidate A is wrong and candidate B is right, beyond those where the reverse happens — and on the rare label, at thirty-one items, each item is worth over three points and the sixteen-point gap is five. A different forty-five long documents drawn the same way from the same source could as easily have produced five or six the other way. So the honest reading of those rows is that the ranking is not established in that regime, rather than that it reverses there. That does not rescue the headline. It leaves both candidates unmeasured on long inputs, which is a worse position to be in than a reversal, and it is why the right response to those rows is to measure the long regime rather than to switch models on the strength of them.

Behaviour on long inputs#

The items in that benchmark set averaged around 400 tokens, and the longest was 1,900. A product whose inputs run to several thousand tokens is therefore being ranked by a test set containing no item anywhere that exercises the regime it runs in.

The belief worth checking here is the reasonable one: that accuracy falls off gradually as inputs get longer, so a figure measured at 400 tokens is roughly the figure at ten thousand with a wider margin around it, and the only thing missing is confidence. That is how a quantity with a slope behaves — measure it at two lengths, and a reading a little beyond them is a question of how much error to allow rather than of whether the reading means anything. Applied to input length it is wrong, and wrong in a way that no amount of extra caution repairs.

This is not a quantitative extrapolation problem where the answer degrades smoothly and you can estimate the slope. Behaviour on long inputs is a different capability, in the sense that it is supported by different mechanisms and fails in ways short-input behaviour never does. A model can lose track of an instruction given at the top of a 10,000-token prompt while following the same instruction perfectly at 400 tokens. Retrieval accuracy from the middle of a long context is measurably worse than from either end, an effect stable enough across model families that it has its own name in the literature. Output-format compliance decays as generated output grows, so a model that emits clean JSON for a two-field answer starts dropping a closing brace on a forty-field one. None of these show up in a set whose items are 400 tokens long, because the set never puts a model in the situation where they occur.

That last clause is the mechanism, and it is worth taking literally, one failure at a time, because "never puts a model in the situation" is a much stronger statement than "does not test it thoroughly". An instruction that has to survive thousands of intervening tokens cannot fail in a 400-token item, because there are no thousands of intervening tokens for it to survive. A fact that has to be found in the middle of a long input cannot be missed in an item whose middle sits a sentence away from both of its ends. A closing brace forty fields after its opening one cannot be dropped from an answer that has two fields. None of the three is a harder version of something the short items already test; each needs a situation the short items do not contain. Their sizes are not constants to be looked up either: how much worse the middle of a long context is, and whether a given model shows the effect at all, is measured per model and per release rather than assumed.

The consequence is the difference between a measurement with a wide margin and no measurement. Widening an error bar is something you do to a number you already have, and a slice nobody ran has no number to widen. That is the sampling point from the start of the chapter arriving with teeth: a benchmark reports agreement between generated responses and certified answers on items somebody chose, so where nobody chose an item, nothing was generated, nothing was compared, and there is no verdict for a percentage to be computed from. Even the table's longest slice falls short of the product's regime, topping out at 1,900 tokens against traffic of 8,000 to 12,000, so the inverted row taken at face value is evidence about a regime an order of magnitude below the one that matters.

So the honest reading of a three-point gap measured at 400 tokens, applied to traffic an order of magnitude longer, is close to nothing. Not "probably still true but with more uncertainty" — close to nothing, because the ranking was established in a regime you never operate in, on a capability the benchmark does not sample at any point in its data.

The output format nobody scored#

The last row of the table is the one that most often goes unnoticed. That benchmark accepted any response containing the correct option token, so it never penalised a model for wrapping its answer in three paragraphs of preamble. A product that needs strict JSON matching a schema cares about the difference between candidate A's 91.3% schema-valid output and candidate B's 99.4%. In a pipeline where a parse failure means a retry, that is an eight-point difference in effective throughput and cost that the leaderboard, by construction, could not see. It was not that the benchmark measured this badly. It did not measure it at all.

"By construction" is the load-bearing phrase there, and it names a property of the scoring function rather than a lapse by the people who ran it. Substring containment asks a response exactly one question: does the certified answer's text occur anywhere inside it? That is the correct_contains line in the loop above, a membership test on a string, and its value depends on the presence of the gold token and on nothing else. A response that is the token alone and a response that is the token buried in three paragraphs of preamble, a hedge and a stray code fence return the same verdict, because the function reads nothing that could tell them apart. There is no threshold that was set too leniently and no penalty that was too small. There is no term for formatting in the function at all, and running it over more items or better models would not create one.

Every protocol named at the start of the chapter has that shape, and choosing one is choosing what will be invisible. Exact match on a letter penalises preamble absolutely, since one extra character makes the two strings unequal, while log-likelihood ranking generates no text at all and so has no formatting to observe. The habit worth forming is to ask, of any score handed to you, which properties of the response its scoring function read, because every property it did not read is one the score cannot distinguish however much the difference costs in production.

The throughput claim is itself an assertion until the calls are counted, and counting them takes one line of arithmetic. Candidate A produced schema-valid output on 91.3% of responses, so a strict parser rejects a little under nine responses in every hundred, which is roughly one call in eleven. Candidate B's 99.4% leaves rather more than half a rejection per hundred, or about one call in a hundred and seventy. In a pipeline where a rejection means a retry, each rejection costs a second full call at the same prompt and roughly the same output length. Per answer actually delivered, candidate A pays for about one and one-tenth calls where candidate B pays for a shade over one, and the eight-point gap in validity returns as about eight extra calls in every hundred — the same eight points, denominated now in spend and in throughput consumed.

That step is allowed under conditions worth stating, because the arithmetic is only as good as they are. It assumes the retry succeeds, which is fair when the failure is incidental — a dropped brace, a preamble the parser will not tolerate — since a second sample from the same model is a fresh draw and usually lands. It stops being fair when the failure is systematic: if some schema field is one the model reliably mangles, the retry reproduces the failure, and the cost is not one extra call but every call until something else catches it. It also assumes the failure is noticed, and a pipeline that accepts a partially parsed object instead of raising has turned a cost problem into a data problem, which is worse and appears on no bill.

And the call-count figure ignores latency, which is a separate axis rather than a refinement of the same one. A retry can only be issued after the first response has come back and been rejected, so the user behind that one call in eleven waits two full round trips instead of one. A per-call average of cost or throughput has nowhere to put that: eight extra calls in a hundred is a budget line, one user in eleven waiting twice as long is a product decision, and neither figure can be read off the other.

Clean measurements of the wrong slice#

None of this is an argument that the benchmark is broken or that its authors were careless. A well-built public benchmark is a careful measurement of a specific construct on a specific sample, and it is usually honest about which. The failure is in the reading, not the instrument.

It is also not the same problem as contamination. Contamination is the question of whether the test items were in the training data, which would make the number wrong even for the distribution it claims to cover. That is a real and separate concern and it gets chapter 3, on how a published test set becomes training data. Everything in this chapter holds even for a benchmark with a perfectly clean, never-published test set: an uncontaminated measurement of the wrong slice is still the wrong slice.

And it is not a claim that slicing is always available. Some benchmarks release only aggregate scores, or withhold the test set to resist contamination, and for those you cannot do any of this. That is a genuine limitation and it constrains what you can conclude — which is that you should conclude less, not that you should conclude the same thing with more confidence because the data was harder to get.

The instrument is not stable in time either, which is a condition on every leaderboard reading rather than an exception to one. The set of benchmarks in circulation turns over as the leading ones saturate, so the names in a procurement deck are the names of a moment. A public leaderboard recomputes its ordering as it adds models, and it sometimes changes its harness or its prompt template underneath the table, so two screenshots of one leaderboard a year apart are not guaranteed to be readings from the same instrument. A percentage quoted in a slide therefore carries a date whether or not anybody wrote one down.

A leaderboard percentage remains three decisions wearing the costume of a measurement: which capability, which items, which rule. You can recover all three in an afternoon, and the recovery changes what the number is allowed to decide — from "candidate A is better" to "candidate A is better on short items with common labels under a lenient extractor", which is a sentence you can check against your own traffic rather than adopt on faith.

What you have at the end of that afternoon is two per-item run files, a slice table, and a defensible reason to distrust a headline gap. The next thing that happens when you run a public benchmark yourself is that your total does not match the published one — and the size of that mismatch is usually larger than the gap you were trying to interpret in the first place.