Benchmark scores and model cards

The quality column of a candidate comparison is the one with the most published material available and the least of it usable. A model card — the document a publisher ships alongside a release, describing what the model is, how it was trained and how it scored — will offer a table of benchmark results, and a public leaderboard will offer a ranking. MMLU, the most quoted single number in the industry, is a public exam of just over fourteen thousand four-option multiple-choice questions spread across fifty-seven academic subjects, from anatomy to formal logic, and a strong MMLU result is what most cards lead with.

Two properties of that format decide how much a difference between two MMLU scores can possibly mean, and both are visible before any model is involved. Four options per item means a model that reads nothing and answers at random lands near a quarter, so the scale does not start at zero and the usable range runs from about a quarter upwards rather than from nothing. And a fixed exam written once has a ceiling below a perfect score, because some fraction of its items are wrong — a point this chapter returns to with a figure. A gap of a couple of points is therefore a smaller share of the room available than it looks when it is read as a percentage of one hundred.

Verifying that such a number is real is a reasonable first move, and it answers a question about honesty rather than about relevance. This chapter is about the three questions that come after it, which fail in different ways and are checkable to different degrees: whether the number was produced under conditions comparable to anyone else's, whether the model had already seen the answers, and whether the benchmark bears any relationship to the work you are buying the model to do. The last of those is the one that cannot be fixed by better reporting, and it is where the chapter ends up.

The same benchmark, run four ways, gives four numbers#

A benchmark name is not an experimental protocol. MMLU alone can be run as a log-likelihood comparison, scoring which of the four answer strings the model assigns the highest probability, or as free generation parsed for a letter; with zero, five or twenty-five worked examples supplied in the prompt; with or without chain-of-thought, which lets the model write out intermediate reasoning before it commits to an answer; with the model's own chat template or with a bare completion format; on the full 14,042 questions or on a subset. The scripts that administer all of this are called evaluation harnesses, and the two in common use differ in how they build the prompt and how they parse the answer out of the reply. Those choices move the reported score by several points on the same weights, and the gap between the most and least favourable protocol is routinely larger than the gap between two models being compared.

Those effects are not noise, and treating them as noise is what makes the table look like a caveat rather than a mechanism. Each row is a different question being asked of the same weights. Scoring by log-likelihood asks which of four answer strings the model considers the most probable continuation, and every item yields an answer because one of the four always scores highest. Scoring by generated letter asks the model to write something and then asks a parser to find a letter in it, so an item on which the model writes a correct explanation in the wrong shape is recorded as wrong. The two protocols disagree about what counts as knowing the answer, and no amount of care in running either one reconciles them.

The chat-template row has the largest spread on the list and the clearest cause. An instruction-tuned model was trained on conversations wrapped in particular markup — a marker where the system text begins, a marker where the user's turn ends, a marker that starts the reply. That markup is what the model was taught to treat as a request for an answer. Hand it a bare block of text with no markers and it is being asked to continue a document rather than to answer a question, and the behaviour the instruction tuning installed does not switch on. The score that comes back measures a model in a mode nobody deploys it in. Worked examples in the prompt help for a related reason: they demonstrate the shape of a correct answer, so the model's most likely continuation is one that matches the shape the parser is looking for.

The practical consequence is the one the table is easy to read past. If the spread between the most and least favourable protocol is routinely wider than the gap between two candidates, then two numbers from two different cards cannot be subtracted. Whatever the difference between them is, it is smaller than the difference the protocol could have produced on its own, and the only comparison that survives is one where both numbers came out of the same harness at the same settings.

A model card that reports "MMLU: 82.1" and nothing else has told you a number and not a result. The version worth trusting looks more like the second block below than the first:

# what most cards publish
evaluation:
  mmlu: 82.1
  gsm8k: 79.4
  humaneval: 71.3

# what a decision needs
evaluation:
  mmlu:
    score: 82.1
    n_shot: 5
    method: generated_answer_letter
    chat_template: applied
    harness: lm-evaluation-harness v0.4.3
    build: bf16 weights, full 14042 questions
    reported_by: publisher (self-reported)
    independent_reproduction: 81.6 (third-party, same config)
  decontamination:
    method: 13-gram overlap against training corpus
    removed_from_training: 0.4% of MMLU test items detected and dropped

The second block can be argued with. It tells you which build was measured, which matters because the quantised version a marketplace serves is often not the one that was evaluated, and it distinguishes a self-reported figure from a reproduced one. Self-reported numbers are not automatically dishonest; they are produced by people who ran the evaluation many times while tuning the prompt and reported the best configuration, which is a bias that operates without anyone lying.

Two beliefs about that block are tempting and neither holds. The first is that an independently reproduced figure is a trustworthy figure. Reproduction fixes one thing only — that the number can be obtained again under the stated conditions — and it says nothing about whether the model had seen the items or whether the benchmark relates to your task. A reproduced score is a comparable score, not a relevant one. The second is that a self-reported figure is a suspect one. It is the best of many runs, which is a different and more predictable defect: it means the reported number sits at the top of that model's protocol range while a competitor's independently produced number may sit in the middle of theirs, so the two are not on the same footing even when both were honestly obtained.

Contamination invalidates the number rather than shifting it#

Everything above is about comparability. Contamination is about whether the number means anything at all, and it is the first thing to look for on a card reporting strong public benchmark results. Public benchmarks live on the open web — in GitHub repositories, in papers, in blog posts quoting example questions, in Hugging Face datasets, in forum threads where people paste the tricky items. A model trained on a large web crawl has very likely seen some fraction of any benchmark published more than a year before its training cutoff, and a model that has seen the test items is being scored on recall rather than on capability.

Why seeing an item helps is worth one step rather than being left implied, because the step also says why contamination cannot be corrected for afterwards. A model is built by being trained to continue text, so a passage that appeared in its training material has its own continuation as a high-probability output. An exam question that appeared there, with its answer key a line below it, is a passage of exactly that kind. Presented with the question at evaluation time, the model can produce the recorded answer by continuing a document it has already read, and none of the reasoning the item was written to probe has to happen. The output is indistinguishable from the output of a model that worked it out, which is precisely why no inspection of the answers can separate the two and why the check has to be made against the training data instead.

Detection is imperfect and the standard approach is n-gram overlap: take each test item, look for a matching span of 8 to 13 tokens in the training material, and either drop the overlapping items from training or exclude them from the reported score. The length of that span is a compromise between two failures. A span too short matches by coincidence — any ordinary phrase occurs somewhere in a web crawl, so short matches would flag most of the benchmark and remove material that was never contaminated. A span too long is defeated by trivial edits, since a single changed word breaks the match. Eight to thirteen tokens sits where a run of text is long enough to be unlikely by chance and short enough to survive minor reformatting. What it cannot survive is a real paraphrase, a translation, or a forum post that discusses an item in different words, so the method has a floor on what it detects and that floor is unmeasurable.

Overlap detection also works only if you have the training data, which for an open-weight release you usually do not, so a card that states its decontamination method is asserting something you cannot verify and a card that says nothing has not even asserted it. The observable proxy available to you is the gap between performance on a benchmark and on a fresh variant of the same task: if a model scores 82 on a 2021 benchmark and 61 on a 2025 re-authoring of the same question types, memorisation is the most economical explanation.

# the check you can actually run: does the model complete a test item
# it should have to reason through?
prompt = MMLU_ITEM["question"][:60]   # first 60 characters only, no options
out = generate(prompt, max_tokens=120, temperature=0.0)

# verbatim continuation of the remaining question text and the option list
# is evidence of memorisation, not of knowledge
print(similarity(out, MMLU_ITEM["question"][60:] + str(MMLU_ITEM["options"])))
# 0.94

A 0.94 similarity on a held-out test item means the model can recite it, which tells you the benchmark score for that item is measuring the wrong thing. Running this over a hundred sampled items takes minutes and produces a contamination estimate specific to the candidate in front of you.

What makes that a valid probe is worth stating, because the test looks circumstantial and is not. The model is given the opening of a question and nothing else — no options, no instruction, no hint that an exam is involved. The rest of a specific exam question, followed by that question's specific four options in their published order, is one continuation out of an enormous number of plausible ones, and there is no route to producing it by reasoning about the subject. Anatomy knowledge does not tell you which four distractors an exam author chose. Reproducing them means the sequence is stored, and a stored sequence is what contamination is. The probe is one-sided, which is its limitation and not a flaw: a high similarity is strong evidence of memorisation, while a low one only means this item was not memorised verbatim, and a paraphrased item would score low while still having been seen.

What else the card is not saying#

Benchmark tables absorb attention that the prose sections of a model card deserve more of, because the prose is where the constraints on your use case live. Language coverage is regularly stated as a list of supported languages without per-language evaluation, and a model listed as supporting 29 languages may have seen 92% of its pretraining tokens in English and perform two tiers below its headline on the language your customers write in. Safety and refusal behaviour is described qualitatively when it is described at all, and an aggressively tuned refusal policy is a functional defect for a security-tooling or medical-records product even though it registers as a virtue on a safety evaluation. The training cutoff bounds what the model knows without retrieval. The out-of-scope or intended-use section frequently contains the sentence that ends the evaluation, in the same way the acceptable-use policy did in chapter one.

Reading a card this way takes half an hour per candidate and produces a page of extracted facts rather than a feeling about the model. Keeping that page matters more than it seems, because the reasons a candidate was accepted or dropped are exactly what someone will ask about in eight months when the model is due to be reconsidered, and by then the card itself will have been edited. Cards are living documents: publishers add benchmark rows after the fact, revise licence text, and quietly update limitation sections. Recording the date you read it, alongside the facts you extracted, is the difference between a decision you can revisit and one you have to make again from scratch.

Construct validity and the pressure on public scores#

Suppose every caveat above comes back clean: the protocol is stated, the number is independently reproduced, decontamination is documented, the languages you need are evaluated separately. The score is now trustworthy, and it still does not tell you what you want to know, because MMLU measures four-way multiple-choice recall across academic subjects and your product does not ask the model to answer multiple-choice questions about anatomy. The technical name for the gap is construct validity: whether the thing being measured is the thing you care about. A benchmark with perfect internal rigour and no construct validity for your task is a precise measurement of something else.

The scenario above is a clean instance, and the two tasks are worth setting side by side to see where they part. An MMLU item hands the model four candidate answers, written out in front of it, and asks which one. The 34-way routing task hands it a schema of categories described once in the prompt and asks it to choose one of them without any of the alternatives being restated beside the input, then to emit a structured object that a parser will either accept or reject. One of those requires discriminating between four things that are present; the other requires holding thirty-four distinctions from earlier in the prompt and maintaining an output contract while doing it. They are both called classification and they exercise different capacities, which is why a model can be nearly level on the first and well behind on the second. Naming what a benchmark physically asks the model to do, rather than what the benchmark is about, is how the mismatch becomes visible before a migration instead of after one.

Some benchmarks are closer to some tasks than others, and the mapping is worth making explicit before you rely on any of them. Code-generation benchmarks correlate reasonably with code-generation products. Long-context retrieval benchmarks say something real about retrieval systems. Chat arena rankings measure which response anonymous voters prefer on open-ended prompts, which correlates with perceived helpfulness and tone and is close to uncorrelated with strict schema compliance or extraction accuracy. Nothing in the public set measures a 34-category routing schema, your document layouts, your terminology or your JSON contract, because nobody has your data.

There is also a pressure acting on every public benchmark that does not act on a private one. Leaderboard position sells models, so the scores are optimised toward directly and indirectly: instruction data is generated in benchmark formats, training mixtures are weighted toward the subject areas that benchmarks sample, and prompt templates are tuned against the public test sets because those are what everyone reports on. None of that requires cheating, and all of it erodes the correlation between the score and the general capability the score was supposed to stand in for. A measure that becomes a target stops being a good measure, and public benchmarks have been targets for several years. That is one reason a benchmark released last month is more informative than one released four years ago, and why the more useful signal in a comparison is often a model's rank on a benchmark that did not exist when it was trained.

The mechanism behind that saying is specific enough to be worth spelling out, because it explains why the erosion is permanent rather than something better methodology repairs. A benchmark works as a proxy when capability is what causes the score: a model that is genuinely better at reasoning over academic material gets more items right, so the number moves with the thing you wanted to know. Optimising the score directly introduces a second cause. The number now also moves with how much benchmark-shaped instruction data went into the training mixture and how carefully the prompt template was fitted to the public test set, neither of which has any bearing on capability. Two candidates separated by a couple of points may differ in the second cause alone. The proxy has not become inaccurate — it reports exactly what it always reported — it has stopped being evidence about anything except itself.

What benchmarks do well is eliminate. A model scoring 40 points below the field on code generation will not be a good coding assistant, and that is enough to keep it off a shortlist without spending a day on it. Used as a filter that removes candidates, published scores save real time; used as a selector that picks the winner, they transfer a decision about your product to a test somebody else designed for a paper. The distinction matters most where the scores are close, since a two-point separation between two models on a public benchmark is inside the range that protocol differences and item errors can produce on their own — MMLU itself has been shown to contain a few per cent of mislabelled or unanswerable items, which puts a ceiling on how finely it can discriminate.

So a shortlist can survive the model cards with three candidates still standing, and the published evidence has been reduced to what it can support, which is that none of them is obviously unfit. Choosing between them requires a test that measures your task, and the useful discipline is deciding what that test is — and what result would settle it — before anyone runs anything.