Screening candidate models

Screening is the step where a field of candidates becomes a shortlist — the point at which most of the options are set aside without being tried, on evidence nobody claims is conclusive. There are somewhere between fifteen and thirty models a team might plausibly consider at any moment, across four or five vendors plus open-weight releases, and no team evaluates thirty of anything properly. Those counts describe the market as it stands at the time of writing, and they move every quarter as vendors ship, rename and retire; what the argument below needs from them is only that the field is several times larger than the number of models a team can afford to integrate, which has held through every reshuffle so far. The screen is what makes the rest of the process affordable, and a public leaderboard is a genuinely good instrument for it — which is a narrower claim than "leaderboards are useful" and a much stronger one, because it survives everything the previous three chapters established about slices, harnesses and contamination.

That survival is the point worth being careful about, because after three chapters on slices, harnesses and contamination the sentence a reader is most likely to write next is that public benchmarks are unreliable and should be ignored in favour of something built in-house. It is tempting for a good reason. Every one of those chapters showed a published number moving under something that had nothing to do with the model, and a measurement that moves under the measurer's own choices is not one to stake a decision on. The step that does not follow is the one from unreliable at the margin to unreliable at all. A score that cannot separate the second-best model from the fourth-best can still separate a model in contention from a model that is not, because the disturbances those chapters catalogued have a size, and that size is what decides which comparisons survive them.

Stated as a position rather than held as a suspicion, that belief also loses on the merits, and it is worth seeing how before defending anything in a room. "Leaderboards are unreliable because of contamination, so we ignore them and build our own" throws away the one thing public benchmarks do well, overstates a case that is real but bounded, and invites the entirely fair response that if the number is meaningless you should not have used it to pick the four candidates in the first place. The defensible position is that a screen and a decision are different instruments with different requirements, and that public scores meet one set and not the other.

What screening is worth#

Start with what the alternative costs. Integrating one model well enough to judge it — provisioning access, adapting the prompt to its formatting conventions, handling its error modes, running a hundred real inputs, reading the outputs — is two to three engineer-days. That figure assumes a team that already has a prompt, a set of real inputs and somewhere to run them; the first model integrated costs more, because it pays for the scaffolding the rest reuse, and a model whose interface differs in kind rather than in detail costs more again. Twenty candidates at two and a half days each is fifty engineer-days, which is not a two-week project, it is a quarter.

Reading a leaderboard is an hour. Both numbers in that comparison are worth working out rather than accepting, because the entire case for screening lives in the gap between them. Twenty candidates at two and a half engineer-days each is twenty lots of two days plus twenty lots of half a day, which is forty plus ten, or fifty engineer-days. If the leaderboard eliminates seventeen of the twenty, three candidates remain, and three at two and a half days each is six days plus a day and a half, which is seven and a half. An hour of reading has therefore converted fifty engineer-days into seven and a half. That is a very good trade, and it is the trade the leaderboard is actually built to support.

Saying the hour was free is the step that comparison usually leaves out, and it is not a remark about an hour being a small quantity. Free here means free relative to the alternative, and the alternative was never fifty engineer-days of careful evaluation, because a team that does not have fifty engineer-days does not spend them. It narrows the field some other way, and the other way is reputation: what a colleague mentioned, which vendor the company already has a contract with, which model was in the news that month. The honest comparison is therefore not screen against thorough evaluation. It is screen against guess, and a guess has no error rate anybody has ever measured.

The trade works because the errors a screen makes are cheap in the direction that matters, and that asymmetry is mechanical rather than a matter of temperament. A screen fails in two ways: it can let through a model that turns out to be bad, and it can eliminate a model that would have been good. The first is a false pass, and it is discovered by the very next stage, because the next stage is precisely a test of the candidates the screen let through. Its cost is bounded by the length of that stage — two days later, at the cost of two days — and it is corrected, since finding it out and fixing it are the same act.

The second failure never meets a stage that could detect it. Nothing downstream ever runs the rejected candidate on your inputs, so no measurement anywhere in the process will ever report that the screen was wrong about it. That is a difference in kind rather than in degree, and it is why the two error rates cannot be traded off against each other in the way error rates usually are: one of them is observed and bounded, the other is not observed at all. The consequence for how to set a screen is direct. Set the bar so that the error it makes is the recoverable one — let through a few candidates you suspect are weak, rather than tightening the bar until good candidates start falling out of it.

That choice is not free, and the usual telling of it makes it sound free. You are electing to accept a loss you have decided not to measure, and the reason you accept it is that measuring it would cost exactly the fifty engineer-days the screen exists to avoid: the only way to learn whether an eliminated model was secretly the best one is to integrate it and find out. As an analogy, this is the position a résumé screen is in — it rejects people who would have been excellent, knowingly, because reading two thousand résumés carefully is not a thing anyone does. The analogy breaks at the place that matters here. A hiring market eventually reveals some of its false rejections, because the person who was passed over is hired elsewhere and does the job in public; a model screened out is never run on your inputs by anyone, so there is no elsewhere in which the mistake could surface. The invisibility is more complete than the hiring picture suggests, and the tolerance has to be correspondingly more deliberate.

A screen only has to be roughly right about the tail — the far end of the field, where the candidates that are clearly not in contention sit — and public benchmarks are: the correlation between headline scores and general capability is loose but genuinely positive, and a model twenty points below the field on four different benchmarks is not secretly the right choice for your document extraction pipeline. Loose is sufficient at the tail because the tail is where the differences are large, and a loose relationship still gets the sign of a large difference right. What it cannot do is order two models that are close together, which is the boundary the rest of this chapter keeps arriving at from different directions.

Screening well is mostly about reading magnitudes rather than ranks. A model that sits fourth instead of second tells you nothing; a model that is fifteen points below the leaders on every board you look at tells you plenty. Those two readings differ because of a size comparison, and the previous chapters supplied both sides of it. Chapter 2, on the evaluation harness, showed one model's total moving under nothing but the harness — the prompt template, the number of worked examples in front of the question, whether reasoning was elicited, how an answer was extracted from the response — by more than the distance that separates adjacent entries on a board. Chapter 1, on reading a benchmark score, showed the same total moving again with the composition of the slice of items it was computed over.

A difference of one or two places is therefore inside the machinery's own error, and reporting it as a finding is reporting the harness. A difference of fifteen points is larger than anything the machinery has been shown to contribute, so it is still there after every disturbance those chapters catalogued has been subtracted from it. That is the whole reason a screen remains trustworthy after three chapters of demolition: the demolition has a magnitude, and screening deliberately works above it.

It also helps to screen on boards measuring genuinely different constructs — a knowledge benchmark, a code benchmark, an instruction-following benchmark — because a candidate that is weak across three unrelated constructs is weak in a way that is unlikely to be an artifact of one harness. Three boards that all measure textbook knowledge under near-identical protocols give you far less than their count suggests. Chapter 5, on rubric weights and abstention, makes precise why correlated evidence does not accumulate, and the one-line version is available now: boards measuring the same construct under similar protocols agree with each other before any particular model is run on them, so the second and the third mostly repeat what the first already said. Boards measuring genuinely different constructs do add, for a reason worth stating as a mechanism — a quirk of one harness, an extractor that mis-parses a particular answer format or a slice that happens to be short, has no reason to reproduce in a harness other people built for a different capability.

And a good screen is not made of benchmark scores alone. Context window, published latency, price per million tokens, regional availability, data-retention terms and whether the vendor offers batch processing are all screening criteria, they are all published, and several of them eliminate candidates faster and more definitively than any accuracy number. A model that cannot accept a 12,000-token prompt is not a candidate for a contract-review product regardless of where it ranks, and finding that out costs a glance at a table.

A decision is a different instrument with different requirements. It has to be right about the margin — about which of three roughly comparable models is better for your inputs — and it has to be right in a way that survives being wrong, because you will build six months of prompt engineering, cost modelling and failure handling on top of the answer. Everything the previous chapters established is precisely a catalogue of reasons a public score is unreliable at the margin. None of it makes the score unreliable at the tail.

Six points, and two decisions#

The concrete form of the question is: model A is six points above model B on a public benchmark; name a decision you would make on that and a decision you would refuse to.

Three properties separate the two halves of that table. Reversibility: if being wrong costs two days and you find out in two days, a weak signal is enough. Distance from the measured distribution: the closer your inputs are to the benchmark's items in length, format and difficulty, the more the score transfers, and the contract-review case from chapter 1, whose inputs ran an order of magnitude longer than the benchmark's items, sat about as far away as it is possible to sit. Whether a better instrument is available at acceptable cost: refusing to decide on six public points is only reasonable if you are proposing something better.

Those three run as a procedure on a decision of your own, and the order matters because the first two can be answered from facts already in hand. First, reversibility: write down what being wrong costs and when you would find out. If the cost is bounded by a stage that is going to happen anyway and the discovery is automatic, a weak signal is enough and there is nothing further to do. Second, distance: put your inputs beside the benchmark's items on length, format and difficulty, the three properties chapter 1 showed a score to be sensitive to. If they differ on any of the three, the score's transfer to your case is a claim somebody has to argue for rather than the default. Third, availability: name the better instrument, price it, and count the decisions it will also serve.

Availability is the step that does most of the work, and it is what makes the position defensible rather than merely sceptical. The argument is not "the leaderboard might be wrong". Every instrument might be wrong. The argument is that for this specific decision a materially better instrument costs two weeks and will be reused for every model evaluation the team does for the next two years, which makes the marginal cost of the second decision nearly zero.

Skipping that third step is what turns the position back into scepticism. Reversibility and distance between them can only establish that six public points are weak evidence, and weak evidence still beats no evidence: a team told the number does not transfer, with nothing offered in its place, will use the number anyway, and will be right to. Refusing to decide on weak evidence is a position only when something better is on the table with a price on it, which is why the third question ends in a cost and a reuse count rather than in a verdict.

When the screen stays on the dashboard#

The screen ends when the shortlist does. What frequently happens instead is that the benchmark stays on the dashboard: the team picked one model six weeks ago, they are now iterating on their own prompts, and every Monday someone re-runs the public benchmark and reports the number.

This is worse than useless, and it is worth being specific about why rather than invoking Goodhart's law — the observation that a measure stops being a good measure once it becomes a target — and moving on. Consider what raises a score on a multiple-choice benchmark scored by first-token extraction. Instructing the model to reply with a single letter and no explanation raises it, because the extractor stops mis-parsing. Removing the schema instructions from the system prompt raises it, because they consume attention the benchmark does not reward. Lowering reasoning effort raises it on the items where the model overthinks. Every one of those changes is a real, measurable improvement against the public number, and each one is a small deletion from the product.

The mechanism is not that measurement corrupts, which is what invoking the law and moving on leaves a reader holding. Nothing in that list involves anyone gaming anything, and an honest engineer with no incentive at all would arrive at the same three changes by trying to make a number go up. What is actually happening is a disagreement about content. Benchmarks and products can disagree about what a good response contains, and a scoring rule is a statement of one side of that disagreement: a first-token extractor reads a letter and discards the rest of the response, which is a way of saying that the rest is worth nothing. Optimising against a rule that assigns no value to a field is indistinguishable, in its effect on the output, from deleting that field, because by the rule's own arithmetic the two score the same.

# Variant A — tuned against the public benchmark
system: "Answer with a single letter. Do not explain."
→ public benchmark: 74.3%  (+3.1 vs baseline)
→ product eval:     61.0%  (−12.4; extraction pipeline needs the JSON
                             object and the per-field confidence values,
                             both of which this prompt suppresses)

# Variant B — tuned against the product eval
system: "Return a JSON object matching the schema. For each extracted
         clause include a confidence value and the source span."
→ public benchmark: 68.9%  (−2.3)
→ product eval:     79.6%  (+6.2)

Walk Variant A one link at a time, because those two variants are this chapter's proof and the numbers on their own do not show why it had to come out that way. The public benchmark's extractor reads a letter out of the response and discards everything else. An instruction to answer with a single letter and no explanation therefore removes everything the extractor would otherwise have had to skip past, so responses that were being marked wrong for their packaging start being marked right. The model has learned nothing at all between the two runs. This is the five-shot effect from chapter 2, on the evaluation harness, wearing a second costume: there too a score rose because responses started arriving in a shape the extractor could read, and not because the subject was better understood.

Now read the same instruction from the product's side. The extraction pipeline needs a JSON object, a confidence value for each extracted field and the source span the value came from, and every one of those is made of tokens that "answer with a single letter, do not explain" suppresses. The instruction that cleared the extractor's obstacles cleared away the product's payload, because they are the same tokens. The three-point gain and the twelve-point loss are therefore not two results. They are one edit, read by two instruments that want opposite things from the same response, and Variant B is that edit run backwards — it spends a couple of points on the board to buy the fields back.

Every point gained after the shortlist closed was a point earned on somebody else's task, and in that pair it was paid for out of the task the team is actually shipping. The benchmark's scoring quirks became design constraints on a product that has none of those quirks.

There is a second, quieter cost. A public leaderboard's numbers move for reasons that have nothing to do with your system: the operators update their harness, a vendor ships a point release under the same model alias, new entrants change the ranking without changing any score, or the board re-runs with a different sampling seed. Tracking that weekly produces a stream of movements that look like signal, and each one generates a meeting. The team's attention is a real budget, and this spends it on somebody else's release schedule.

Which of those four a given board actually does is a property of that board and of the month you are reading it in. Operators publish their versioning and re-run policies, they revise those policies, and some freeze a harness for a season while others change it continuously. So the four are a list to check against a specific board's published practice before reading any movement as news, rather than a fixed account of how leaderboards behave.

Triggers, and the instrument you build instead#

Nothing here is an argument for ignoring public benchmarks, and if the position gets stated that way it will lose to the CTO on the merits. Screening twenty models down to three is genuine value delivered for an hour of work, and no private eval replaces it — building your own eval for twenty candidates would mean integrating twenty candidates, which is the cost the screen exists to avoid.

It is also not an argument against watching leaderboards after the decision. New models appear, and a release that tops several boards by a wide margin is a legitimate trigger to re-open the shortlist and run your own eval again. Watching for a trigger and tracking a metric are different activities: the first is a threshold that occasionally fires, the second is a weekly number someone is implicitly being asked to move.

The difference between those two is not a matter of framing, and it can be traced. A threshold costs attention only on the occasions it fires, because between firings there is nothing to read and no state to carry; its whole output is a decision to go and look, taken a handful of times a year. A number on a recurring report costs attention every week, and it does something a threshold cannot: a number reported to the same people on the same day is a number somebody is implicitly accountable for, and accountability creates pressure to move it. That pressure has exactly one available outlet, which is the part that settles the argument. You do not control the vendor's model, so the only thing you can change to move a public benchmark score is your own prompt — which lands you back in the block above, where moving the public number cost the product twelve points.

And it is not a claim that private evals are unbiased. A private eval built from your own traffic has its own contamination problem — the items are drawn from a period when a particular model was in production, so they reflect what that model handled and what your users learned to ask it. It has its own slice problem, its own scoring quirks, and its own staleness. Everything the first three chapters did to public benchmarks applies to the instrument you build to replace them; the difference is that you control it and can audit it, and chapter 7, on building the set that decides, is about making it survive that audit.

The rule that comes out of all this is a boundary rather than a verdict. A public benchmark is trustworthy for exactly as long as the question is which candidates are worth an engineer's time, and it stops being trustworthy the moment the question becomes which of two survivors to ship. Keeping those two questions apart is most of the discipline, because the same number answers one of them well and the other one badly, and nothing on the page tells you which question you are asking.

The immediate question is smaller and sharper. You have the shortlist, you have two engineer-weeks, and you build the eval. Then you run both finalists on it and it tells you the opposite of what the leaderboard did.