The evaluation specification

Three plausible candidates and an argument about which is better is a conversation with no terminating condition. The evidence available is abundant, cheap to cite and weakly related to any particular task, so each participant can weight the evidence that supports the conclusion they already hold, and none of it is decisive. Two features of this landscape make the drift worse than it would be for, say, a choice between message brokers: releases arrive weekly, so an open question is guaranteed to acquire new candidates while it stays open, and each arrival resets whatever consensus had formed in the meantime. The second feature is that the public scores of the previous chapter are abundant enough and cheap enough to cite that an argument conducted in leaderboard positions can be sustained indefinitely on both sides.

A shortlist becomes resolvable when its deciding test is named first, and the remedy is therefore a document written before the first evaluation call is made. An evaluation specification states the test data, the thresholds a candidate must clear, the budget it must fit inside, and what happens when two candidates come out close together. The discipline is borrowed from clinical trial pre-registration, where the outcome measure, the population and the analysis plan are published before enrolment begins.

Three things a deciding test has to name#

The test data comes first because it is the part people skip. "We'll try them on some real examples" is not a test set: it is a sampling procedure with no defined population, chosen after the fact by whoever runs the trial, which means it can be argued with forever. A test set is a fixed, frozen collection of inputs drawn from a stated population with stated labels, and once it is frozen nobody adds examples because a model did badly on something. For a ticket-routing task with 34 categories, a defensible construction is 400 items sampled from the last 90 days of production traffic, stratified so that each category contributes in proportion to its volume but with a floor of five items so the long tail is represented at all, labelled independently by two support leads with disagreements adjudicated by a third, and with their level of agreement recorded so a reader knows how solid the labels are. That set costs about a day and a half of somebody's time and is reusable for years.

The floor of five is the part of that construction that looks arbitrary and is not, and working out why explains what the set can and cannot then measure. Sampling 400 items in proportion to volume from a distribution where the top twelve categories carry 80% of traffic leaves the remaining twenty-two to share a fifth of the set between them, which is a handful of items each and, for the rarest, none at all. A category represented by one item has an accuracy of either zero or one hundred per cent, so the measurement has no resolution there whatever the model does. Five items each across twenty-two categories is well over a hundred of the four hundred, which buys enough resolution in the tail to distinguish a model that handles it from one that does not.

What it costs is that the set no longer has production's shape. The tail is deliberately over-represented, so an accuracy computed by counting correct items on this set is an accuracy on a distribution nobody's customers generate. Where a threshold is meant to describe usefulness at the real traffic mix — which is what the metric table below claims for it — the per-category results have to be weighted back to production volumes before the overall figure is read, or the number will sit below the one the service would actually deliver. Macro-F1 has no such problem, because it never used the volumes in the first place. Stating which of the two weightings each threshold is computed under is a line in the specification, and leaving it out is how two people end up quoting different accuracies from the same run.

The success threshold has to be absolute rather than comparative. "Whichever scores highest" always has a winner, including in the case where all three candidates are unfit for the product, and it converts the exercise into a ranking rather than a decision. An absolute bar states what the product needs: 90% top-category accuracy overall, macro-F1 of at least 0.72 across the 34 categories so the long tail is not sacrificed to the head, and schema validity above 99.5% because downstream systems consume the JSON directly. An F1 score combines precision and recall for one category into a single figure between zero and one, and the macro-averaged version computes it separately for each of the 34 and then averages the results with every category counting the same. Precision for a category is the share of the items you put in it that belong there, and recall is the share of the items that belong there which you actually put in it, and F1 combines them because either alone is trivially gameable: a model that assigns everything to billing has perfect recall on billing and useless precision, while one that assigns almost nothing to it has the reverse.

Back to the bar itself. Stating it in absolute numbers does one more thing besides naming what the product needs: it makes "none of them" a legitimate outcome, which is the outcome that most needs to be sayable before the work starts. The difference between an absolute bar and a comparative one is worth stating as a property rather than a preference, because the property is what makes the document a decision procedure. A comparative rule is a function of the candidate set: change which models are on the shortlist and the answer changes, and the rule returns a winner for every non-empty set, including sets in which nothing is fit for the product. An absolute rule is a function of the product's requirements instead, so it gives the same verdict on each candidate however many others are in the room, it survives a fourth model arriving next week without being re-argued, and it is capable of returning nothing. Those three properties are the same property seen from three sides: the bar does not depend on the field.

Which metric carries the threshold is a decision in its own right, and defaulting to overall accuracy is how long-tail behaviour gets sacrificed without anyone choosing to sacrifice it. In a 34-category routing task where the top twelve categories carry 80% of volume, a model can score 91% overall while getting a third of the rare categories wrong, and the rare categories are frequently the ones with a compliance or escalation consequence attached.

That 91% is not a rhetorical figure, and the arithmetic that produces it is what makes the metric choice unavoidable. Routing every item in the top twelve categories correctly earns eighty of the hundred points on its own, because those categories are eighty per cent of the traffic and accuracy counts items. The entire long tail is worth the remaining twenty points between all twenty-two of its categories. A model that handles a bit over half the tail therefore lands in the low nineties overall, and a reading in the low nineties is perfectly compatible with a third of the rare categories being mishandled every single time. The head is large enough to pay for the tail's failures out of its own margin.

Macro-averaged F1 weights every category equally regardless of volume, so it makes that failure visible; overall accuracy hides it by construction. The reversal is as blunt as the arithmetic above: twenty-two of the thirty-four categories are long-tail ones, so they carry close to two thirds of a macro-average while carrying a fifth of the traffic. Neither weighting is the correct one in the abstract. Overall accuracy answers "how much of the work does this get right" and macro-F1 answers "does it hold every distinction the schema draws", and a product that escalates on a rare category needs both questions answered rather than one of them averaged into the other.

Carrying three or four metrics with separate thresholds is better than collapsing them into one weighted score, because the weights in a blended score are an unargued judgement about how much schema breakage a point of accuracy is worth, and they are usually chosen to make a preferred candidate win. Separate thresholds state each requirement plainly and let a candidate fail on the specific thing it fails at.

Not every task has labels, and a routing task is the easy case. For summarisation, drafting or extraction from unstructured documents, the threshold has to be defined over a judgement rather than a match — either human ratings on a fixed rubric, or a model-based judge whose agreement with human raters you have measured on a calibration sample first. The specification then names the rubric, the judge model and version, and the agreement level that makes the judge admissible, because a judge that agrees with your human raters 68% of the time is not evidence about a two-point difference between candidates.

Why that follows is a comparison of two magnitudes, and it generalises past judges to any instrument. Agreeing with humans 68% of the time means disagreeing on close to a third of items, and those disagreements are not spread evenly — they concentrate on the borderline cases, which are precisely the cases two similar candidates differ on. The difference being measured is two points and the instrument's own error is an order of magnitude larger. Which candidate comes out ahead is then decided by which of them happens to suit the judge's idiosyncrasies, and re-running the same evaluation with a better-calibrated judge can reverse the ordering without either model having changed. None of that is an argument against model-based judging. It is an argument for measuring the agreement before relying on the judge and stating in the specification what margin it is entitled to resolve, which is something on the scale of its disagreement rate rather than anything finer.

Pinning the judge model matters as much as pinning the test set: an evaluation re-run six weeks later against a silently updated judge is not comparable with the first one.

The operating budget is the constraint set the winner must satisfy while achieving that quality: p95 end-to-end latency under 2.5 seconds at the peak concurrency the service actually sees, cost under $18 per thousand requests at the measured token split of chapter two, and enough rate-limit headroom or self-hosted throughput to sustain 40 requests per second. Quality without a budget picks the largest model available every time, which is a decision the budget was supposed to participate in.

Two more clauses turn those three into a procedure. A tie-break rule says what happens when candidates are separated by less than the margin you consider meaningful — usually the cheaper one, or the one with better exit properties, decided in advance so it cannot be reasoned toward afterwards. And a disqualifier list carries forward the licence, acceptable-use and serving-feasibility gates of chapter one, so a candidate that fails them is removed regardless of its score.

The artifact, and the procedure it encodes#

The document is short and its value comes entirely from its timestamp.

decision: default model for ticket routing v2
authored: 2026-04-02        # before any evaluation run
owners: [platform-eng, support-ops]
candidates: [model-a-hosted, model-b-hosted, model-c-open-weight-self-host]

disqualifiers:              # from the chapter-one gates, checked first
  - acceptable-use policy conflicts with our domain
  - licence forbids the deployment shape we need
  - no supported serving path within 4 weeks

test_data:
  set: eval/ticket-routing-2026-04.jsonl
  n: 400
  sampling: last 90 days, stratified by category, floor of 5 per category
  labels: 2 support leads, 3rd adjudicates disagreement (kappa 0.87)
  frozen: true             # no items added or removed after this date
  holdout: 100 further items, opened only for the confirmation run

thresholds:                # absolute; "none pass" is a valid outcome
  top_category_accuracy: ">= 0.90"
  macro_f1_all_34: ">= 0.72"
  schema_validity: ">= 0.995"

budget:
  p95_latency_ms: "<= 2500 at 40 rps"
  cost_per_1000_requests_usd: "<= 18.00"
  sustained_throughput_rps: ">= 40"

decision_rule:
  1: drop candidates failing any disqualifier
  2: drop candidates failing any threshold or budget item
  3: rank survivors by macro_f1
  4: if top two are within 0.02 macro_f1, take the cheaper
review_by: 2026-10-01

Every line of that file is a commitment that costs something to break. If model C comes in at macro-F1 0.71 against a stated bar of 0.72, the person who wants model C has to argue that the bar was wrong and say why, in front of the version of themselves who wrote it down when they had no stake in the answer. That argument is sometimes correct and it is a different, better conversation than an open-ended thread.

The branch ending in "no candidate passes" is what makes the whole structure honest. A comparative procedure cannot produce that outcome, so a team using one will ship whichever model was least bad and discover the gap in production. A threshold-based procedure forces the trade-off into the open: someone has to decide whether 87% accuracy is acceptable with a human review step in front of it, or whether the task needs to be narrowed, and that decision gets made deliberately by people who can weigh it.

The limits of pre-commitment#

What pre-commitment does not do is make the test correct. A frozen 400-item set with a stated threshold can measure the wrong thing just as thoroughly as an informal trial, and writing it down early only guarantees that everyone finds out. The protection against that is a review of the specification by whoever owns the product outcome, at the moment it is written, when changing it is free. The question to ask at that review is whether a candidate that scored 0.95 on this set and passed every budget line would actually be a good default model for the product — and if the honest answer is "not necessarily", the specification is incomplete and it is much cheaper to discover that now.

The mechanism pre-commitment does defeat is worth naming precisely, because it operates without dishonesty. With dozens of defensible metrics, sampling choices and prompt variants available, a determined advocate can find a configuration in which their preferred model wins, and will do so without noticing that they stopped looking when the answer came out right. Fixing the metric and the set in advance removes the degrees of freedom rather than asking anybody to resist them.

The size of the problem is a multiplication, which is why willpower is the wrong tool for it. A handful of defensible metrics, times a handful of defensible ways to sample the test items, times a handful of prompt variants each candidate could reasonably be given, is already dozens of complete and defensible evaluations of the same shortlist. Candidates separated by less than the spread across those configurations will each win some of them. So finding a configuration in which any particular candidate leads requires no bad faith and no unusual persistence — it requires only trying configurations until one looks right and then stopping, which is what everybody does with any search that has a satisfying end. Writing the configuration down first collapses the dozens to one, before anybody knows which one flatters which model.

The review date in the specification does a quieter job. Model choices are made against a landscape that changes on a scale of months, so a decision with no expiry either gets reopened opportunistically every time a competitor ships something, or calcifies into "the model we use" long after it stopped being the right one. Naming a date converts both failure modes into a scheduled event: between now and October the question is closed and the team builds, and in October the same frozen test set and the same thresholds are pointed at whatever exists then, which is a few hours of work rather than another nineteen-day thread. The frozen set is the asset that makes re-deciding cheap, and it accumulates value as it ages, provided nobody edits it to flatter a model.

There is also a cost to running the procedure, which should be estimated in the same document. Three candidates against 400 items, with three repeats to get a sense of variance, is 3,600 calls — the three counts multiply because every candidate sees every item on every repeat, and none of the three is optional: drop the repeats and you have no way to tell a real gap from run-to-run noise. At the ticket workload's token split that is a few dollars per hosted candidate and an afternoon of engineering, plus whatever it takes to stand up the self-hosted candidate at all. That last item is usually the schedule risk, and naming it in the specification prevents the common outcome where the two easy candidates are evaluated properly and the third is judged on a handful of manual prompts because time ran out.

A specification of this kind terminates the argument by making the answer a matter of running something rather than winning something: what settles the shortlist is a result, not whoever cited leaderboard rank most insistently. It has one weakness that shows up immediately in the results: suppose the run comes back and the cheapest candidate leads on macro-F1 by 0.02, exactly at the tie-break margin. Whether that separation means anything at all depends on how much the number moves when you run it again — and on a set of behaviours that an offline evaluation, however carefully specified, never observes.