The bake-off

An evaluation run against a frozen test set answers one question well: which candidates can do the task at all. It is a script issuing requests one at a time, with unlimited patience, on a weekday afternoon, against inputs that contain no adversarial text, no truncated documents, no customer writing in Portuguese and no ticket carrying 40,000 characters of pasted logs. A bake-off is the stage after it — a head-to-head trial of the survivors under the conditions the service will actually operate in, run before a candidate is adopted rather than after.

Suppose the cheapest of three shortlisted candidates leads the offline run by two points. Two gaps sit underneath that margin: whether it is a real separation, and whether the candidate that owns it behaves acceptably under the conditions the offline run never created. Both are addressable in about two days of work, and both are routinely skipped, because a completed evaluation feels like a completed decision. The specification did its job by making the question answerable; what it cannot do is guarantee that the answer it produced is stable, or that the properties it measured are the ones that will hurt.

The four dimensions it covers are chosen because each has independently reversed a decision that the offline numbers had already settled. Whether a two-point margin is a real separation or sampling noise is a statistical question answerable from results you already have. Behaviour at peak concurrency is a serving-configuration question that offline evaluation cannot raise, because offline evaluations do not create queues. Which errors a candidate makes, as opposed to how many, determines what each one costs to work around. And the contractual terms are a failure mode with a longer fuse, invisible to every measurement in the exercise.

The margin has to survive being measured again#

A score computed once on 400 items is a point estimate with a sampling distribution around it. At an accuracy near 0.90 on 400 items, the standard error is about 1.5 percentage points, which puts a 95% confidence interval at roughly plus or minus three points on that metric alone.

That figure is one expression and it is worth evaluating rather than accepting, because the shape of it is what tells you whether more items would help. Treat each test item as an independent trial that the model gets right with some fixed probability. The spread of the average over many such trials is the square root of that probability, times one minus it, divided by the number of trials. Here that is 0.90 multiplied by the remaining tenth, which is nine hundredths, divided by the 400 items, and the square root of the result is the 1.5 percentage points quoted. Two standard errors either side of the estimate is the conventional 95% interval, which is where the plus or minus three points comes from.

The item count sits under a square root, and that is the whole practical lesson of the formula. Halving the interval requires quadrupling the set rather than doubling it, which is why the paragraph at the end of this section quotes 1,500 items for roughly half the width: 1,500 is not quite four times 400, and the square root of a little under four is a little under two. Labelling effort grows in proportion to items while precision grows in proportion to their square root, so buying a decisive answer about a small margin is expensive in a way that buying a decisive answer about a large one is not.

Macro-F1 on a stratified set with a floor of five items per rare category is noisier still, because half the categories contribute a handful of items each and a single flipped prediction moves a per-class F1 by a large fraction. The same formula says why in one step: a per-category score computed over five items has an item count of five under the square root rather than 400, so its spread is an order of magnitude wider, and a macro-average is an average of many such wide numbers rather than one narrow one.

Sampling is only one of the sources. Generation is not deterministic even at temperature zero on hosted infrastructure, since the arithmetic depends on how requests are batched at the moment they arrive and mixture-of-experts routing can differ run to run; the practical effect on a classification task is small but non-zero, typically a fraction of a point. Prompt sensitivity is larger: re-ordering the category list in the system prompt, or moving the schema from the system message to the user message, can move a candidate by two or three points, which is the size of the margin being argued about. If the candidates were each given one prompt and no variants, part of what the run measured is which model happened to suit the prompt somebody wrote first.

The check is a paired bootstrap over the per-item results you already have, which costs no additional API calls. A bootstrap draws a new sample of the same size from your own results, with replacement, a few thousand times over, and reads the spread of the answers as an estimate of how much the score would move if you had happened to label a different 400 tickets.

The logic of that substitution is the part worth holding on to, because it looks like getting something for nothing. What you want is the spread across many different samples of 400 tickets, and you have exactly one. The 400 you drew is, however, the best available description of the population they came from, so drawing repeatedly from it imitates drawing repeatedly from the population. Sampling with replacement is what makes each draw a genuinely different set: without it, every resample would be the same 400 items in a different order and the spread would come out as zero. The answer is only as good as the assumption that your 400 represent the population, which is the same assumption the original score rested on, so the bootstrap adds no new leap of faith.

import numpy as np

# per-item correctness vectors from the same 400 frozen items, 3 repeats each
def paired_bootstrap(a, c, n_boot=10_000, seed=0):
    rng = np.random.default_rng(seed)
    n = len(a)
    diffs = np.empty(n_boot)
    for i in range(n_boot):
        idx = rng.integers(0, n, n)          # resample items, not models
        diffs[i] = c[idx].mean() - a[idx].mean()
    lo, hi = np.percentile(diffs, [2.5, 97.5])
    return diffs.mean(), lo, hi, (diffs > 0).mean()

mean, lo, hi, p_c_better = paired_bootstrap(model_a_correct, model_c_correct)
print(f"C - A = {mean:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]  P(C>A) = {p_c_better:.2f}")
# C - A = +0.020  95% CI [-0.019, +0.058]  P(C>A) = 0.84

An interval spanning zero means the run did not establish an ordering between the two candidates. That is a result rather than a failure: it says they are indistinguishable on this evidence, so the decision should be made on the dimensions where they are distinguishable — cost, latency, terms, operational fit — instead of on a difference that would not reproduce. It does not say the two models are equally good, which is the reading that gets it quoted wrongly in a meeting. It says this experiment lacked the resolution to order them, and the honest report of that is "we do not know", not "they are the same".

Pairing matters here, because resampling items rather than models removes the variance both candidates share from finding the same items hard, which usually tightens the interval considerably and is the difference between an inconclusive result and a real one. The cancellation is visible in the line of the bootstrap above that computes diffs[i]: both models are scored on the same resampled idx, so when a draw happens to over-represent the difficult tickets, both scores fall together and the difference between them barely moves. A large part of the wobble in either score is therefore a property of which items were drawn rather than of either model, and subtracting one score from the other on identical items removes it. Score the two on independently resampled sets and that shared component stays in, which is how a genuine separation gets buried under noise that both candidates were experiencing identically. If a genuine two-point separation matters to the product, the way to establish it is more items or more repeats: pushing the set from 400 to 1,500 roughly halves the interval, at proportionally more labelling effort.

Where the offline number stops describing the system#

A load test at your real peak concurrency, with your real prompt, for at least fifteen minutes, produces a different table:

Getting a table like that to mean anything requires holding the right things fixed. The load generator should replay a sampled trace from production rather than a uniform arrival pattern, because real traffic is bursty and a Poisson arrival pattern, in which requests turn up independently of one another at a constant average rate, produces a materially gentler queue at the same mean rate than the real one does. A queue's depth is driven by the peaks rather than by the mean, because a queue only grows while arrivals outrun service and drains whenever they do not. Independent arrivals at a fixed average rate spread the same volume of work more evenly than real traffic does, so the instantaneous peak is lower, the queue never gets as deep, and the tail percentiles come out flattering. The two profiles can carry identical totals and produce completely different p99 numbers.

Prompt, max output tokens, timeout and retry policy have to be identical across candidates, since a shorter timeout flatters a slow model's latency percentiles by converting slowness into errors. The mechanism there is worth naming because it is entirely invisible in the resulting table: a percentile is computed over the requests that completed, and a timeout removes from that set precisely the slowest ones. Cutting the timeout therefore improves every latency percentile of a slow candidate while making nothing about it faster, and it moves the damage into a separate error-rate column that nobody is comparing side by side.

Fifteen minutes is the minimum useful duration because per-minute token limits reset on a window boundary and a five-minute run can sit entirely inside one allocation. The budget refills on each boundary, so a run has to spend faster than the refill for long enough to exhaust it before any throttling appears at all, and a short run can finish while still inside its first full allowance. What that produces is not a slightly optimistic number but a completely different regime: zero throttling and clean percentiles from a configuration that would have thrown 429s all afternoon.

It is also worth telling your provider contact that you are load-testing at a stated rate; the request is usually granted, and it avoids an anti-abuse response that looks exactly like a capacity limit in your data.

Capacity on shared tiers varies with the time of day, which is a real property of the product you would be buying rather than an artefact to be controlled away. Running the same load profile at 09:00, 14:00 and 22:00 local time on two different days typically reveals a spread of 20 to 40% in p95 latency for hosted candidates and almost none for a self-hosted replica, since the self-hosted one is contending only with your own traffic. If your peak coincides with the provider's global peak, the number that belongs in the comparison is the one measured then, not the median across the day.

The 429 rate deserves separate attention because of what happens downstream of it. A client with naive retry logic turns a throttling event into a load amplifier: the failed requests come back immediately, add to the queue that caused the throttling, and the error rate climbs faster than traffic does.

The shape of that is a loop rather than a chain, which is why it is worse than it first reads. Throttling is the provider's response to demand exceeding what your allocation permits. An immediate retry is a new request, so it is more demand. More demand produces more throttling, which produces more retries, and the system's reaction to being overloaded is to increase the load. Nothing in the loop is self-limiting, so the error rate does not settle at some elevated level — it climbs until something outside the loop stops it, usually client timeouts, which is why the sequence below ends in user-visible failures rather than in slow success. Backing off before retrying is what breaks it: a retry deferred long enough for the budget to refill is demand moved out of the window that was already full, and the retry-after header on the response is the provider stating how long that is.

Discovering that during the bake-off gives you a choice — negotiate a higher limit, add a token-budget-aware queue in front of the client, or drop the candidate. Discovering it in production gives you an outage.

Logging the failure modes beside the scores#

An aggregate score is a mean over outcomes that are not equivalent, and the cost of fixing them differs by an order of magnitude. Recording what each wrong answer looked like turns the evaluation into something you can act on rather than rank with. It is also where a public win tends to evaporate on private inputs: the leaderboard reports one number, and the private inputs are where the shape of the errors behind it becomes visible.

The refusal row is the one that changes a decision on its own. A model that declines 1.2% of tickets because they describe harassment is unusable for a support queue whose escalation path depends on exactly those tickets, and no amount of prompt engineering reliably removes a refusal behaviour that was trained in. Two points of macro-F1 does not compensate for a category the model will not process.

Why prompting does not reliably fix it is worth one step, because the belief that it can is what keeps such a candidate on a shortlist for another fortnight. A refusal is not a rule the model is following that a later instruction can supersede. It is a disposition installed by training the model to produce declines on content of a certain shape, and part of that training consists of examples where an instruction asking it to proceed anyway is present and it declines regardless. The behaviour was optimised against exactly the move you would make to override it. Prompt work will move the rate, sometimes a long way, which is the trap: a rate that falls from noticeable to occasional in testing is still a rate, and the tickets it applies to are the ones the escalation path exists for. What you can establish cheaply is which content triggers it, which turns an unknown into a bounded exclusion you can decide about.

The wrong-language row is a related instance of a point from chapter four: a model card that lists a language without evaluating it is describing a capability that degrades under pressure, and the pressure here is a short, angry ticket that switches language halfway through a sentence rather than a clean benchmark item.

The pattern is one that shows up often enough to expect it: the candidate that led the public evidence loses on private inputs, and it loses on axes the public evidence has no column for. Public test sets are cleaned, well-formed and written in the benchmark's own register, while your inputs carry pasted stack traces, truncated email threads, inconsistent product names and the occasional message that is mostly emoji. A model tuned to perform on the former can be systematically fragile on the latter without that fragility appearing anywhere in its scores. The direction of the surprise is not random either — it tends to run against the smaller and cheaper candidate, because robustness to malformed input is one of the things capacity buys.

The last row of the failure-mode table is worth keeping visible because it is the honest floor. Roughly five to six per cent of items get a confidently wrong answer in valid JSON from either candidate, which is the error rate the product has to be designed around — a confidence threshold, a human review queue for low-confidence routes, or an undo path — and no model choice on this shortlist removes it.

Terms are a failure mode with a longer fuse#

The remaining category is contractual and does not appear in any measurement. Default data retention on a provider's standard tier may be 30 days, which is a problem if your customers' tickets contain personal data under a contract promising no third-party retention. Some tiers train on submitted data unless you are on a plan that excludes it. Region of processing may not be configurable below an enterprise agreement, the subprocessor list may include a jurisdiction your customers have excluded, and the deprecation policy may promise as little as 90 days' notice before a model version is withdrawn — which for a product with a validated prompt and a downstream schema is a re-evaluation project, not a config change. Every one of these is knowable during the bake-off by reading the terms and asking a sales contact three direct questions, and every one is capable of eliminating the candidate that won on quality.

None of this argues for skipping the offline evaluation. It stays the filter that removes candidates which cannot do the task, and its frozen set is what makes re-deciding cheap later. What it cannot do is settle a close call, because a close call is decided by the properties it does not measure: whether the margin reproduces, how the candidate behaves when 40 requests arrive at once, which errors it makes rather than how many, and what you would be signing. Running the bake-off across those four dimensions turns a two-point gap into a decision with reasons attached, and the reasons are what make it re-examinable later.

That leaves one candidate standing and one question unanswered, which is what the choice obliges the organisation to for as long as it holds — and whether the winner is a supplier you can replace or a system you now operate.