Validating a platform cutover

A parity eval is the standard gate on a model change: take a golden set — a fixed collection of inputs, each with a reference answer somebody wrote by hand — send the same prompts to the old endpoint and the new one, score both, and ship if the scores match. Scoring is usually done by an LLM judge, a second model prompted to compare each answer against the reference and return a number, with a bootstrapped confidence interval on the mean so that "match" has a threshold rather than a feeling. It is a good instrument and it is the right one for the question it was built for.

The question it was built for is "is this model good enough at our task". A platform cutover asks something else: "does this endpoint behave the same as the one it replaces, on our actual traffic, including the parts of our traffic that were never put in a test set". That is a question about distributions, metadata and price. An eval cannot answer it, and the failure is arithmetic rather than carelessness.

Four hundred and eighty examples cannot see a four-in-a-thousand failure#

Take a platform content filter rejecting 0.4% of production traffic, which is the rate the previous chapter's claims service measured. On a 480-example test set, the expected number of rejections is 1.9, and the probability of drawing zero is about 15%.

Both figures come out of two lines of arithmetic, and doing them by hand is what stops the result feeling like a statistical claim to be taken on trust. Four in a thousand of 480 examples is 1.9 rejections, which is the expected count: multiplying a rate by a size gives the average number of events across many runs of the same eval. The probability of seeing none is a different question, and it is answered by multiplying rather than by adding. Each example independently avoids rejection with probability one minus four in a thousand, and for all 480 of them to avoid it, that same factor has to hold 480 times over — so the probability of a clean run is that factor raised to the 480th power. It comes out at about 15%.

Fifteen percent is the figure to sit with. It says that an eval which is working perfectly, against a rejection rate that is real and steady, reports a completely clean run roughly one time in seven, and a clean run is what everybody in the review will read as evidence that there is nothing there. The blindness is not a defect in the harness or in the person who ran it. It is what a sample of that size can and cannot resolve, and no amount of care in executing the run changes it.

Even a test set that does contain a couple of rejections has a second problem: a golden set is not a sample of production. It was curated, which means somebody chose examples clear enough to write a reference answer for, and then it was maintained, which means examples that broke the harness got quietly removed over eighteen months of use. Injury descriptions graphic enough to trip a violence classifier are exactly the kind of example a curator drops as "not representative".

Then there is what the harness does with a failure. Most eval runners treat an empty completion as an infrastructure problem and either retry it or exclude it from the aggregate, because the alternative — scoring transport errors as quality-zero — makes the metric jump around whenever the network does. Two empty responses in a 480-example run get logged as generation errors and the mean is computed over 478.

Suppose the runner had not excluded them. A rejection scored as 1 against a population mean of 4.3 moves the average by about 0.013, which is a fifth of a ±0.07 confidence interval. The eval still passes. A metric built to summarise the centre of a distribution cannot be made to report a tail event by scoring the tail event correctly; the arithmetic of averaging removes it either way.

That 0.013 is worth deriving, because it is the whole case against fixing this by scoring more honestly. Replacing one score of 4.3 with a score of 1 takes a bit over three points out of the run's total. A mean divides the total by the number of examples, so removing three-and-a-bit points from a 480-example total moves the mean by three-and-a-bit divided by 480, which is well under one hundredth of a point. Two rejections move it by twice that — a little over one and a third hundredths, which is the 0.013. Set that against a confidence interval of ±0.07, and the entire effect of scoring both failures as complete failures is about a fifth of the width of a band inside which the two endpoints are already being called indistinguishable.

The general shape is more useful than the specific figure, because it says the same thing about any golden set of any size. A mean divides by the size of the set, so the influence any single example can have on it is bounded by one over that size, whatever score the example is given. A rare event is rare precisely because it occupies a small share of the set. Those two properties work against each other exactly: the more faithfully a golden set reproduces production's rate of tail failures, the smaller the share those failures occupy, and the less they can move the average. Lowering the pass threshold does not recover the signal, because the signal was destroyed by the division rather than by the threshold.

Trace that 0.4% forward into a support queue and the second-order numbers reconcile. Thirty-four failed claims a day is 238 a week, and if roughly one in six of those customers is annoyed enough to open a ticket rather than retry and give up, that is 38 extra tickets a week. The ticket text never says "content filter", because the customer sees a claim stuck in processing with no summary attached. Reaching the filter from that starting point requires somebody to correlate ticket timestamps against a metric nobody was recording.

The one-in-six is not a constant anybody can look up, and it is worth saying where it comes from rather than letting it read as an industry figure. It is the ratio implied by a ticket increase that was measured after the fact and then run backwards. Thirty-four failures a day is 238 a week; the observed rise was 38 tickets a week, which is a bit over six failures per ticket. Read forwards it is a guess about how annoyed people get. Read backwards it is a reconciliation, and that is the only direction in which it is worth anything: it closes the loop between a rejection rate nobody was recording and a support volume somebody was, which is what lets a support manager's number and a platform's number be recognised as the same event.

The judge never saw the invoice#

Cost is the sharper case, because an eval has no channel for it whatsoever. There is no field in the scoring output where a price could appear, and the run itself is a poor economic simulation: 480 sequential requests sharing one warm prefix cache, executed in eight minutes, is the most cache-friendly traffic shape that exists.

Production is not shaped like that. A claims service of the size described in the previous chapter sends a 12,000-token prefix — system prompt, policy extract, six worked examples — in front of a mean 1,050 tokens of variable input, answers with a mean 310 tokens, and does it 252,000 times a month. Its vendor cache had been configured with a one-hour extended TTL, which held a 92% hit rate through the overnight trough where requests arrive six to nine minutes apart. The platform's cache implementation offered a five-minute TTL and nothing longer, so every gap longer than five minutes forced a rewrite of the full prefix, and the measured hit rate settled at 85%.

Seven percentage points sounds negligible until both sides of it are priced. At $0.30 per million tokens for a cache read and $3.75 per million for a cache write, moving one percent of 3.024 billion monthly prefix tokens from the read column to the write column costs $104. Seven points is $730, and that single mechanism is most of the overrun.

Every figure in that sentence is derivable from the workload above, and it is worth deriving because this is the arithmetic the review will argue about. A 12,000-token prefix sent 252,000 times a month is 3.024 billion prefix tokens, and 252,000 is itself 8,400 requests a day across a month. One percent of 3.024 billion is a bit over thirty million tokens. Those tokens are billed either way, so what changes is not whether you pay for them but which price they attract — which means the cost of moving them is the gap between the two prices rather than either price on its own. A write at $3.75 a million against a read at $0.30 a million leaves a gap of just under three and a half dollars a million. Thirty million tokens at that gap is the $104, and seven of those is the $730.

Naming the mechanism plainly is what makes the number stop being surprising. A cache miss on a prefix does not cost you a read you failed to get. It costs you a write you did not want, at more than ten times the price of the read it replaced, and it costs that on every request that misses. A seven-point drop in hit rate is therefore not a seven percent cost increase; it is seven percent of the traffic moving between two prices that differ by an order of magnitude.

The delta column adds up, and adding it up is how the mechanism gets identified rather than argued about. Sixty-four dollars came off the read line, because fewer prefix tokens were read. Seven hundred and ninety-four went onto the write line, because those same tokens had to be written instead and a write costs more than ten times what the read it replaced cost. Twenty-six went onto variable input. Output did not move at all, because nothing about caching or about templates touches the tokens the model generates — the model produced the same answers of the same length on both stacks. Those four movements sum to seven hundred and fifty-six dollars, and seven hundred and fifty-six against a modelled $3,708 is the 20.4%.

The $26 on variable input is the chat-template overhead from the previous chapter, plus a small contribution from generic retry-on-empty-response logic, which turned each filtered request into three billed calls. Note what the table does to the intuitive explanation. Per-million prices were identical on both stacks, every unit price compared during procurement matched, and the invoice was still a fifth over model.

What a pre-cutover run has to compare#

The replacement for the parity eval is a shadow run, which is a different instrument measuring different things. A fraction of live requests is sent to both endpoints, the vendor's response is the one returned to the user, and the platform's response is recorded for comparison. What makes it work is that the input distribution is production's rather than a curator's, and that the comparison is over distributions and metadata rather than over a quality score.

Those two properties map exactly onto the two failures above, which is the reason for choosing this instrument rather than a bigger golden set. A curated set cannot contain a rejection nobody chose to include, and the examples most likely to be rejected are the ones a curator most reliably drops; live traffic contains rejections at whatever rate live traffic produces them, because nobody selected it at all. And a quality score has no field in which a token count, a cache hit or a stop reason could appear, whereas a log line has a field for each of them because you wrote the line. A shadow run is not a better eval. It is an instrument for the quantities an eval has no channel to carry.

import random, time, structlog

log = structlog.get_logger()
SHADOW_RATE = 0.05  # 5% of production requests are mirrored

def answer(request):
    t0 = time.perf_counter()
    primary = vendor_client.messages.create(**request)
    primary_ms = (time.perf_counter() - t0) * 1000

    if random.random() < SHADOW_RATE:
        try:
            t1 = time.perf_counter()
            shadow = platform_client.converse(**translate(request))
            log.info(
                "shadow_compare",
                primary_stop=primary.stop_reason,
                shadow_stop=shadow["stopReason"],
                primary_in=primary.usage.input_tokens,
                shadow_in=shadow["usage"]["inputTokens"],
                primary_cache_read=primary.usage.cache_read_input_tokens,
                shadow_cache_read=shadow["usage"].get("cacheReadInputTokens", 0),
                primary_out=primary.usage.output_tokens,
                shadow_out=shadow["usage"]["outputTokens"],
                primary_ms=primary_ms,
                shadow_ms=(time.perf_counter() - t1) * 1000,
                shadow_empty=len(shadow_text(shadow)) == 0,
            )
        except Exception as exc:  # shadow failures must never affect the user
            log.warning("shadow_failed", error=type(exc).__name__)

    return primary

Every field in that log line exists because one of the platform differences is visible in it and in nothing else. The stop_reason pair catches the endpoint default on output length. The token pairs catch the tokenizer revision and the cache behaviour, and because they are the platform's own reported usage they are the numbers the meter will bill. The shadow_empty flag catches the content filter, and it has to be a separate flag because a filtered response arrives as HTTP 200 with a well-formed body. The latency pair is the only one that requires the shadow to run at production concurrency to mean anything.

Run it long enough to see the tail. At 5% of 8,400 daily requests and a 0.4% filter rate, you see 1.7 rejections a day, so a three-day run is a coin flip and a fourteen-day run gives you around 24 and a usable rate estimate.

Those two counts are the same multiplication done twice. Five percent of 8,400 daily requests is a bit over four hundred mirrored calls a day, and four in a thousand of four hundred is between one and two rejections, which is the 1.7. Fourteen days of that is around 24. Why 24 is qualitatively different from a handful is a property of counting rare events: the uncertainty on a count of this kind grows like its square root, so five observations carry an uncertainty of about two — nearly half the count itself — while two dozen carry an uncertainty of about five. A three-day run that produces five rejections and a three-day run that produces none are both ordinary outcomes for the same underlying rate. That is what makes a short run a coin flip rather than a measurement, and it is the same arithmetic that made the 480-example eval blind.

A fourteen-day run also spans a weekend, which is where the cache TTL difference is largest, and it spans a month boundary often enough to catch the batch jobs that only run on the first.

Sample rate and duration trade against each other for the rate estimate but not for the cost estimate. Mirroring 100% of traffic for one day and 5% for twenty days both yield about 8,400 mirrored requests and a similar confidence interval on the filter rate, yet only the twenty-day run sees the overnight troughs, the weekend, and the shape of the demand curve that determines cache behaviour.

The equality is exact enough to be worth seeing rather than accepting. One day at the full rate mirrors that day's 8,400 requests. Twenty days at one request in twenty mirrors about four hundred a day, which over twenty days is the same 8,400. The rejection count, and therefore the confidence interval on the rejection rate, is the same in both. What is not the same is every quantity that depends on when requests arrive rather than on how many arrive: a cache entry expires because of a gap between two requests, not because of a total, and a compressed replay has no gaps in it at all. When a schedule forces a choice, duration is the more informative variable, because the differences that cost money depend on the gaps between requests rather than on the requests themselves.

The empty-response gate is deliberately absolute. A filter rate of 0.4% is not a quality regression to be traded off against price; it is a set of specific customers whose requests fail, and it is fixable by configuring the filter thresholds, so there is no reason to accept it as a cost of migrating.

A shadow run has no reference answers#

A shadow run compares two endpoints against each other on live inputs, which means it cannot tell you that both of them have become worse. That is the thing the golden set is for, and the reason to keep running it. The two instruments answer different questions and a cutover needs both.

It is also a point-in-time measurement of a moving target. The platform's filter model, chat template and cache implementation are all things the platform ships on its own schedule, so a clean fourteen-day comparison in March says nothing about June. That is the argument for promoting the log line's fields into standing production metrics rather than retiring the comparison once the cutover lands.

There is a cost to the technique that has to be named. Mirroring 5% of traffic means paying for 5% more inference during the trial, and it means sending customer content to a second processor, which is a data-flow question rather than an engineering one and is exactly the subject of a later chapter in this topic. On a workload subject to a residency clause, the shadow run itself needs sign-off before it starts.

Matching eval scores tell you that two endpoints answer a curated test set about equally well. They do not tell you that one of them rejects four requests in a thousand, that its cache expires twelve times sooner, or that its meter rounds. Those three live in the tail, in the gaps between requests, and on an invoice, and the only place all three are visible at once is a comparison run against real traffic over a period long enough to contain a weekend.

The obvious way to shorten a fourteen-day shadow run is to raise the sample rate, and a team in a hurry will reach straight for 100% and a compressed replay of last week's traffic. That doubles the request rate against an account whose capacity is administered by the platform rather than by you, and the first thing such a replay discovers is that the platform's ceiling is not a single number and does not sit where the console appears to say it does.