Most of your replay sample cannot show a difference

The bug report arrived three weeks after the rollout, from the data team rather than from a customer. The multi-step extraction workflow — take an uploaded invoice, pull twelve fields, emit a record — was producing records with the wrong supplier tax ID roughly one time in four. The records had gone into the ledger. Someone would have to reconcile nine thousand of them by hand.

The pre-launch evaluation was still in the ticket, three scrolls up: 5,000 replayed production requests, both arms, the same judge rubric CI has used for a year. Cascade 4.31. Always-strong 4.34. The gap was 0.03 on a five-point scale, comfortably inside the confidence interval anyone would compute, and the reviewer's comment said equivalent within noise, ship it.

That evaluation was not unlucky. Its design cannot detect a regression of this kind, and it would have reported approximately the same thing if the extraction workflow had been ten times worse.

Which rows in the sample carry any information#

Every row in the comparison is one request scored twice, once per arm. A row can only contribute to the difference between the arms if the two arms produced different answers and the judge scored them differently, and in a cascade evaluation most rows fail at least one of those conditions.

The first branch is pure construction: on the 12% the cascade escalated, the cascade also ended up calling the strong model, so the two arms ran the same model on the same input and produced answers that differ only through sampling noise. Six hundred of the five thousand rows are a comparison of a model against itself.

The second branch is the design's own premise turned against the measurement. Routing is worth doing precisely because a large majority of traffic is easy enough that the cheap model handles it as well as the strong one — the entire business case is that most requests do not need the expensive path. On this sample that was 3,970 rows scoring identically in both arms, most of them at 4.5 or 5.0. Each of those rows adds one to the denominator of both means and zero to the difference between them.

That leaves about 430 rows out of 5,000 — under 9% — where the arms could diverge at all. The aggregate mean divides everything those rows say by eleven before anyone reads it.

What the slices said#

The same 5,000 rows, grouped by task, tell a completely different story from the two numbers at the bottom.

Eighty-nine percent of the aggregate gap comes from 3% of the traffic, and within that slice the effect is enormous. Weighted by 0.03, a catastrophic result becomes a rounding error; read on its own slice, a drop of 0.85 points on 150 requests with a paired standard deviation around 1.1 is a t-statistic near 9.5, meaning the measured gap is nine and a half times the amount a fresh sample of the same size would typically shift it, which is not a marginal call.

There is a second compression at work in that table, and it flatters the cascade further. The extraction failure is functionally total — a record with the wrong tax ID is not a partially useful record, it is a record that has to be found and fixed — and the judge scored it 3.55 rather than 1.0 because the output was well-formed JSON with eleven correct fields out of twelve and a plausible twelfth. Rubric scales are built to reward partial quality, so they compress exactly the failures that downstream systems treat as binary. Any evaluation whose consumer is a pipeline rather than a reader needs at least one slice scored on a pass/fail predicate — does every field match the ground truth — alongside the graded rubric, or the arithmetic will keep softening the failures that matter most.

The same data, analysed properly#

The reviewer's "inside the confidence interval" was computed the way A/B results usually are, treating the two arms as independent samples. With a score standard deviation near 0.9, the standard error of the difference between two independent means over 5,000 requests each is about 0.018, so a gap of 0.03 sits at 1.7 standard errors and fails a two-sided test at the 5% level.

The arms are not independent. Every request appears in both, so the difference can be computed per request and averaged, which removes all the variance contributed by requests being intrinsically easy or hard. Ninety-one percent of the paired differences are exactly zero, and the ones that are not are concentrated. The standard deviation of the paired difference on this data is 0.17, giving a standard error of 0.0024:

paired = df.assign(delta=df.cascade_score - df.strong_score)

n     = len(paired)                       # 5000
mean  = paired.delta.mean()               # -0.0286
se    = paired.delta.std(ddof=1) / n**0.5 # 0.0024
t     = mean / se                         # -11.9

print(f"paired delta {mean:+.4f} ± {1.96*se:.4f} (95% CI), t = {t:.1f}")
# paired delta -0.0286 ± 0.0047 (95% CI), t = -11.9

by_task = (paired.groupby("task")
                 .agg(n=("delta", "size"),
                      delta=("delta", "mean"),
                      se=("delta", lambda d: d.std(ddof=1) / len(d)**0.5))
                 .assign(lo=lambda x: x.delta - 1.96 * x.se,
                         hi=lambda x: x.delta + 1.96 * x.se)
                 .sort_values("delta"))

The identical 0.03 gap goes from "inside the noise" to twelve standard errors away from zero purely by pairing rows that were always paired. The unpaired test was measuring the variance of the workload rather than the variance of the treatment effect, and the workload is enormously more variable than the routing decision is. Anyone comparing two arms over replayed traffic should be pairing by default; the cascade case just makes the cost of not doing so unusually vivid, because the treatment touches so few rows.

Pairing fixes the significance question and not the attribution question. A paired delta of −0.0286 with a tight interval says something real has changed and gives no clue that one workflow owns nine tenths of it. The per-slice breakdown is what turns a real-but-tiny number into a decision, which is why the grouping in the second half of that snippet matters as much as the test in the first.

Two arms are one fewer than you want#

The comparison as described has a cascade arm and an always-strong arm, which answers "is the cascade as good as the expensive thing". Adding a third arm — always-cheap, the same 5,000 requests through the small model with no escalation at all — costs about $6 of inference and answers two more questions that the two-arm design leaves open.

The first is how much of the cascade's quality is coming from escalation rather than from the cheap model being adequate. On this sample the always-cheap arm scored 4.28, against 4.31 for the cascade and 4.34 for always-strong, which means the entire escalation apparatus — 12% of traffic sent to a model twenty times more expensive, plus a verifier, plus the operational surface of a two-model dependency — recovered 0.03 of the 0.06 point gap between the cheap model alone and the strong model alone. Whether that is worth building is a question the two-arm design cannot even pose, and it is the single most useful number for deciding not to build a cascade.

The second is per-slice, and it is where the third arm earns its place. Compare always-cheap against always-strong on each task and you get the maximum quality at stake in routing that task, independent of how good your escalation signal happens to be. FAQ lookup showed 4.51 cheap against 4.52 strong, a gap of 0.01, which says the routing decision on 41% of traffic is nearly free regardless of signal quality — that traffic could be pinned to the cheap model with no signal at all. Extraction showed 3.10 cheap against 4.40 strong, a gap of 1.30, which says the slice is entirely dependent on the escalation decision being right and is the one place a weak signal is expensive. Ranking tasks by the cheap-to-strong gap tells you where to route unconditionally, where to verify carefully, and where to pin to the strong model and stop thinking about it.

Two smaller design choices are worth settling at the same time. Sampling window: draw from a period long enough to cover weekly seasonality but recent enough to reflect the current prompt and retrieval index, which in practice means the last two to four weeks rather than a year of history, since older traffic was answered by a system that no longer exists. And determinism: run every arm at temperature 0 where the product does, and where it does not, run each arm three times and average, because a single sample per arm at temperature 0.7 introduces per-row variance that pairing cannot remove and that will show up as spurious slice-level deltas on small slices.

Designing the run so the answer is readable#

Five things separate an evaluation that could have caught the extraction failure from the one in the ticket.

Pin everything and record it. Both model versions, the judge model version, the prompt template hash, temperature and top-p, the retrieval index snapshot, and the escalation threshold. A comparison whose escalation path silently moved to a new strong-model build is not a comparison of the cascade against always-strong, and the record is what lets you say so three weeks later.

Stratify the sample rather than mirroring the traffic mix. Natural sampling gave the extraction workflow 150 rows, which was enough here only because the effect was huge. Detecting a 0.5-point regression at a paired standard deviation of 1.1 with 80% power — an eight-in-ten chance of the run coming back significant when the regression is genuinely there — needs roughly 77 paired rows in the slice; detecting 0.25 needs about 300. Sampling proportionally guarantees that the rarest workflows — which are usually the most specialised and the most likely to break under a weaker model — get the least statistical power. Draw a floor of 300 rows per named task and reweight to the traffic mix only when computing the headline number.

Slice by difficulty as well as by task. Task labels are convenient and they are not the axis routing acts on. A difficulty proxy — input token count in bands, number of retrieved chunks, presence of multi-step structure, or the labelled needed escalation flag from chapter 4 — puts the rows the cascade actually treats differently into their own bucket, where the cheap-model penalty is visible undiluted.

Report the divergent subset separately. The mean over rows where the two arms called different models answers a different and more actionable question: given that this request stayed cheap, how much did that cost in quality? On this sample that subset was 4,400 rows with a delta of −0.033, and its extraction slice showed −0.87.

Make the gate a worst-slice rule. "Aggregate difference inside the confidence interval" is a gate that a cascade passes almost regardless of behaviour. A gate of the form no named slice regresses by more than 0.15, and no slice falls below the acceptance bar of 4.0 would have blocked this launch on the extraction row alone, without anybody needing to have predicted in advance which workflow was at risk.

None of this is exotic, and all of it costs a few hours plus the inference for a larger stratified sample. The extraction reconciliation cost two people three weeks.

What the offline run still cannot deliver is the behaviour of the thing once real traffic, real retrieval and real model updates are in play — and the report that eventually goes to the people funding this usually leads with the only number that survives all of that unchanged, which is the invoice.