Four benchmark points are an assumption until your own queries test them

The first thing on the migration board in week one was a capacity request: reserved instances for the dual-index window, a quota increase with the provider, and a purchase order for the second query tier. All three are real needs with long lead times, and all three are bets on a proposition nobody had tested, which is that this particular successor model retrieves better than the incumbent on this particular corpus. The whole of the previous chapter's $150,600 rests on it.

Testing it costs about $1,500 and takes six weeks, most of which is people reading documents. The asymmetry is what makes the sequencing obvious once it is written down, and it is worth writing down, because the capacity request feels urgent in a way that a judgement-collection exercise never does.

Where the queries come from, and who says what is relevant#

The query set has to be drawn from production logs rather than written by the team, because engineers write the queries the system already handles. A sample of 1,500 queries stratified by frequency band and by the dimensions along which the corpus varies — three languages, five content types — gives enough per-stratum mass to notice a regression that is confined to one of them. Head queries, the few hundred distinct strings that make up a third of traffic, are over-sampled relative to their count because a regression there is felt immediately; tail queries are where a new model most often wins, since they are the ones the incumbent's tuning never reached.

Judgements are pooled. Both systems retrieve their top 20 for each query, the union is deduplicated, and the result is judged blind, without any indication of which system produced which document, on a graded scale rather than a binary one. Pooling matters because judging only the incumbent's results guarantees the incumbent wins: anything the new model surfaces that the old one never did is unjudged, and unjudged defaults to irrelevant. On this corpus the pool came to about 18 distinct documents per query, so 27,000 judgements, which at roughly 90 seconds each is 675 hours of domain-expert time.

An LLM judge can carry most of that volume, and at this scale it is the difference between a six-week exercise and a six-month one. It has to be calibrated rather than trusted: have humans judge a random 1,500 of the pool, measure agreement against the model's labels, and only proceed if agreement is at the level you would accept between two human judges — Cohen's κ, which measures how often two judges agree after discounting the agreement they would reach by chance alone, above about 0.6 on a graded scale. Where the model and the humans disagree systematically rather than randomly, the model is encoding a different notion of relevance than the domain does, and the offline result will be measuring that instead of the retrieval difference. The alternative source of judgements, click logs from production, has the advantage of being free and the disadvantage of being available only for the incumbent's rankings, so it cannot evaluate documents the incumbent never showed. It is a good instrument for the online phase after cutover and a poor one for choosing between two models offline.

The slice has to be big enough to separate them#

The most common way this evaluation gets rigged without anyone intending to is by running it against a small corpus slice. Retrieval quality is a function of how many plausible distractors sit near a query in embedding space, and distractor density scales with corpus size, so a 400,000-chunk evaluation index makes both models look excellent and makes them look alike. The scores below are nDCG@10, normalised discounted cumulative gain over the top ten results: a number between 0 and 1 that rises as the documents a judge rated highly appear, and appear early, in the ten a query returns.

The absolute numbers keep falling as the slice grows, which is expected and not a problem, since nothing about the decision depends on the absolute level. The gap is what matters, and it is still moving at 0.5% and has stabilised by 2%. The procedure that follows from this is to build the slice at two or three sizes and check whether the gap has stopped changing; if it is still moving at your largest slice, you cannot rank the models yet.

The cost of doing this properly is small enough to remove the argument. An 8% slice is 32 million chunks, 13.1 billion tokens, $918 to embed at the successor's price, and 43.5 GB of memory at a single replica, which is $174 a month. Both indexes at that size, running for six weeks, come to roughly $1,500 including the incumbent's copy. That is one percent of the migration it is deciding.

The slice must be a random sample of documents rather than a recent slice or a single content type, because the near-duplicate structure of the corpus is a large part of what makes retrieval hard, and taking only the last six months of documents removes most of it.

Fix the metric before you look at anything#

The primary metric, the guard metrics, and the decision thresholds are all written down before the first result is computed. Choosing them afterwards means choosing the metric on which the answer you want is true, and with five content types, three languages and half a dozen plausible metrics there are enough degrees of freedom that some cut of the data will favour either model.

nDCG@10 is the primary here because the product surfaces ten results. It is the wrong primary for a system with a cross-encoder reranker consuming the top 50 — a slower model that reads each candidate together with the query text rather than comparing stored vectors — where the first-stage retriever's job is to get the right documents into the candidate set at all and recall@50, the share of a query's relevant documents that appear anywhere in those fifty, is what the downstream stage depends on; a first-stage model that improves nDCG@10 while dropping recall@50 by two points has made the pipeline worse. Latency at a fixed recall target belongs in the set too, since a model whose vectors need a higher ef_search to reach the same recall costs more per query forever.

Sample size follows from the paired variance rather than from a round number. Per-query nDCG@10 differences on this corpus have a standard deviation of about 0.15, which is typical for graded relevance on heterogeneous documents. The figures below assume the conventional settings: a 5% chance of calling a difference real when it is not, and 80% power, meaning an 80% chance of detecting a difference of the stated size when one genuinely exists.

import math
from statistics import NormalDist

def paired_n(delta: float, sd_diff: float,
             alpha: float = 0.05, power: float = 0.80) -> int:
    z_a = NormalDist().inv_cdf(1 - alpha / 2)
    z_b = NormalDist().inv_cdf(power)
    return math.ceil(((z_a + z_b) ** 2) * sd_diff ** 2 / delta ** 2)

paired_n(delta=0.02, sd_diff=0.15)   # 442
paired_n(delta=0.05, sd_diff=0.15)   # 71

Detecting a two-point difference needs 442 queries; the 1,500 in the set leave room to cut by language and content type and still have 150 to 300 per stratum, which is enough to detect a five-point subgroup regression and not enough to detect a two-point one. Knowing that in advance stops a flat subgroup result from being read as evidence of no difference.

The comparison itself is paired and reported with an interval, because both models answer the same queries and the per-query pairing removes most of the variance that comes from some queries simply being harder. The interval comes from a bootstrap, which resamples the list of per-query differences with replacement a few thousand times and reads the range covering the middle 95% of the resampled means, and it needs no assumption about the shape of the distribution.

import random

def paired_bootstrap(diffs: list[float], iters: int = 10_000) -> tuple[float, float]:
    n = len(diffs)
    means = []
    for _ in range(iters):
        sample = [diffs[random.randrange(n)] for _ in range(n)]
        means.append(sum(sample) / n)
    means.sort()
    return means[int(0.025 * iters)], means[int(0.975 * iters)]

Deciding in advance what each outcome means#

Three outcomes are possible and each has a different consequence, so each needs an owner and a written threshold before the numbers exist.

The third row is the one that needs stating plainly, because "abort" under a deprecation deadline does not mean staying put. It means the migration target changes: a different successor, a self-hosted continuation of the incumbent architecture with weights you control, or a hybrid arrangement where lexical retrieval carries the query types the new embedding model handles worse. Each of those has its own cost profile and its own lead time, which is the practical reason the evaluation belongs in month one rather than month five.

On this corpus the overall result was the first row and the subgroup analysis was the second. The successor gained five points overall, comfortably clear of zero, and lost nine points on German-language maintenance manuals — a stratum that was 11% of traffic and whose queries were dominated by compound nouns the new tokeniser split differently. That finding cost $1,500 to obtain and changed the plan: the migration proceeded, with a per-language routing rule holding German queries on a retained incumbent index for two further quarters. Finding it after cutover would have meant discovering it from support tickets, with the old index already deleted.

What the evaluation still does not tell you#

Offline retrieval metrics measure the retriever, and the product is a pipeline. A five-point nDCG@10 gain can be entirely absorbed by a reranker that was already recovering those documents, in which case the end-to-end answer quality does not move and the migration's quality justification evaporates even though the retrieval measurement was correct. Running the evaluation with the reranker in the loop on a subset of queries is the cheap guard against that.

The judgement set also ages. Judgements collected against a corpus and a query distribution have a shelf life of a year or so before the query mix has drifted enough that the set no longer represents traffic, so the 27,000 judgements are an asset to maintain rather than an artefact of one project — refreshing 10% a quarter keeps them usable and costs a fraction of building them again.

What remains is the work itself, and it starts with the smallest piece. Look at what the current incremental job selects on. If it is a timestamp, the index is already carrying staleness that nothing in the system will report, and one afternoon spent moving invalidation onto a fingerprint of the embedded text will tell you how much. The quote, the token budget, the resumable backfill and the costed transition all sit on top of that one answer.