Two fully built versions with atomic routing make a regression attributable

Two complete indexes over one corpus, with a single switch deciding which of them answers a query, is a different design from a migration that converts one index in place, and this chapter is about the second design being worth its extra storage. A corpus, here, is the set of documents a retrieval system has embedded and stored. The system worked through below holds 4.1 million documents, which chunk into eighty million vectors; the two figures count the same system at different grains, documents against chunks, and both appear because different parts of the arithmetic need different ones.

Four terms carry the rest of the chapter. An alias is a name that queries are addressed to and that resolves to exactly one concrete index, so chunks_search might resolve to chunks_v2_e5large_1024, and changing which index answers becomes a change to the name's target rather than a change to any query. Shadow means mirroring live queries to the new index and logging what it returns while every user is still served from the old one. Canary means serving a small share of real sessions from the new index for real. nDCG@10 — normalised discounted cumulative gain at ten — is the relevance number these decisions get argued over: it looks at the first ten results a query returned, credits a relevant document more the nearer the top it sits, and divides by the score of the best possible ordering so that a perfect ranking comes out at one.

The reason to care is attribution. When a relevance number moves after a migration, the question in the review is whether the new model is worse on this domain or whether something about how the migration ran was worse, and that question is answerable only if the two possibilities were ever separated — in time, in traffic, or in the artefacts left behind. Separation is not something you can add afterwards. It is a property of how the migration was run, and by the time the number moves, the run is over.

One variable, one switch#

The design that keeps the two possibilities separated treats the two generations as two complete artefacts. Index v3 is built to completion beside v2 using the backfill and catch-up machinery from chapter 6, where a job reading a snapshot is kept level with the writes landing while it runs. Both indexes are queryable. Neither ever contains a vector from the other's encoder, which is the property chapter 4 makes non-negotiable, since a distance computed between one model's vector and another model's vector is arithmetic without meaning. Traffic moves by changing which collection a query resolves to, and that resolution happens in one place.

An in-place tranche migration — re-embedding ten percent of the corpus at a time and writing each new vector over the old one — destroys two separate things, and they are worth naming separately because they fail differently. The first is the previous state: the old vectors are overwritten tranche by tranche, so at the end there is nothing to roll back to and nothing to compare against. The second is the independent variable: throughout the run two things change together, which model produced the vectors and what share of the index each model owns, so every measurement taken during or after the run has two candidate causes behind it. Storage recovers from the first, given a snapshot. Nothing recovers from the second, because the observations that would have separated the two causes were never made.

The belief that produces a tranche plan is that a slow change is a safe change, and it is worth stating in its strongest form: it is a defence that stays constant while the thing it must discriminate varies. A ten-percent tranche looks identical on a dashboard whether the new model is better than the old one or worse — the same share of the index has moved, the same latency, the same error rate, the same absence of exceptions. It looks identical because what a tranche changes is the mixture rather than the quality, and mixing two generations degrades retrieval regardless of which generation is better.

That belief is tempting because almost everywhere else in engineering it holds. A small blast radius really does bound the damage of a bad deploy, and canarying a release to a handful of hosts is exactly this reasoning applied correctly: the failure is that the new code is broken, brokenness shows up inside the share that received it, and stopping there confines it to that share. Retrieval breaks the pattern for the reason chapter 4 gives. A query against a half-migrated index scores some candidates in v2's space and others in v3's, the cross-generation scores land at the noise floor, and the two halves of the candidate set are therefore not being ranked against each other at all. The intermediate states of this particular gradual change are worse than either endpoint, and a plan that is more careful spends longer in them.

Every edge out of Canary and Primary that points backwards exists because v2 is still there, fully built and still receiving writes. That is the operational cost of this design — you keep dual-writing to the old generation for as long as you want the option to return, which by the storage arithmetic of chapter 5, where eighty million 1024-dimension float32 vectors come to 328 GB before any index structure at all, is that much raw data held for the length of the retention window — and it buys a rollback that takes as long as an HTTP request.

# Elasticsearch / OpenSearch: alias moves atomically, both actions or neither
curl -X POST "$ES/_aliases" -H 'content-type: application/json' -d '{
  "actions": [
    {"remove": {"index": "chunks_v2_e5large_1024", "alias": "chunks_search"}},
    {"add":    {"index": "chunks_v3_voyage3_1024",  "alias": "chunks_search"}}
  ]
}'

An alias swap is a single cluster-state update, and that mechanism is what makes the claim of atomicity true rather than approximate. The two actions in that list are applied to one version of the cluster state, the new version is published as a whole, and a query resolves the alias against whichever version it happens to see. There is no published version in which both indexes hold chunks_search and none in which neither does, so no query can see both indexes or none. Qdrant offers the same primitive through update_aliases, and its own migration guide walks the build-validate-cutover sequence for that engine specifically. Where the store has no alias concept, the equivalent is a routing record in the application's configuration store that the query path reads on every request, with a short cache — a feature flag evaluated per request works, provided the flag value and the encoder fingerprint from chapter 4, which pins which model a vector came from, are resolved together, so that a request can never encode with v3's model and query v2's collection.

The important property is not speed. It is that the collection a query used is a fact you can log. Every retrieval trace records index=chunks_v3_voyage3_1024, so a relevance complaint from Thursday can be attributed to a generation without archaeology, and an A/B comparison is a group-by rather than a reconstruction.

The gates, and which of them actually catches anything#

Four checks run before the switch, and they are not equally useful.

Count parity is the check everybody writes first and the weakest of the four, and the multiplication says why. A difference of 0.1% is one part in a thousand, and eighty million vectors divided by a thousand is 80,000 chunks. So a tolerance most people would describe as tight leaves that many chunks free to be present on one side and absent on the other. That much drift arises with both indexes entirely correct, because writes land in different orders on the two sides and either index can be tens of thousands of documents ahead of the other at the instant you count. The threshold therefore has to be loose enough that it only fires for the failures you would have noticed anyway: a shard that never built, a backfill process that died partway through.

Freshness is the one that earns its place at cutover, because of the particular shape in which a migration goes wrong. The new index is rarely incomplete; it is behind. The backfill finished, the catch-up is still draining, and the replay consumer is six hours in arrears, which shows up as a stale document rather than as a missing one. A lag of six hours on v3 means that flipping the alias silently reverts the corpus by six hours for every user: every query succeeds, every result is a real document, and six hours of edits, deletions and new arrivals are simply not represented. Nothing in the system is in a position to complain, which is exactly why the check has to be made deliberately before the flip rather than inferred from alerts after it.

Top-k overlap needs interpreting rather than thresholding. Run a few thousand queries against both indexes and compute the mean Jaccard overlap of the top ten, which is the share of ids the two result sets have in common; between two genuinely different, genuinely good embedding models on the same corpus, expect something in the range of 0.35 to 0.6. An overlap of 0.97 means the config did not take effect and both queries hit the same generation. An overlap of 0.03 means something is broken — an encoder mismatch, a normalisation difference, a collection built with the wrong prefix scheme. A reading inside the band in the middle tells you the migration is real and tells you nothing about whether it is better.

The reason that middle band carries no verdict is a property of the measure itself. Jaccard overlap is symmetric: swap the labels on the two result sets and the number is unchanged, so the measure has no notion of which set is the correct one. It can report that the two indexes are answering differently, and it has no vocabulary in which to report that one of them is answering better. A reading in the middle of that band is equally consistent with a v3 that is a large improvement and a v3 that is a large regression, and any instinct to read the band as reassuring is reading a quality signal into an instrument that carries none.

The gate that people over-trust is a frozen evaluation set, and it fails in a way worth stating plainly, because the failure is that the test passes with the control deleted. Follow the chain. A query set frozen six months ago exercises documents that existed six months ago → those are precisely the documents the backfill handled most reliably, since they sat in the snapshot from its first minute → and precisely the documents least affected by chapter 6's catch-up machinery, which only ever touches what changed after the snapshot was taken → while the failures a migration introduces concentrate in recent documents, which are the ones that arrived mid-run. The frozen set therefore scores 0.71 on both generations, and that reading is fully compatible with a v3 that is missing everything written in the last four days.

Shadow evaluation on live queries fixes this by construction rather than by tuning: mirror production queries to both indexes for a day, serve only v2's results, and compare offline. The queries are the ones users are asking now rather than the ones they were asking six months ago, and the documents are the ones in the corpus now, so the comparison is made on the real query distribution over the real current content. It costs double query load on the retrieval tier, which is the honest price and the reason it is a day rather than a permanent fixture.

One confound survives all four gates unless you go looking for it, and it is index tuning. Take the chain in order: the M, ef_construction and ef_search values on v2 were chosen — or inherited — for v2's vectors → a different model produces a different distribution of distances → so the same parameters do not deliver the same recall on v3 as they did on v2 → so comparing a well-tuned v2 against a default-configured v3 measures the tuning rather than the models. The part that makes this dangerous is that the direction of the error is unhelpfully plausible. The new index looks slightly worse, which is exactly what a sceptic expects a new index to look like, so the result confirms a prior instead of provoking a question, and nobody goes looking for the cause. Running the recall-against-exact measurement from chapter 3, where an approximate index is scored against a brute-force search over the same vectors, separately on each generation before the comparison, and adjusting ef_search until both sit at the same recall — 0.97 is a reasonable target — makes the relevance comparison a comparison of models at equal structural quality. Skipping that step is how teams reject a better model.

Moving the traffic, and keeping the comparison honest#

Canarying at 5% then 50% then 100% is standard practice and has one retrieval-specific requirement: assignment must be sticky per session, not per request. A user who asks three follow-up questions and gets served by v2, v3 and v2 again has a conversation whose retrieved context changes generation mid-thread. The second answer is grounded in passages the first answer was not grounded in, which produces contradictions inside a single thread, and contradictions of that kind read to everybody involved as a model failure when they are an experiment design failure.

def resolve_index(session_id: str, flags: Flags) -> str:
    bucket = int(hashlib.sha256(session_id.encode()).hexdigest()[:8], 16) % 100
    name = "chunks_v3_voyage3_1024" if bucket < flags.v3_percent else "chunks_v2_e5large_1024"
    span.set_attribute("retrieval.index", name)     # every trace carries the generation
    return name

Caches are the routing layer people forget, and they reproduce the same failure one layer further out. A retrieval result cache keyed on the normalised question text serves v2's results to a v3 session and vice versa, which destroys the experiment and, during rollback, keeps serving the generation you just backed away from until the TTL expires. Including the index name in the cache key costs nothing and makes the switch actually atomic from the user's point of view; the alternative is a cache purge at cutover, which throws away a warm cache at the moment you are least able to absorb the extra load.

The comparison also needs enough traffic to say anything, and the arithmetic is worth doing before the canary rather than after. Detecting a 2-point nDCG@10 difference against the query-level variance typical of production retrieval takes on the order of a few thousand queries per arm. A system serving 4,000 queries a day at a 5% canary sends five of every hundred queries to v3, and five hundredths of 4,000 is 200 queries a day into the treatment arm. Reaching a few thousand at 200 a day is a matter of weeks, over which the corpus, the query mix and possibly the model itself will have moved.

What follows is a division of labour between the two mechanisms. Systems in that traffic range should spend their canary time on error rates and latency, where 200 queries a day is plenty to see a fault, and get their relevance answer from the shadow evaluation instead, where every query goes to both arms and the comparison is paired. A paired comparison needs less traffic because most of the variance in a retrieval metric sits between queries rather than between systems: an easy query scores high on both indexes and a hard one scores low on both. Comparing the two arms' averages, that query-to-query spread is noise that has to be averaged away by collecting more queries. Comparing per-query differences, it cancels — both arms saw the same query, so whatever made that query easy or hard is subtracted out on both sides, and what remains is the difference the two indexes made on it.

Capacity is the other thing the alias swap can expose all at once, and it is a chain rather than a surprise. Until the flip, v3 has served either nothing or a small canary share → so the flip is potentially the first moment it takes full production QPS, the live queries-per-second rate → with a cold page cache and cold filter caches, because nothing has yet pulled its segments off disk or populated its filter structures → so an index that answered a 5% canary in 40 ms can answer 100% at 300 ms while it warms. Shadow mode is the cheap insurance here as well, because mirroring 100% of queries to v3 while serving from v2 puts the new index under real load with no user exposure, and it turns the capacity question into an observation made a day early rather than an incident made at 09:00. Running shadow at full mirror for a few hours before the canary costs one retrieval tier's worth of extra compute and answers both the quality and the capacity question with the same traffic.

Deciding when to drop v2 is a judgement about detection latency, and for retrieval that latency is longer than for most services. Latency and error regressions appear in minutes, because a monitor is watching every request. Relevance regressions appear when enough users have asked enough questions for someone to complain or for a weekly review to run, which is one to two weeks. Keeping v2 alive and receiving writes for fourteen days after cutover is a common choice for that reason, and the cost is the storage plus the dual-write path staying in the code. When v2 is finally dropped, its removal is itself an erasure with the fan-in shape from chapter 1, where deleting one thing means deleting from every place a copy of it landed: the collection, the snapshots that contain it, the dashboards and alerts that reference it, and the dual-write branch in the writer, which is the one that lingers for months if nobody deletes it deliberately.

A generation is not a cause, and drift has no cutover#

The claim that a relevance change becomes attributable to one variable holds under a condition that is easy to lose sight of during a migration: v3 must differ from v2 in exactly one respect. A rebuild is a tempting moment to also change the chunk length, adopt a new prefix scheme or move to a different distance metric, and each of those folds another variable into the thing being called "the generation". The switch is still atomic and the rollback still works, and the measurement is still a measurement of a package. If more than one thing has to change, the honest options are to sequence them as separate cutovers or to accept that the number you get is about the package and cannot be decomposed.

Even with a single variable, atomic routing makes a regression attributable to a generation and does not make it attributable to a cause. Knowing that v3 is 3 points worse on legal documents and 4 points better on support tickets is the beginning of an investigation into chunk length, domain vocabulary or prefix handling, and the switch has only handed you a clean measurement to start from. The value of that measurement is that it narrows the search: a difference that splits cleanly by document type points somewhere different from one that splits by query length or by language.

Everything in this chapter is built around a discrete event with a before and an after, which is the tractable case precisely because the variable is known and the previous state was preserved on purpose. The difficult case is an index where nothing was deployed, no model changed, no cutover happened, and retrieval quality is measurably worse than it was six months ago — where there is no v2 to compare against, because the thing that changed was the corpus and the index's relationship to it. Diagnosing that requires working through the accumulated failure modes in a deliberate order, which is where this topic ends.

The argument for building the second index is not a preference for thoroughness over speed. A migration that converts an index in place produces exactly one number at the end of it, and one number cannot be decomposed into the contributions of the two things that were changing while it was being produced. Two indexes that exist at the same time produce two, and the difference between them is a third quantity that no single-index migration can ever recover.

That is why the gates before the switch are worth running even when they feel like ceremony, and why the ones that matter are the ones that can distinguish something. Count parity and top-k overlap tell you the migration happened; freshness tells you the new index is not quietly older than the old one; the shadow evaluation is the only one of the four that has anything to say about whether the change is an improvement. Run them, keep v2 alive and writing for the two weeks it takes a relevance regression to surface, and the worst outcome available is a rollback that costs an HTTP request rather than three weeks of re-running work you had already done.