Changing the embedding model invalidates every stored vector

An embedding model is a function that takes a piece of text and returns a fixed-length list of numbers, called a vector. A retrieval system runs every chunk of its documents — a chunk being one passage-sized slice of a source document — through that function once, stores the resulting vectors in an index, and answers a query by embedding the query text with the same function and returning the stored vectors closest to it. A corpus of a few tens of millions of chunks is an ordinary size for this arrangement. Replacing that model with a newer one is the subject here, and it is the one change to a retrieval system that has no in-place form.

The reason it has no in-place form is that the stored vectors are not a description of the documents. They are a description of the documents relative to axes that one particular training run happened to produce, and a different training run produces different axes. That makes a model swap a full re-embed and reinsert of every chunk — a migration with a duration, a bill, a verification step and a rollback plan — and it makes several things that are not vectors invalid at the same moment, including every score threshold somebody calibrated against the old model's output.

Where a vector's axes come from#

Each position in a vector is a coordinate along one axis, and the number sitting at position i is how far the vector reaches along the i-th axis. Nobody specifies in advance what those axes are. They are whatever the training run settled into: a consequence of the random values the weights started from, the order the training data arrived in, the objective the run optimised, and every other detail of that particular run. Two runs of the same architecture on the same data, differing only in their starting values, produce two sets of axes that are each internally consistent and have no correspondence with one another.

The consequence follows in three steps, and each is checkable on its own. First, the similarity measures a retrieval system uses — cosine similarity, and the dot product underneath it — work by multiplying the component at position i of one vector by the component at position i of the other, and summing those products across every position. Second, that multiplication only asks a meaningful question when position i means the same thing on both sides: it is asking whether the two vectors agree about that one axis, which presupposes there is a single axis for them to agree about. Third, nothing anchors dimension seventeen of one model to dimension seventeen of another. There is no registry of dimensions, no standard ordering, no shared vocabulary that both training runs consulted.

Put those together and a similarity computed between a vector from one model and a vector from another is a sum of products over pairs of coordinates measured against unrelated rulers. The arithmetic is valid arithmetic. Every multiplication is legal, the sum is a real number, and the number lands in the range cosine always lands in. What it is not is a statement about the two documents, because the quantity it computes — how much these two agree, axis by axis — was never defined across two models.

One requirement falls straight out of that, and it is the only one the mechanism forces: the query and the documents must be embedded by the same model. Which model is a design decision, revisited whenever the evidence justifies it. That the two sides match is not a decision at all, and a system where they can drift apart is broken in a way no amount of tuning reaches.

A detail that makes this harder to catch rather than easier is that two different models can agree on the number of dimensions. Matching dimension counts are necessary for the arithmetic to run at all, since a component with no partner has nothing to multiply, and they are nowhere near sufficient for the result to mean anything. When a new 1024-dimension model replaces an old 1024-dimension one, the only structural guard the storage layer has quietly passes.

The plain version of all of this is worth stating before moving on. A stored vector is not a property of a document, in the way that its length in bytes is a property of it. It is a measurement of that document, taken on an instrument, and a measurement is only interpretable next to other measurements taken on the same instrument. Changing the model changes the instrument, and every reading already on file was taken on the old one.

The check that passes either way#

The belief that lets this reach production is not a claim about vectors. It is a claim about defences: that if the vectors in an index were wrong, something in the path would say so. That belief is tempting because it holds almost everywhere else in a data system. Corrupt a row, drop a column, deserialise with the wrong schema, and you get a crash, a null, a type error, or output that is visibly garbage. A vector has no type beyond its length and no self-description at all — nothing inside a stored vector records which model produced it.

Work through what a query against a mixed-vintage index actually meets. A vector(1024) column in Postgres checks the number of components and the element type, and checks nothing else. A norm check is no help either, because encoders that normalise their output return unit-length vectors whatever the input was, so the norm is the same constant under both models. A range check on the score is no help, because cosine is bounded at both ends by construction and no cross-model score can fall outside those bounds. The approximate-nearest-neighbour search walks its graph and terminates with the requested number of results whatever the query vector is, since the graph has neighbours everywhere.

So the query returns neighbours. They are ranked. Their scores sit in the band the team is used to seeing. Nothing errors, nothing logs, and no dashboard moves. Every automatic guard in the path — the column type, the norm, the score range, the fact that results came back at all — returns exactly the same answer whether the query vector came from the model that built the index or from a different one. A check that is constant when the thing it must discriminate varies cannot discriminate at all, and that is the precise reason this fault survives review: the defence people are relying on has no ability to fail.

One defence does vary, and naming it is more useful than naming the ones that do not. A judged query set — a set of queries paired with the documents a human has marked relevant — either retrieves those documents or does not, and that answer genuinely changes when the vintages are mixed. The golden set (chapter 3 of this topic, where a team builds its own judged query set in an afternoon) is therefore not merely a quality instrument here. It is the only instrument in the path that knows what the right answer was supposed to be, which is what makes it the only one capable of noticing that the answer is wrong.

What re-embedding forty million chunks costs#

The work a swap implies is arithmetic before it is engineering, because every chunk goes through the new model exactly once. Forty million chunks at an average of 420 tokens each is 16.8 billion tokens to embed, which is the chunk count multiplied by the tokens in an average chunk. Hosted endpoints quote a price per million tokens, so turning that into an invoice is a division followed by a multiplication: 16.8 billion tokens is sixteen thousand eight hundred lots of a million, and at $0.02 for each lot the bill is $336, while at $0.13 it is $2,184. The invoice is not the constraint at any plausible price. What constrains the migration is throughput and time.

The two wall-clock rows are the interesting contrast, because they show the constraint is contractual rather than physical. A hosted tier rated at 5M TPM accepts five million tokens a minute and no more, so the elapsed time is the token count divided by that rate: three thousand three hundred and sixty minutes, which at sixty minutes to the hour is the ~56 hours in the table. Nothing in that number is a property of the model or of the hardware behind it. It is the tier you are on, and a request to raise it or a temporary burst allowance changes the figure more than any code you write.

The self-hosted row is the same 16.8 billion tokens against hardware rented directly. Eight L4s at roughly 1,100 chunks a second each retire eight thousand eight hundred chunks a second between them, and forty million chunks divided by that rate is a little over four and a half thousand seconds, which is the ~75 minutes in the table. Identical work, and the two rows differ by a factor of about forty-five. The compute behind those 75 minutes is nearly free: an hour and a quarter of eight GPUs is ten GPU-hours, and at roughly $0.80 per GPU-hour that is about $8 for the entire corpus, which is a useful figure to hold onto for the hosted-versus-self-hosted argument later in this topic.

The table does not add up, and saying so plainly is more honest than presenting a total. Its rows carry four different units — dollars, hours, engineer-weeks, and dollars again for a storage window — so there is no total line an itemisation could be checked against. Two of the rows are alternatives rather than additions, since you pay the hosted token bill or you rent the GPUs and never both. The dollar figures that do exist come to a few thousand at the very top of their ranges, and the one row carrying no dollar figure at all is the row that dominates. A migration justified by observing that it costs $336 of tokens has been justified on the smallest line in its own budget.

Engineering time is that dominant line, and it nearly vanishes the second time. Building a resumable backfill worker, a way to verify that every chunk was re-embedded exactly once, and a cutover that can be reversed takes 2–3 engineer-weeks the first time. Running the same machinery again eight months later takes a day, which means the first migration is partly an investment in the ones after it, and a team that expects to change models more than once should price it that way.

Everything else that was calibrated to the old vectors#

The vectors are the obvious casualty and they are not the expensive one, because re-embedding is mechanical. What breaks quietly is every constant somebody tuned against the old model's similarity distribution.

The threshold row causes the most incidents, and the reason becomes visible as soon as two models' score distributions are set side by side. Encoders that normalise their output produce cosine values in a narrow band, and the band moves between models: one family clusters relevant pairs around 0.82 with irrelevant ones around 0.71, while another separates them at 0.45 and 0.19. Read those four numbers as levels and the first family looks better, since its scores are higher everywhere. Read them as gaps and the verdict reverses. The first family leaves about a tenth of a point between its relevant cloud and its irrelevant one, and the second leaves more than twice that distance, which makes the second far easier to put a threshold on despite every number it produces being smaller.

Following that through settles what a migrated constant is worth. A 0.78 cut-off against the first family sits between 0.71 and 0.82, which is exactly where an operating point belongs — above the irrelevant cloud, below the relevant one. Carried unchanged onto the second family, the same constant sits above every score either cloud produces, so nothing ever passes and the fallback fires on every query. The old number is not approximately right under the new model. It carries no information about the new model whatsoever, and because the code path still works, the failure surfaces as a support ticket about the assistant refusing to answer rather than as an error anybody can page on.

Recalibrating is mechanical once you accept that. Run the golden set against the new index, collect the top score for each query, and split the queries into those where a relevant document was retrieved and those where none was; the two score distributions overlap somewhere, and the threshold is chosen at the point on that overlap where the false-answer rate and the false-fallback rate trade the way the product wants. Doing it properly needs the negative half of the set — twenty or thirty queries whose answers are genuinely absent from the corpus — which most golden sets omit because they were built by sampling queries that worked. Adding those queries is worth doing once, since every future model change reuses them, and without them a threshold can only be tuned by watching production complaints.

The chunking row deserves care because it can invalidate the experiment that justified the migration. If the new model has a 512-token limit and your chunks run to 700 tokens, migrating means rechunking, which changes what the judged documents are, which means the pooled judgments no longer refer to objects that exist. A margin measured on a corpus that will not survive the change has to be remeasured on the rechunked corpus first. Rechunking on its own, with the old model still in place, is also a comparison worth running, because it may be where some of the gain was coming from.

The shape of a migration that can be reversed#

Two vintages of vectors cannot be compared, so any design in which the search path might see both is broken. That one sentence generates the requirements, and it is worth separating what the mechanism forces from what a team chooses, because the two are usually argued about as if they were the same kind of question.

Three things are requirements. Any single query must be answered entirely within one vintage, which is the mechanism restated. The new index must hold every chunk before any reader is moved onto it, because a chunk that is missing produces no error — it produces a document that has silently stopped being findable. And writes must fan out to both indexes from before the backfill starts rather than from part-way through it, since a chunk written after a backfill worker has already passed its shard exists in v1 and not in v2, and that gap stays invisible until a user searches for it.

Everything else is a design decision. Serving from the old index while the new one builds is one: a read-only freeze or a maintenance window would also keep every query inside a single vintage, and continuous serving is chosen over them on the grounds that a 56-hour freeze is unacceptable to the product. The mechanism itself is indifferent to which of the three is used. Moving readers with an alias flip is another — the search path resolves a name such as chunks_current to a concrete index at request time, so repointing the name moves every reader at once and repointing it again moves them back, where a config deploy would achieve the same thing more slowly. Keeping v1 resident for 14 days is a third, and it is the one line of the cost table a team sets directly, since the duplicate-storage figure scales with the length of the window.

The verification step is a count comparison plus a re-run of the golden set against the v2 index, and it is where the recalibrated thresholds get computed rather than guessed. The count comparison is what the previous section makes necessary: since nothing in the query path can tell a complete index from an incomplete one, completeness has to be established by counting rather than by observing that search still works.

In a store that keeps vectors in Postgres, the same shape is a second column and a partial index:

ALTER TABLE chunks ADD COLUMN embedding_v2 vector(1024);
ALTER TABLE chunks ADD COLUMN embedded_by text;   -- 'model@revision'

-- Backfill in resumable batches; workers claim rows by id range.
UPDATE chunks
   SET embedding_v2 = $1, embedded_by = 'candidate@2f1c9ab'
 WHERE id = ANY($2::bigint[]);

-- Build the new index concurrently so reads on v1 keep serving.
CREATE INDEX CONCURRENTLY chunks_embedding_v2_hnsw
    ON chunks USING hnsw (embedding_v2 vector_cosine_ops)
    WITH (m = 16, ef_construction = 200);

-- Gap check before any flip.
SELECT count(*) FROM chunks WHERE embedding_v2 IS NULL;

Recording embedded_by with the model identifier and its revision hash is the cheapest insurance in the whole procedure, and it is the direct answer to a vector carrying no self-description. It makes "which vintage is this vector" a query rather than an investigation, it makes the gap check trivial, and it is what lets you detect a mixed-vintage index at all — which matters even when nobody intends to change models, for reasons the next chapter takes up, on providers changing a model under a stable name.

The alternative that tempts teams under storage pressure is migrating in place: overwrite each vector as the worker reaches it and keep serving from the single index throughout. It halves the storage requirement and it makes the index incoherent for the duration, because a query embedded with the new model is being scored against a corpus that is part old and part new, and the new vectors will systematically dominate the results in a way that has nothing to do with relevance. For a 56-hour backfill that is more than two days of quietly wrong search with no clean way to reason about it, and no rollback except running the whole migration backwards. Where storage genuinely will not stretch, the workable compromise is to migrate one shard or one tenant at a time, keeping each unit coherent and flipping it independently, which preserves the property that any single query is answered entirely within one vintage.

The rollback is the alias flip in reverse, which is why the old index has to stay resident rather than being dropped the moment the new one goes live. Two weeks is a reasonable window: long enough for a quality regression that only shows on rare queries to surface in support tickets, short enough that the duplicate storage bill stays a rounding error against the migration.

When a real gain is still not worth taking#

Six points that survive the confidence interval, the stratum breakdown and the reranker comparison can still lose to the alternatives, and naming those cases is part of the recommendation rather than a hedge against it.

The gain can be in the wrong place. Six points on retrieval that produce no measurable change in end-to-end answer quality is common when the generator already receives five chunks and the correct one was usually among them; moving it from rank 4 to rank 1 improves nDCG and changes nothing the user sees. If there is an end-to-end evaluation, the migration should be justified against it and not against retrieval alone.

The timing can be wrong. A migration that needs a 56-hour backfill, a two-week dual-index window and three weeks of engineering, proposed six weeks before a launch that the same engineers are staffing, is a good change at a bad moment, and the honest recommendation is to schedule it after.

The new dependency can be worse than the old one. Trading a self-hosted model with pinned weights for a hosted endpoint that has no versioned snapshot buys 6 points of nDCG and takes on the risk that the vectors you spend three weeks writing stop matching the vectors written next quarter. That trade may still be correct, and it has to be stated in the proposal rather than discovered afterwards.

A vector is a reading on an instrument that one training run built, and the axes it is measured against exist nowhere else. Two models can agree on how many numbers to return and still disagree about what every one of those numbers means, which is why a cross-vintage comparison produces a plausible score and a meaningless ranking, and why no dimension check, norm check or range check in the storage layer will ever notice. Changing the model is therefore a migration of the whole corpus, with a wall clock set by a rate limit rather than by compute, an engineering cost that dominates the invoice, and a second casualty list of thresholds, caches and stored assignments that were all calibrated against a distribution that no longer exists.

That last risk generalises past this one decision. A migration you plan has a start date, a verification step and a rollback; the same change arriving without your involvement has none of those, and an embedding model is uniquely exposed to it because its output is not a response you consume and discard but a stored artefact sitting in your database for years.