Embeddings are a stored artefact, so provider drift corrupts an index

An embedding model is a function that takes a piece of text and returns a fixed-length list of numbers — a vector — placed so that texts a person would treat as being about the same thing come back pointing in similar directions. A retrieval system embeds every chunk of its documents once, stores the resulting vectors in an index, and at query time embeds the incoming question with the same function and returns the stored vectors closest to it. The word carrying the weight in that sentence is same. When the function that wrote the stored vectors and the function that writes the query vector are not the same function, every comparison the system performs is taken against a ruler the stored points were never drawn with, and nothing in the code path is in a position to complain.

That exposure is the one argument in the hosted-versus-self-hosted question that belongs to embeddings specifically rather than to model dependencies in general. It is also the argument most often stated wrongly, because the protection it points at is version pinning rather than hosting location — a pinned hosted endpoint and an unpinned local checkpoint sit on opposite sides of this risk from where the usual framing puts them.

Why a model name is not a version#

Start with what actually travels when a client calls a hosted embedding endpoint. The request carries a piece of text and a string — the model name — in its body. The provider's service reads that string, resolves it against a routing table of its own, and sends the text to whichever set of weights the table currently points at. Back comes a list of floating-point numbers of the advertised width. Nothing in the request obliges the provider to use any particular weights, and nothing in the response says which weights ran.

The whole failure lives in one property of that chain. A name is a pointer: it identifies something by reference, so the thing referred to can be replaced while the name stays put. A content hash is the opposite kind of identifier, naming something by its bytes, so different bytes produce a different hash by construction and there is no way to serve you something else under the same string. The revision hash in the code below is an identifier of the second kind, and a bare model name is one of the first — a pointer somebody else holds the pen for.

Only one link in that chain is a design decision rather than a requirement, and naming it makes the risk assessable instead of superstitious. Nothing about serving a model over HTTP forces a name to float. A provider can promise that a given string resolves to one frozen set of weights for the life of the string and ship every improvement under a new one, which is exactly what a dated snapshot identifier is: a model name with a date or version in it that the provider contracts never to repoint. A provider can equally treat the name as the product and the weights as an implementation detail to improve in place, which is a defensible policy for a chat model and a destructive one for an embedding model. Which of the two you face is a fact about the contract and is not visible from the wire.

So a version pin is only ever as strong as the provider's published promise about that particular string. Writing a model name into a configuration file and calling the model pinned confuses what your client asks for, which was never the variable, with what the other side answers with, which is.

A coordinate has no meaning outside the function that made it#

A chat model that gets updated underneath you changes the responses you receive from that moment on. The responses you already delivered are unaffected, because they were consumed and are gone, and if the new behaviour is worse you switch back and the exposure ends with the last request. The relationship is transactional and the blast radius is bounded by how long you take to notice.

An embedding model's output is a coordinate, and a coordinate has no meaning except relative to the other coordinates produced by the same function. What you store when you index ten million chunks is not ten million facts about those chunks; it is ten million positions in a space defined entirely by one particular set of weights. The moment a different set of weights starts producing query vectors, every stored position is being measured against a ruler it was not drawn with. Nothing errors, because a vector of the right dimension arrives and cosine similarity is happily computed against it, and the numbers that come back are plausible — 0.72 rather than 0.79, a decline small enough to be attributed to corpus growth for six months.

The corruption that follows is partial, and the partiality is what makes it hard rather than what makes it survivable. The weights change at some instant on the provider's side. Every chunk embedded before that instant is already written and is never touched again; every chunk embedded after it is written by the new function and lands in a different region of the space. Ingestion in a live system is continuous, so the index becomes a mixture of two vintages whose boundary is a moment in time rather than any property of the documents.

Follow what that mixture does to one search, because the bias falls out without knowing anything about either model. The query arrives after the change, so its vector comes from the new weights, and ranking sorts every stored vector by its cosine against it. For a chunk written by the new weights, query and chunk are two points placed by one function, and their cosine reports what the model was trained to report: how much the two texts are about the same subject. For a chunk written by the old weights the same arithmetic runs and returns a well-formed number, but the two vectors were placed by different functions, so the number is part agreement about meaning and part arbitrary offset between two placements — and that second part has nothing to do with the chunk's content.

The direction of the bias is therefore fixed. Every new-vintage chunk is measured cleanly, every old-vintage chunk carries the same extra disagreement on top of its true relevance, and a sort puts a systematically penalised group lower down. The system is now ranking partly by ingestion date with relevance as a residual effect. Two consequences matter for detection: the damage grows as the index fills with the new vintage, so the symptom curve is indistinguishable from gradual quality decay, and the failure is invisible to any aggregate measurement that does not split by ingestion date, because the healthy portion keeps the average respectable. Plainly put, the index has stopped being one space and become two, and averaging over both hides it.

The same problem appears whenever two vintages coexist for any reason. It is what the dual-write and vintage-column discipline from the previous chapter exists to prevent during a migration you control, where a second index is built alongside the first and a column on each row records which model wrote it. Provider drift is that failure arriving without a change ticket, without a start date, and without the shadow index you would have built.

Pinning rather than hosting#

Being clear about what protects against it matters more than the hosted-versus-self-hosted framing suggests, because the protection is pinning rather than hosting. A hosted endpoint that offers dated snapshots — a model identifier that includes a date or version and is contractually frozen — has the same immutability as a local checkpoint. An open-weight model — one whose trained parameters are published for anyone to download and run on their own hardware, as against a hosted model you can only reach through someone else's API — pulled from a hub by name, with no revision hash, can be updated by its authors and re-downloaded on your next container build, which produces exactly the same mixed-vintage index in a system nobody would describe as hosted. Self-hosting only closes this hole when the revision hash, the tokenizer, and the inference stack are all pinned.

That version of the failure has the same shape link for link. A hub name resolves to a branch, a branch is a moving pointer to whichever commit its owner last pushed, and a container build resolves it again on the day it runs. The routing table has moved from a provider's service into a model hub, and the property that breaks survives the move intact.

from sentence_transformers import SentenceTransformer

# Pinned: a specific commit, not a moving branch.
model = SentenceTransformer(
    "intfloat/e5-base-v2",
    revision="b533fe4d1ad3c4b1d1b7c9a02f2f6d5e0b8b1a1c",
)
# Record this string on every row you write.
FINGERPRINT = "intfloat/e5-base-v2@b533fe4d+st3.0.1+torch2.4+fp32"

Each part of that fingerprint is there because changing it changes the vectors. The identifier and commit say which weights; the library version says which pooling and normalisation code ran over the model's output; the framework version and the dtype say how the numbers were computed. Two of the four can move without anyone touching the model, since a dependency bump upgrades the library and a deployment onto different hardware can change the precision. Writing the whole string onto every row is what makes "which vintage is this vector" a query rather than an investigation.

There is a floor of nondeterminism you should expect and ignore. Batch size, GPU kernel selection, and dtype, the numeric precision the arithmetic runs at, all move individual components at the fourth or fifth decimal place, so the same pinned model can produce vectors whose cosine against each other is 0.99998 rather than exactly 1. That is far below anything retrieval notices. Real drift shows up at 0.99 or below, and the gap between the two regimes is wide enough that a threshold can sit comfortably between them.

Detecting a change that raises no error#

The belief that stops most teams from building anything here is that they already have a check which would catch it, and it deserves its strongest form before being dismantled. Embedding endpoints get monitored like any other dependency: request latency, error rate, and a synthetic call asserting that a vector of the expected width comes back. A team that has thought one step further adds a semantic smoke test — embed a fixed pair of strings that ought to be near one another, take the cosine between them, and assert it clears a bar. That looks like a check on the model rather than on the transport, which is why it convinces.

Every one of those returns the same answer whether or not the weights have changed, and a check that is constant across the thing it must discriminate carries no information about it. The width is unchanged, because output dimension is part of the model's published contract and a different width would break every client's schema on the first request. Latency and error rate are unchanged, because the new weights serve normally. The pairwise cosine is unchanged in the sense that matters: two strings near one another under the old weights are near one another under the new ones, since both sets of weights were trained to do the same job. That test computes a number from the weights currently running and compares it against a bar a human chose, so it passes plausibly in both worlds.

What discriminates is a comparison against something the current weights cannot regenerate, and the only such thing is the output vector the old function wrote. Three things have to be stored at index build time, and each earns its place. The exact texts, because the comparison is valid only on byte-identical input. The full vectors those texts produced, every component at the precision it was returned in, because any summary computed from them — a norm, a mean, a pairwise similarity — throws away direction, and direction is the part that moved. And the fingerprint of the configuration that produced them, because when the alert fires the useful question is which pinned configuration the current one diverged from, and a bare number cannot answer it.

The detection mechanism is therefore a canary set: a fixed list of strings, embedded once at index build time, with the resulting vectors stored as reference data and re-embedded on a schedule. Two hundred and fifty-six strings sampled across the vocabulary of a corpus — the whole set of documents you have embedded and stored — is enough, and the check is a cosine against the reference.

import numpy as np

CANARY = load_jsonl("canary_256.jsonl")          # {"id", "text"} × 256
REFERENCE = np.load("canary_reference.npy")      # (256, d), written at build time

def check_drift(encoder, tol: float = 0.999) -> dict:
    current = encoder.encode(
        [c["text"] for c in CANARY],
        prompt_name="passage",
        normalize_embeddings=True,
    )
    sims = np.einsum("ij,ij->i", current, REFERENCE)
    worst = int(np.argmin(sims))
    return {
        "min_cosine": float(sims.min()),
        "mean_cosine": float(sims.mean()),
        "worst_id": CANARY[worst]["id"],
        "alert": bool(sims.min() < tol),
    }

Read what one of those cosines says. Cosine similarity between two vectors of equal width is one when they point in exactly the same direction and falls as the angle between them opens. The two vectors here are the output for one canary string now and the output for that same string at build time, so under an unchanged function they would be the identical vector and the cosine would be one exactly. It is not one exactly, for the reason above: batch composition, kernel selection and precision move components at the fourth or fifth decimal place, which is why a fully pinned model checks in at 0.99998.

That fixes the size of each regime, and the space between them is the whole basis of the test. Repeated runs of one function on one input depart from a cosine of one in the fourth or fifth decimal place. A different function departs from it in the second decimal place or the first, because the string has been placed somewhere genuinely else and the distance owes nothing to rounding. Once the input text is byte-identical and the tokenizer and instruction prefix are held fixed, no third source of variation remains, so a value outside the floor band is attributable to the weights by elimination rather than by inference.

The thresholds in that code are a judgement and should be read as one. Nothing about 0.999 falls out of a property of embeddings; it is a line drawn in the empty space between the two regimes, near enough the floor that ordinary nondeterminism never crosses it and far enough from it that a real weight change always does. Neither line is a guarantee, since a provider could in principle ship a change small enough to land between the floor and the alert, and the check would pass while the index quietly split in two. What makes the threshold safe in practice is the width of the gap, plus the fact that a weight update worth a provider shipping is rarely a small perturbation.

Run that daily, alert below 0.999, and page below 0.99. The check costs 256 embeddings, which is a fraction of a cent and a couple of seconds, and it fires within a day of a silent update rather than within a quarter. Store the reference vectors alongside the fingerprint string that produced them so that the alert tells you which pinned configuration the current one has diverged from.

A second canary catches a wider class of problem for the same effort. Take twenty queries with a known-correct document each, run them against the live index every hour, and record the rank of the gold document. That watches drift, index corruption, a bad deploy of the query-side prefix, and a filter regression, all through one number, and unlike the vector canary it fails when the query path breaks rather than only when the model changes.

What the two options cost#

The argument that self-hosting wins at scale usually rests on an intuition that reindexing forty million chunks must be expensive and that paying per token for it must be worse than owning hardware. The previous chapter's arithmetic contradicts that: the full reindex is 16.8 billion tokens, which is $336 at $0.02 per million, and roughly $8 of rented GPU time if you run it yourself on eight L4s — a mid-range inference GPU rented by the hour — for 75 minutes. Both reproduce from their inputs. Divide the token count by a million and multiply by the per-million rate for the first. For the second, multiply eight devices by the hourly GPU rate quoted below and by a run of seventy-five minutes, which is an hour and a quarter, and the product is the figure exactly, with no other line inside it.

Neither figure is large enough to decide anything, and the reason is that a reindex is bursty. It happens once a year, it wants enormous throughput for a short window, and then it wants nothing. That shape is precisely what per-token pricing handles well and precisely what owned hardware handles badly, since the hardware sits idle for the other 364 days. One hourly GPU rate feeds both the burst row and the always-on row below, so the entire gap between those two GPU figures is duration.

The steady-state load is where the money actually is, and it runs the other way from the intuition. Query-side embedding is one short string per search: at 40 queries per second and 12 tokens per query, that is about 1.2 billion tokens a month, or $25 at $0.02 per million — the query rate times the seconds in a month times the tokens in a query, divided by a million and priced. Incremental ingest of 400,000 new chunks a month is 168 million tokens, another $3. Against that, self-hosting the query path means always-on capacity with a p99 latency budget, so two GPU replicas for redundancy at roughly $0.80 per hour each is about $1,170 a month before anyone's time, which is two devices multiplied by the rate and by every hour in a month. The crossover is far further out than the folklore suggests, and for a moderate-QPS internal search system the hosted option is cheaper on every line including the reindex.

Those two hosted token lines are the only hosted steady-state spend there is, and their sum rounds to the hosted steady-state row in the table with no residual. That closure is what makes the comparison usable: no unitemised third line hides inside the hosted total and quietly flatters it.

One line in that comparison is routinely mispriced in the hosted column, which is the cost of the reindex measured in engineering attention rather than dollars. The previous chapter's 56-hour hosted backfill is the token count divided by a tier's tokens-per-minute allowance, so it measures a contract rather than how fast the model runs, and a job that long is not one you start and walk away from: it needs concurrency tuned to stay under the limit, backoff that distinguishes a 429, the response a provider returns when you are over your rate limit, from a 500, which means their side failed, checkpointing so a restart does not redo forty hours, and someone watching it across two nights. The self-hosted equivalent runs in 75 minutes on rented hardware and mostly needs a batch size and a progress bar. Teams that have done both usually report the hosted reindex as the more irritating of the two, and the direction is what surprises: the self-hosted run costs about a fiftieth as much in compute and also consumes far less of a person, so on this line the cheaper option is the easier one. The irritation is worth naming in a proposal because it is the part that consumes a person rather than a budget line.

Two rows in that table override cost entirely when they apply. If a corpus cannot leave your network for regulatory reasons, the decision is made and the price of GPUs is a budgeting exercise rather than a comparison. If the plan involves fine-tuning an encoder on your own query-document pairs — which is the single largest available quality gain on a specialised corpus, larger than any model swap discussed here — the hosted catalogues mostly do not offer it, and that forecloses the option.

Deprecation, and the calendar it runs on#

Pinning a dated snapshot replaces one risk with another rather than removing risk, and the replacement is deprecation. A pinned snapshot cannot change under you, and it can be retired: providers announce a sunset date, typically with six to twelve months of notice, and on that date the migration from the previous chapter happens on their calendar instead of yours. The trade is still a good one, because the two risks differ in kind — drift is silent, undated and partial, while deprecation is announced, dated and total.

Detection for deprecation is unglamorous and has to be deliberate, because nothing in the runtime tells you and the snapshot serves normally until the sunset date. Subscribe the on-call rotation to the provider's deprecation feed, assert the exact snapshot string in a startup check so a silent config change to a floating alias fails loudly, and keep a calendar entry that re-reads the model lifecycle page each quarter. The startup assertion works for a reason worth separating out: the fault it catches originates on your side of the boundary, in a config edit or a default resolving to an alias, so your own process can observe it. No check running on your machines can observe a change on the provider's side, which is why that one needs a stored reference vector instead. The mitigation is that the migration harness built for the last swap still exists, which turns a forced migration from three weeks into a few days.

Where the choice inverts#

Two arguments that get made for self-hosting in this context are weaker than they sound and should be dropped from the case rather than defended. The first is that self-hosting removes a dependency, which is true only in the sense that it replaces a dependency on a provider's uptime with a dependency on your own GPU capacity, your own inference server's memory behaviour under load, and your own ability to keep a CUDA stack — the GPU drivers and numerical libraries the inference server sits on — patched. The second is that self-hosting is faster, which is true for the network hop and often false end to end, since a hosted provider running large batched inference on hardware you would not buy can return a 12-token query embedding in 25 ms while an under-provisioned local replica queues behind an ingest batch and takes 200. Both arguments are about models in general; the one that is specific to embeddings, and therefore the one that carries weight here, is the stored-artefact argument.

The choice inverts on a small number of specific facts rather than on taste. No dated snapshot on offer means the drift risk cannot be closed and self-hosting becomes the answer. Query volume in the thousands per second moves the steady-state column decisively, because that column scales with traffic while always-on GPU cost does not. A plan to fine-tune on domain pairs rules the hosted option out. Each of those is a sentence someone can check, which is the property that makes a decision reviewable, and assembling the whole choice into a handful of such sentences is what a design review is actually asking for.

For a system that fully reindexes about annually, runs tens of queries per second, and has no residency constraint, the hosted API is the right choice — taken with a dated snapshot identifier rather than a floating alias, with the model fingerprint recorded on every row, and with both canaries running from the day the index is built. The workload's expensive-looking part is bursty and cheap, its steady part is small, and the one risk genuinely specific to embeddings is closed by snapshot pinning rather than by hosting location. Paying $14,000 a year in always-on GPU — the monthly always-on figure times the months in a year — to solve a problem that a version string solves is a poor trade.

What earns that conclusion is the mechanism rather than the price list. The vectors in an index are the output of one function, held for years, and readable only against that same function, while a model name is a label the other party controls, so it records what you asked for and never what you got. Close the distance between those two by naming the weights with something immutable, writing that name onto every row, and keeping a build-time vector that a silent replacement cannot regenerate. Do that and the location of the hardware becomes what it should have been — a question about latency, compliance and cost, none of which can quietly corrupt six months of an index.