Embedding models

Read these first

• What an embedding is

What you will know

• Choose an embedding model with a stated reason beyond leaderboard rank

  Choose for a stated reason — domain, language, dimension, cost — rather than leaderboard position.

• Trace a leaderboard score to its task, corpus, metric, and evaluation protocol, then test whether those assumptions hold for your corpus

  Aggregate benchmarks hide domain, language, query, and chunking effects; a small golden set can reverse the published ordering.

• Compare two models on your own golden query set

  Compare two candidates on your own golden query set; it takes an afternoon and settles the question.

• Measure storage, retrieval latency, and recall as vector dimensionality changes at your projected corpus size

  Higher dimensions increase index footprint and memory bandwidth, while compression or smaller vectors can trade recall for cost.

• Plan for the fact that changing this means reindexing everything

  Changing the embedding model means re-embedding everything, so treat it as a migration.

Chapters

1. 1

   A leaderboard number is one average over dozens of datasets

   A single score per model hides the query style, the domain vocabulary, the language mix, the chunk lengths, the metric and the evaluation protocol underneath it. The reversal that bites most often on a real corpus is query style: benchmarks ask well-formed natural-language questions while your users type three words and a product code. Learn to trace a published number back to what it actually measured.

   Read chapter
2. 2

   Retrieval encoders expect a different instruction prefix on each side

   Many retrieval encoders were trained with a short instruction glued to the front of the text, and with a different one on the query side than on the document side — a "represent this query for retrieval" string against a passage-side counterpart, or none at all on passages. The strings are neither interchangeable nor optional: omit them and the vectors land in a region the model was never trained to score, which reads as a mediocre model rather than as a missing input. Find them where they are actually stated — the model card, the reference implementation, the library's own defaults — and verify by embedding one query both ways and watching what happens to the rank of a document you know is correct. Any comparison that gives one candidate its prefixes and not the other has already chosen its winner.

   Read chapter
3. 3

   An afternoon on your own golden set settles what a leaderboard cannot

   The head-to-head that ends the argument takes half a day: same corpus, same chunking, same query set, same judgments, same k, same index type, with only the model changing — and each candidate given its own documented instruction prefixes, since withholding them quietly handicaps one side. This chapter specifies the run, the number to report, and how many queries you need before a difference is readable.

   Read chapter
4. 4

   Dimensionality is a storage bill before it is a quality decision

   Ten million chunks at 1024 float32 dimensions comes to roughly 40 GB of raw vectors, before the index structure, the metadata and the replicas multiply it. Extra dimensions also cost memory bandwidth on every single query. Measure footprint, retrieval latency and recall together at your projected corpus size, varying the dimension while holding the model fixed — pick a family that publishes several sizes, or truncate and renormalise a model trained to permit it — so the result is about dimension and not about two different models. Treat truncation or quantisation as an explicit recall-for-cost trade rather than a free saving.

   Read chapter
5. 5

   Changing the embedding model invalidates every stored vector

   Old vectors are meaningless coordinates under a new model, so a swap re-embeds and reinserts the whole corpus: a migration with a duration, a bill and a rollback plan, not a configuration change. Work the decision at 40 million chunks where a newer model wins by 6 nDCG on your own properly run golden set — and interrogate the margin before the cost, because on 120 queries it has to survive a confidence interval or a per-query sign test before anyone budgets against it. Then count what else the swap invalidates: every score threshold calibrated to the old model, and the window in which two vintages coexist in one index. Name what would make a real gain still not worth taking.

   Read chapter
6. 6

   Embeddings are a stored artefact, so provider drift corrupts an index

   Unlike a chat model you can swap between requests, an embedding model leaves its output sitting in your database, so a provider quietly updating weights leaves an index holding two vintages of vectors that no longer compare. That is the self-hosting argument specific to embeddings. Weigh it against reindex load being bursty rather than steady, which is the shape hosted pricing handles well, then commit and say how you would detect the failure.

   Read chapter
7. 7

   A defensible model choice names domain, language, dimension and cost

   "It is top of the leaderboard" is not a reason. "It covers our two languages, its 768 dimensions fit our chunk size and storage budget, it beat the alternative by 5 nDCG on our own 120 queries, and we accept the hosted-provider drift risk because we reindex yearly anyway" is. With the storage arithmetic, the migration cost and the hosting trade-off now in hand, this chapter is where they are assembled: which justifications survive a design review, which collapse under a single follow-up question, and how to record the choice so the next engineer inherits the reasoning rather than the conclusion.

   Read chapter