The model card names the metric; the database default cannot

A model card is the README a publisher ships alongside a set of weights, describing what the model was trained on, what it is for, and how its output is meant to be scored. That last part is the one this chapter is about, and it is the part a vector database cannot supply. The database is handed an array of numbers. It has no way to know which model produced them, what objective that model was trained with, whether the training loss compared pairs by angle or by inner product, or whether the vectors arriving have been scaled to unit length.

The consequence is a defect with an unusual signature. A metric mismatch does not raise an error, log a warning, or produce an obviously broken ranking. It produces a plausible ranking that is slightly worse — on topic, readable, sensibly ordered, and missing several points of recall — which is indistinguishable from a mediocre embedding model. That is why the mistake gets attributed to the model rather than to the configuration, and why it can end a model's candidacy in a bake-off without anybody noticing that the harness was scoring it the wrong way.

Why the default cannot be right#

Every default a vector database ships is a guess about the most common case, and cosine is the sensible guess because the majority of popular embedding models are trained with a cosine objective and emit normalised vectors. A guess that is right most of the time is still a guess, and it is silent when it is wrong.

Cosine's status as the default is also older than any of these systems, which is part of why it feels like a neutral choice rather than a decision. It comes from the vector space model of the 1970s, where documents were represented as sparse term-frequency vectors and length normalisation was introduced specifically to stop long documents from dominating retrieval — a long document contains more terms, so it overlaps more with any query, and dividing by the vector's length was the fix. Cosine was the right default for tf-idf, the term frequency–inverse document frequency weighting those sparse vectors carried, because nothing in that representation was meant to encode length as a signal; length was purely an artefact of how much text there was. Dense embedding models inherited the convention along with the software, and for most of them it remains correct. The models it is wrong for are the ones where a designer deliberately put length back in, which is exactly the case a fifty-year-old default cannot anticipate.

The information the database lacks is written on the model card, in language that is usually explicit. A card that says the model produces normalised embeddings and should be compared with cosine similarity is telling you that magnitude is meaningless and that all three metrics will agree anyway. A card that says the model was trained with dot-product similarity on unnormalised vectors is telling you the opposite: that something was deliberately encoded in the length of the vector, and that a metric which divides length out will discard it.

For the -dot- variants in the multi-qa family, what the magnitude encodes is roughly how much answer-bearing content a passage contains. The training objective rewarded longer, more complete passages with larger norms, so that a passage which actually answers a question outranks a one-line mention of the same topic. That is a deliberate design choice made by people who measured it on question-answering benchmarks, and running such a model under cosine deletes the mechanism they built. The published sibling model multi-qa-MiniLM-L6-cos-v1 exists precisely so that teams who want cosine semantics get a model trained for them, rather than getting the dot-product model scored the wrong way. The existence of that pair is itself the clearest statement available: a publisher who shipped two models differing only in scoring protocol did so because the protocol is not interchangeable.

The last column is worth counting rather than reading. It says "nothing" for five of the seven rows, which is the default being right most of the time, which is the property that makes it dangerous: a team that has used four embedding models without ever setting the metric has four pieces of evidence that the field does not matter, and none of them tested the case where it does. The two rows where the column says something are the entire content of the table, and they differ in kind. One is a degradation and one is a category error.

Two rows are worth reading against each other. The OpenAI row is safe under any similarity setting because the API returns unit-length vectors, so by the collapse established in the previous chapter, cosine and dot product are numerically the same function and Euclidean sorts identically. The Cohere binary row is the case where a wrong metric does not merely degrade results but produces meaningless ones, because the bytes are packed bits rather than floats and the arithmetic being applied to them has no interpretation at all.

The E5 row carries a warning that is worth stating plainly, because a reader who has just learned to check the metric will start checking only the metric. E5 and BGE are asymmetric models: queries and passages go through the same weights but must be prefixed differently, query: and passage: for E5, a retrieval instruction on the query side for BGE. Omitting the prefixes typically costs more retrieval recall than any metric mismatch in this chapter — measurements on standard benchmarks routinely show several points of nDCG@10 for that alone — and it fails silently in the same way, producing scores that look normal because the model still returns vectors and the vectors still have plausible similarities. The general form of the rule is that a model card describes a scoring protocol, of which the metric is one part; the prefixes, the pooling strategy, the maximum sequence length at which text gets truncated, and whether the output is normalised are the others, and each of them is capable of costing more than the metric.

The same setting under five different names#

The names for the same metric differ enough across systems that a mismatch can survive a code review by two people who both know what they are doing:

FAISS — Facebook AI Similarity Search, a library that runs inside your own process rather than a service you call over the network — is the row that catches people during migrations, because it has no cosine option at all. The idiom is to L2-normalise the vectors yourself and then use inner product, which by the same collapse gives cosine ranking. A codebase that did this for two years contains no string anywhere saying "cosine", so when it is ported to a managed service the person doing the port sees IndexFlatIP, picks dotproduct, and inherits a system that only behaved correctly because of a normalisation step living somewhere else entirely.

That is a specific kind of trap and worth naming as one, because the natural way to investigate a port is to search the old code for what it did. A grep for "cosine" over a correct FAISS codebase returns nothing, and a grep over an incorrect one also returns nothing. The search cannot distinguish the two cases, so a negative result carries no information while feeling like evidence. What does distinguish them is a normalize call somewhere in the write path, which is a different string in a different file from the one anyone thinks to look in.

Pinning the metric where it cannot drift#

The fix is to make the metric a property of the model in configuration, so that changing the model string forces the metric to travel with it. A registry keyed by model name, checked into the repository next to the code that reads it, is enough:

# embedding_models.yaml — the metric is not a tuning knob; it is documentation
models:
  multi-qa-MiniLM-L6-dot-v1:
    dim: 384
    metric: dotproduct        # model card: trained with dot-product scoring
    normalise: false          # magnitude encodes passage informativeness
    source: https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-dot-v1

  bge-base-en-v1.5:
    dim: 768
    metric: cosine
    normalise: true
    query_prefix: "Represent this sentence for searching relevant passages: "
    source: https://huggingface.co/BAAI/bge-base-en-v1.5

The registry only helps if something enforces it against the live index. That check is a few lines and belongs in the deployment test suite rather than in a runbook, because a runbook does not run:

import yaml
from pinecone import Pinecone

REGISTRY = yaml.safe_load(open("embedding_models.yaml"))["models"]

def test_index_metric_matches_model_card():
    spec = REGISTRY[SETTINGS.embedding_model]
    desc = Pinecone(api_key=SETTINGS.pinecone_key).describe_index(SETTINGS.index)

    assert desc.metric == spec["metric"], (
        f"{SETTINGS.index} uses {desc.metric}; "
        f"{SETTINGS.embedding_model} documents {spec['metric']}"
    )
    assert desc.dimension == spec["dim"]

Most vector databases fix the metric at index creation and will not let you alter it afterwards, so this test does not merely catch a bad config value — it catches it while the fix is still a re-index of a staging collection rather than a re-index of production. The dimension assertion rides along because it costs one line and fails loudly for the adjacent mistake.

The immutability is the reason this check earns its place in a test suite rather than a wiki page, and the cost of the late fix is worth working out because it decides how early the check has to run. Fixing a wrong metric means creating a new index, writing every vector into it, cutting traffic over, and deleting the old one. On a 52,000-chunk corpus that is genuinely cheap — the vectors themselves can be copied without re-embedding, which is a few minutes of API calls — but on a corpus of ten million chunks with a hosted embedding model it is a re-write measured in hours and a duplicate index sitting in memory for the duration, which on most managed pricing is the expensive part. If the embeddings themselves also have to be regenerated because the fix involved normalisation or prefixes, add the encoder cost: ten million chunks at an average 350 tokens is 3.5 billion tokens, since ten million times three hundred and fifty is three and a half thousand million, which at $0.02 per million tokens for a small hosted embedding model is about $70 — three and a half thousand multiplied by two hundredths of a dollar — and roughly a day of wall-clock time at a few hundred requests per second, or several GPU-hours if you run the model yourself. None of those numbers are frightening; all of them are avoidable by a test that runs in two hundred milliseconds.

Putting a number on the mismatch#

Arguing about metrics is cheaper than measuring them, and worse. The measurement is cheap enough that there is no reason to argue: embed the corpus once, embed the query set once, hold both fixed on disk, and score them under each metric in turn. Only the comparison function changes, so any difference in the result is attributable to the metric alone. That last sentence is what makes the experiment worth the setup rather than simply re-running the pipeline three times: freezing the embeddings removes every other explanation, including a different batch, a different truncation and a different model revision.

import numpy as np

Q = np.load("queries.npy")        # (400, 384) raw model output
D = np.load("corpus.npy")         # (52_000, 384) raw model output
gold = np.load("gold_doc_ids.npy")  # (400,) one correct doc per query

def unit(X):
    return X / np.linalg.norm(X, axis=1, keepdims=True)

def recall_at_k(scores, k=10):
    topk = np.argsort(-scores, axis=1)[:, :k]
    return float(np.mean([g in row for g, row in zip(gold, topk)]))

print("documented (dot, raw):   ", recall_at_k(Q @ D.T))
print("mismatched (cosine):     ", recall_at_k(unit(Q) @ unit(D).T))
print("mismatched (L2 on raw):  ", recall_at_k(
    -np.linalg.norm(Q[:, None, :] - D[None, :, :], axis=2)))

# documented (dot, raw):    0.80
# mismatched (cosine):      0.68
# mismatched (L2 on raw):   0.54

Twelve points of recall@10 for the cosine mismatch, twenty-six for the Euclidean one — on a model whose card names the correct setting in its opening paragraph. The ordering of the three is the mechanism from the previous chapter showing up as data. Cosine divides the norm out, which discards a signal this model deliberately encoded, and costs twelve points. Euclidean does not merely discard the norm; it actively penalises it, pushing exactly the long answer-bearing passages the model was trained to reward away from every query, and costs more than twice as much. A metric that ignores a signal and a metric that inverts it are different sizes of error, and the two mismatch figures report that difference.

Those numbers are specific to that model and corpus and will not transfer, which is the point of running it on yours. What does transfer is the shape: the mismatch penalty is large enough to change a model selection decision and small enough to look like ordinary model quality, and it does not scale with anything you would think to instrument.

Where the card runs out#

Two things this is not. It is not a claim that the card is always right. Cards are written by humans, frequently copied wholesale from a sibling model, and often silent on the question, particularly for fine-tunes published by teams who never changed the scoring protocol and did not think to restate it. When the card says nothing, the norm check above is the cheap structural clue to reach for first. If the norms all come back at 1.000, the model normalises internally and the metric question is already settled by the collapse from the previous chapter. If they spread, the model has left magnitude in the output and you have a real decision, which the recall comparison above resolves — and the registry entry then records what you measured and the date you measured it, rather than what you read.

That is a narrowing of the chapter's own rule and it should be read as one. The card is authoritative when it speaks; when it does not, the authority passes to a measurement, and a registry entry that cites a card it never checked is carrying the same confidence with none of the evidence. The dated, measurement-backed entry is the honest form when the documentation is silent.

And the metric is not a hyperparameter to sweep for the best number. If a mismatched metric happens to score higher on your query set than the documented one, that is a finding worth investigating, not a configuration to ship, because it usually means something else in the pipeline has already changed the vectors before they reached the index. A normalisation step in the write path that nobody remembers is the common explanation, and it makes the "wrong" metric correct for the vectors as stored while leaving the system one refactor away from breaking.

So the metric lives with the model, not with the index, and not in the head of whoever created the index. The database cannot know it, the default is a guess that is right often enough to teach the wrong lesson, and the failure it produces looks exactly like a mediocre model. A registry entry, an assertion against the live index, and one measurement over frozen embeddings are between them cheaper than a single bad model-selection decision.

Which raises the case a correctly pinned metric does not cover. The registry above says normalise: false, and the code that embeds documents will honour it — but a production corpus is rarely written by one piece of code. There is the bulk job that loaded the initial fifty thousand documents, and there is the worker that has been upserting new ones every hour since — inserting each document, or overwriting it in place when the index already holds that identifier — and nothing in the index configuration notices when those two disagree.