The exact string handed to the encoder is the only evidence worth trusting

An asymmetric embedding model needs a short literal string on the front of its input saying which role the text is playing — query: for a question, passage: for a document, in the case of the E5 family. Chapter 2 established where each of those two strings is evaluated. This chapter is about the gap between the constant you wrote and the bytes the model actually saw, which is wider than it looks, because a sentence-transformer wrapper, a client library, a framework retriever and a hosted provider are all willing to contribute a prefix of their own without saying so.

Three defects live in that gap and they are not the same defect. A missing prefix means one side reached the encoder raw. A reversed pair means each side got the other's string. A duplicated prefix means two layers both applied the correct one, so the model saw it twice. All three produce well-formed vectors, on-topic results and a clean error log, and the only thing that tells them apart is the literal argument at the moment of the encode call.

Prefixing is not idempotent#

An operation is idempotent when doing it twice leaves the system where doing it once did, and prefixing is not one of those operations. Anyone who has debugged an & in rendered HTML, or a URL with %2520 in it where a space was meant, has met this shape before.

That comparison is an analogy and worth holding at arm's length, because the place it breaks is the place the danger lives. Escaping and encoding are transformations that look harmless to repeat and are not, and the reason they get repeated is that each layer applies them defensively so as not to depend on the layer below having done it. That instinct is correct for null checks and for input validation, where doing the work twice costs a branch. So far the two cases match. Where they part company is in what happens next: a double-escaped ampersand renders as visible garbage on the page and a double-encoded URL fetches a path no server has, so the double application announces itself within seconds. A double prefix produces a valid vector that is slightly in the wrong place, and announces itself never. The analogy is useful for explaining why the duplication happens and useless for predicting how you will find out.

Prefixing is wrong to repeat because the encoder does not deduplicate anything; it sees a token sequence and embeds it, and query: query: is a sequence that appeared in exactly zero training pairs. The step underneath that is worth saying plainly, since it is the same step as in chapter 1: the model's response to a prefix is learned association with a fixed token sequence, and the sequence it associates with the query role has one copy of the string in it, not two. A second copy is unfamiliar input rather than emphasis.

The cost of the doubled prefix is smaller than the cost of a missing one, which is part of what makes it durable. The extra tokens push the vector a little way off the position the model would have produced, and since both copies point at the same role, the direction of the error is roughly the direction of "more query-like" rather than a different region entirely. Small, consistent, and just large enough to reshuffle the top of a ranked list.

Five formattings and what each one costs#

Here is the full set of what your pipeline can do to a string, with what each one produces and what it is worth on the same 480 labelled queries and the same corpus of 4.2 million chunks.

Every row in that table is a system where nothing throws, every vector has unit norm, and the results are on-topic. The two numeric columns are worth comparing as ranges rather than row by row. The cosine column runs from 1.000 down to 0.958, a span of about four hundredths. The recall column runs from 0.79 down to 0.24, a span of 55 points. So a geometric change of a few per cent maps onto a ranking change of well over half the metric's usable range, and the only thing that distinguishes the rows is a handful of characters that no type in the system knows about.

That mismatch between the two spans is the single most useful fact in this chapter, and it has a mechanical explanation. A cosine of 0.981 against the correct vector sounds like a near-miss because 0.981 is close to 1.0 on the scale a person imagines for similarity. Ranking does not read that scale. What ranking reads is the ordering of a few thousand candidate documents whose scores against the query are themselves packed into a narrow band — in a dense embedding space, the documents competing for the top ten are separated from each other by hundredths, sometimes by thousandths. A perturbation of the query vector that is of the same order as the gaps between candidates will reorder them, and a perturbation several times larger than those gaps will reorder them thoroughly. The right question about a cosine is never "is it close to 1.0", it is "is it small compared with the distance between the things I am trying to tell apart".

The ordering of the last two rows is worth pausing on, because it is initially counter-intuitive. Having no prefix at all on the documents scores 0.52, while putting the wrong prefix on them scores 0.31, and a fully reversed pair scores 0.24. Doing nothing beats doing the wrong thing. A raw document sits wherever the encoder puts unmarked text, which is a neutral region that still carries the document's topic. A document marked query: has been actively placed in the region reserved for questions, which is where the query it should be answering also lives — so it now competes with the query's own neighbourhood, and the documents that win are the ones that most resemble the question rather than the ones that answer it.

The two places the defect comes from#

Automatic prefixing by a wrapper is the first. Several layers do it, and they do it silently because doing it silently is the feature.

# sentence-transformers >= 5.0 reads prompts from the model repo's config
model = SentenceTransformer("intfloat/multilingual-e5-large")
model.prompts
# {'query': 'query: ', 'document': 'passage: ', 'passage': 'passage: '}

model.encode_query("how do I rotate an API key")        # library prepends 'query: '
model.encode_query(QUERY_PREFIX + "how do I rotate…")   # now there are two
model.encode(text, prompt_name="query")                 # library prepends
model.encode(QUERY_PREFIX + text)                       # you prepend; also fine
model.encode(QUERY_PREFIX + text, prompt_name="query")  # two again

Framework retrievers do the same thing one level up. LangChain's HuggingFaceBgeEmbeddings carries a non-empty query_instruction default and applies it inside embed_query while leaving embed_documents alone, which is correct for BGE and wrong the moment someone points the same class at an E5 checkpoint. Hosted APIs move the logic server-side: Cohere and Voyage both take an input_type parameter with values like search_query and search_document, and the provider prepends or otherwise conditions on it before the text reaches the model. A request that sets input_type="search_query" and sends "query: how do I…" as the text is the doubled case again, with the second copy applied on a machine you cannot log.

The second source is a shared helper reached from both directions. This one produces the reversed case, and it is the more embarrassing of the two because the reversal is usually invisible at every call site.

# embeddings/client.py — written for the ingestion job, six months ago
def embed(texts: list[str], prefix: str = "passage: ") -> list[list[float]]:
    return _model.encode([prefix + t for t in texts], normalize_embeddings=True)

# search/handler.py — added later, by someone reading the signature and not the default
vec = embed([user_query])[0]        # silently becomes "passage: how do I rotate…"

# jobs/reindex_legacy.py — a backfill that predates the search path
vecs = embed(chunks, prefix="query: ")   # someone flipped it during a rebase

Both files read as correct code. The search handler calls a function named embed on a query and gets a vector; the backfill passes an explicit prefix, which looks more careful than relying on the default. There is no test that fails, because a unit test on embed asserts that the prefix it was given ends up on the front, and it does.

The one line that separates the three cases#

The inspection that resolves all of this is to capture the final string on each side, immediately before the encode call, and read it. Not the constant, not the call site, not the config — the argument as it exists at the moment the encoder receives it, with the whitespace visible.

import logging

log = logging.getLogger("embed.trace")

def traced(encode_fn):
    """Wrap the innermost encode call. Nothing below this point can add a prefix
    on our side of the network, so whatever we see here is what the model gets."""
    def wrapper(texts, **kwargs):
        for t in (texts if isinstance(texts, list) else [texts])[:2]:
            log.info("encode side=%s input=%r", kwargs.get("prompt_name", "-"), t[:80])
        return encode_fn(texts, **kwargs)
    return wrapper

_model.encode = traced(_model.encode)

The %r is the part that matters, because repr shows you the trailing space, the leading newline the chunker left in, and the tab that a Markdown table cell contributed. A log line that prints input=query: how do I rotate an API key and a log line that prints input='query: query: how do I rotate an API key' carry very different amounts of information, and only the second one is worth writing to disk. The quotes are what make a trailing space visible at all; a bare %s renders a correct string and a string with two trailing spaces identically, which puts the log itself in the category of check that cannot discriminate the thing it was written to check.

Run one search and one ingestion of a single document with this in place and the output settles the question immediately:

INFO embed.trace encode side=query    input='query: query: how do I rotate an API key'
INFO embed.trace encode side=-        input='passage: To rotate a credential without downtime, first'

The query side is doubled and the document side is right. That is a code fix with no reindex, which — following the cost asymmetry from the previous chapter — is the outcome you hope for. Had the second line read input='query: To rotate a credential…', the same two log lines would instead have told you that 4.2 million vectors need rebuilding, and you would know it in the first ten minutes rather than after a week of model comparisons.

Sampling two texts per batch and truncating to 80 characters keeps the volume sane; you are looking at the front of the string and nothing else. In a service, gate it behind a flag or a sampling rate of one in ten thousand requests and leave it on permanently, because a log line you can grep six months from now is worth more than an investigation you have to reconstruct.

Asserting on the string the encoder receives#

A test can do the same job as the trace, and most tests written in this area do not, for one reason worth naming carefully. The natural unit test asserts that a prefixing function returns its input with the prefix on the front. That is a property of that function alone, and it is true in all three defective systems above as well as in the correct one — the helper that reversed the sides prefixes correctly, the pipeline that doubles the prefix prefixes correctly twice. A check whose result is the same whether the system is right or wrong is not a weak check, it is not a check of that thing at all, and this is the second time in this topic that the shape has appeared.

The test that catches the doubled case has to assert on the string at the boundary where the model is, which means substituting a fake encoder that records what it was handed.

class RecordingEncoder:
    def __init__(self):
        self.seen: list[str] = []

    def encode(self, texts, **kwargs):
        self.seen.extend(texts if isinstance(texts, list) else [texts])
        return [[0.0] * 1024 for _ in self.seen]

def test_query_path_sends_exactly_one_prefix():
    rec = RecordingEncoder()
    service = SearchService(encoder=rec)          # the real service, real wiring
    service.search("how do I rotate an API key")
    assert rec.seen == ["query: how do I rotate an API key"]

def test_ingest_path_sends_the_passage_prefix():
    rec = RecordingEncoder()
    IngestJob(encoder=rec).run([Chunk("c1", "To rotate a credential…")])
    assert rec.seen == ["passage: To rotate a credential…"]

Two properties make this test worth writing where the earlier one was not. It runs the real service through its real construction path, so any prefix contributed by the framework, the wrapper or a middleware appears in rec.seen. And it asserts equality against a literal rather than checking startswith, which is the difference between a test that passes on query: query: how do I… and one that does not. A startswith assertion is the natural thing to reach for and it returns True for the correct string and for the doubled one alike, which makes it blind to precisely the defect that is hardest to find by reading.

The same pair of tests, one per side, also pins the reversal. A shared helper whose default sends the passage prefix down the query path fails test_query_path_sends_exactly_one_prefix on the first run, with a diff that shows the wrong literal rather than a recall number three weeks later.

There is a design rule underneath both the test and the trace, which is that exactly one layer in the stack owns the prefix and every other layer is forbidden from touching it. Which layer you choose matters less than the choice being explicit and written down. Owning it in your own code means constructing the encoder with prompts disabled — SentenceTransformer(..., prompts={}) or the equivalent — so the library cannot contribute one. Owning it in the library means passing prompt_name and never concatenating anything yourself. Delegating it to a hosted provider means sending raw text with an input_type and treating any prefix in your codebase as a bug. The failure mode in every one of these systems is a second layer applying the rule defensively because nobody could say for certain whether the first one already had.

Where the trace runs out#

The wrapper sits at the last point in your process, which means it is authoritative only for the transformations that happen in your process. When the encoder is a hosted API, the string you log is the string you sent, and whatever the provider does with input_type afterwards is outside the frame. For those, the equivalent evidence is a controlled pair of requests: embed the same text twice, once with input_type="search_query" and once with input_type="search_document", and check that the two vectors differ. If they come back identical the parameter is doing nothing, and if they differ you have confirmed the provider is conditioning on it and that you should not be prepending anything yourself.

The trace is also a statement about now. It tells you what the current code does on the current deploy, and the index in front of you was written by whatever the code did on every deploy since the collection was created. A clean trace today is entirely compatible with a corpus containing two million vectors built by a job that has since been deleted, which is why the identity probe against stored vectors remains the right instrument for the historical question and the trace is the right instrument for the live one.

Sampling has a blind spot of its own that is worth knowing about, because one real defect hides in it. Batched encoding takes a list, and a list comprehension that prefixes texts[0] and passes the rest through — the result of an edit that started as a single-text function and grew a loop — produces a batch where the first element is correct and the other 511 are raw. A trace that logs the first two texts of every batch sees the correct one and one wrong one, which is enough if you read both lines and not enough if you glance at the first. Logging the first and last element of each batch, rather than the first two, costs nothing and covers this shape.

And it does not tell you the size of the problem. You will know that the query prefix is doubled or that the documents are raw, and you will not know whether fixing it is worth 3 points of recall or 40 — which matters, because somewhere in the same conversation there is a proposal to replace the encoder entirely, and the only thing that settles that argument is a controlled measurement with a number on the end of it.

The instrument in this chapter is a single line of output, and its authority comes from where it is taken rather than from how much of it there is. Every layer above the encode call is a description of what somebody intended the string to be, and descriptions compose badly: a correct constant, a correct helper and a correct call site can still add up to a string with two prefixes on it. The argument at the innermost call is not a description. It is the input.