Prefix on, prefix off: the run that turns suspicion into a number

A trace tells you what string the model received. It does not tell you what that string is worth. A number for that is obtainable in an afternoon, and the two questions get answered by different instruments. The evidence a trace produces is a literal — either the encoder saw query: how do I rotate an API key or it did not. The evidence this chapter's run produces is a difference between two measurements, and a difference is only attributable to one cause when everything else has been held still.

That is the whole discipline of a controlled arm. An arm here is one complete configuration of the retrieval system, evaluated end to end: a document prefix, a query prefix, an index built from those documents, and a score over a fixed labelled set. Two arms that differ in one respect produce a difference caused by that respect. Two arms that differ in two respects produce a number nobody can interpret, and the second variable usually gets in by accident rather than by choice.

Everything except the formatting has to be nailed down#

The experiment is only worth running if the two arms differ in one respect. That sounds obvious and it is routinely violated, usually because the second arm gets built a week later on a fresher corpus snapshot or with a chunker that has since been improved.

Fix the corpus at a snapshot ID, so both arms see the same 4.2 million chunks and not the 4.23 million that exist by Thursday. Fix the chunker, including its version, its target size, and its overlap, because a change from 180-token to 220-token chunks moves recall@10 by several points on its own. Fix the labelled set — the same 480 queries and the same relevance judgements, with no additions between runs. Fix the rank cutoff, since recall@10 and recall@20 answer different questions and a mixed comparison answers neither. Fix the index parameters, m, ef_construction and ef_search alike, because approximate search recall varies with them independently of embedding quality. Fix the model revision to a commit hash rather than a branch name. And fix the similarity metric and the normalization, which travel together.

from dataclasses import dataclass

@dataclass(frozen=True)
class Arm:
    name: str
    query_prefix: str
    doc_prefix: str

ARMS = [
    Arm("prefixed", "query: ", "passage: "),
    Arm("raw", "", ""),
]

def recall_at_k(ranked_ids: list[str], gold_ids: set[str], k: int) -> float:
    hits = len(gold_ids & set(ranked_ids[:k]))
    return hits / len(gold_ids)

def run_arm(arm: Arm, chunks, queries, qrels, k: int = 10) -> dict[str, float]:
    index = build_index(                     # same params for every arm
        vectors=[encode(arm.doc_prefix + c.text) for c in chunks],
        ids=[c.id for c in chunks],
        m=16, ef_construction=200, metric="cosine",
    )
    scores, reciprocal = [], []
    for q in queries:
        ranked = index.search(encode(arm.query_prefix + q.text), k=k, ef_search=128)
        scores.append(recall_at_k(ranked, qrels[q.id], k))
        rank = next((i + 1 for i, d in enumerate(ranked) if d in qrels[q.id]), None)
        reciprocal.append(1 / rank if rank else 0.0)
    return {"recall@10": mean(scores), "mrr@10": mean(reciprocal)}

for arm in ARMS:
    print(arm.name, run_arm(arm, CHUNKS_SNAPSHOT_7, QUERIES_480, QRELS_480))

Building the index inside run_arm rather than reusing one is the point of the whole function, and it follows directly from where the document prefix is evaluated. A stored vector was produced from a prefixed string, so changing the document prefix changes every vector, so it changes the index. A harness that lets you "switch the prefix" against a fixed index is switching only the string prepended to the query, which means its two arms differ in the query formatting and agree on the documents — a real experiment, answering a smaller question than the one you asked.

What the two arms said#

MRR@10 in the fourth row is the mean over queries of one divided by the rank of the first relevant result, so it reports how far down the list a reader has to go rather than how many relevant documents arrived inside the cutoff. The third column is included because it was the arm the system had actually been running, and it is the reason the number under investigation was 0.31 rather than 0.52. Subtracting the third column from the first at the recall@10 row gives 48 points of recall, on a corpus and a model that had not changed in any other way. Elastic's write-up on multilingual vector search with E5 reports the same direction and a similar magnitude from an independent corpus, which is worth knowing because it means this is a property of the model family rather than a quirk of one dataset.

Latency being identical across all three arms is the practical seal on the argument. Whatever was wrong, it was not something the system was paying for in speed, and there was no efficiency-for-quality trade hiding in the result. It also confirms something about the arms themselves: three configurations that differ in the text sent to the encoder, and agree to the millisecond on how long a search takes, are three configurations that differ in nothing operational. Two extra tokens cost nothing measurable.

Whether 48 points is a fact or a reading is a question about sampling, and it has an answer worth deriving rather than quoting. A recall figure over a labelled set is a proportion estimated from a sample of queries, so its uncertainty is the standard error of a proportion: the square root of the recall times one minus the recall, all divided by the number of queries. Put 0.79 and 480 into that and the result is about 0.019. A conventional 95% interval is roughly two standard errors either side, so the interval around 0.79 spans about ±0.04.

Now compare the gap against that yardstick. Forty-eight points of recall divided by a standard error of a little under two points is more than twenty standard errors, which comfortably clears the ten the result needs to be unarguable — an effect that size is a fact rather than a reading. That arithmetic matters as much for the differences it rules out as for the one it confirms. The same yardstick says anything under about four points on this evaluation set is not distinguishable from resampling noise, and treating a 2-point movement as a signal is a mistake that returns in a more damaging form once you start monitoring a live corpus.

Where 480 labelled queries come from, and how to run this without a second full index#

The objection to all of this is usually that there is no labelled set, and building one sounds like a quarter of work. It is closer to two days. Pull the top few hundred distinct queries from your search logs, weighted by frequency so the set reflects real traffic rather than the interesting cases someone remembers. For each one, run the current system at k=50, put the fifty results in front of a person who knows the product, and have them mark which are genuinely relevant. Judging fifty short results takes two to four minutes per query, and 480 queries at the pessimistic end of that band is a little over nineteen hundred minutes, which is roughly thirty hours of one person's attention. It can be split across a team in an afternoon each.

That set is imperfect in a specific and correctable way: it is drawn from what the current system already retrieves, so a document that has never appeared in the top 50 for any query cannot be judged relevant and the set systematically overstates how well the current system does. Adding a second source of judgements — queries where someone knows the right answer independently, taken from support escalations or from documentation the team wrote in response to a specific question — patches the hole. The absolute recall numbers will still be optimistic. The difference between two arms measured on the same set will not be, which is all this experiment needs, and the reason is that the bias is a property of the labelled set rather than of an arm: both arms are scored against the same inflated gold, so it lifts both by the same amount and cancels in the subtraction.

The other practical objection is the cost of building two indexes over 4.2 million chunks. You do not need the full corpus for the comparison. Sample 50,000 chunks, making sure every gold chunk from the labelled set — every chunk a judge marked relevant for some query — is included, and build both arms over that. At the single-GPU throughput the previous chapter measured for this model, 50,000 chunks is a little under two minutes, and the index build is seconds, which turns the whole experiment into something you can run three times over lunch while you vary a candidate prefix.

Read that table down the columns rather than across the rows. Both levels rise sharply on the sample — the prefixed arm by fourteen points and the raw arm by nineteen — because a gold chunk is competing against 50,000 distractors instead of 4.2 million and the retrieval problem is genuinely easier. The difference between arms falls by five points, from 0.27 to 0.22, compressed by the ceiling effect at the top: an arm already at 0.93 has only seven points of headroom left, so the sample squeezes the gap from above. What survives is the sign, the order of magnitude and the conclusion. Use the sampled run to decide whether formatting matters and to compare candidate formattings against each other; use the full corpus only when you need a level you are going to quote to somebody.

Three checks, in order, before anyone re-embeds#

The first check is the trace from the previous chapter, and it is first because it costs ten minutes and produces a literal string. Its evidence is direct: either the model received query: how do I rotate an API key and passage: To rotate a credential…, or it did not, and there is no interpretation involved.

The second is the on/off run, and its evidence is a delta attributable to formatting alone. It answers a question the trace cannot, which is whether the formatting is worth anything on your corpus with your queries. Some corpora are forgiving — a corpus where every query is a near-verbatim quote from its answer will do respectably in every arm, because lexical overlap carries it — and knowing you are in that situation is useful before you spend a reindex on prefixes.

The third check verifies that the documents you expect to retrieve are actually retrievable. For each labelled query, look up its gold chunk ID directly in the index, confirm the row exists, then fetch the stored payload and compare the text against the source of record.

for q in QUERIES_480:
    for gold in QRELS_480[q.id]:
        row = index.get(gold)
        assert row is not None, f"{gold} missing from index for query {q.id}"
        stored = row.payload["text"]
        source = source_store.text(gold)
        assert stored == source, f"{gold} stored text differs: {stored[:60]!r}"

That loop finds a category of problem no amount of embedding work will fix: chunks dropped by a permissions filter that runs at ingest, payload text stored as raw HTML so the encoder saw <div class="callout"> before it saw any prose, chunks silently truncated because the chunker's token budget was set for a model with a 512-token limit while the splitter counted characters, and IDs that drifted between the labelling spreadsheet and the index. Its evidence is a per-query pass or fail with a chunk ID attached, which is the most actionable output of the three because it points at a row.

The three checks are ordered by cost and by reversibility rather than by likelihood, and the ordering is worth defending on its own terms: together they cost under four hours, none of them changes any stored data, and each one either exonerates a hypothesis or names a defect.

Why the model swap is the dangerous suggestion#

Swapping the encoder while a formatting defect is live is dangerous for reasons that compound. The new model gets embedded through the same pipeline that mangled the old one, so it inherits the handicap and arrives looking mediocre too — and since it is a different family with different prescribed formatting, the swap can introduce a second, independent formatting bug on top of the first. A team that moves from E5 to a model requiring no prefixes, while leaving query: prepended in the search path, has replaced a known problem with a subtler one.

The swap also destroys the evidence. Re-embedding 4.2 million chunks overwrites the vectors that carry the record of how they were built, and unless someone deliberately preserved the old collection, the identity probe stops working and the on/off comparison loses its baseline. The cheapest diagnostic in this whole area is available only while the broken index still exists.

The most damaging outcome is the partial success, and it is worth walking through because it is the case where every individual step is defensible. Suppose the larger model, still handicapped by the same reversed prefixes, lifts recall@10 from 0.31 to 0.38. That is a real improvement. Seven points is well outside the ±0.04 interval derived above, so nobody has misread noise, and it confirms the theory that was proposed in the meeting. The team now believes model capacity is the binding constraint, ships the more expensive encoder, and starts planning the next upgrade — while 41 points of recall sit untouched behind a six-character string. A wrong theory that produces a small confirmation is considerably harder to dislodge than one that produces nothing, because the confirmation is genuine and the reasoning from it is the only step that was wrong.

Whatever the three checks conclude, write down the arm definitions, the corpus snapshot ID, the labelled set version, the index parameters and the resulting table, and keep them next to the code rather than in a chat thread. The next person to propose a model swap will propose it against whatever numbers are findable, and a recorded on/off run is what makes the second version of this conversation take twenty minutes instead of a week. It also gives the eventual, legitimate model comparison a baseline to be measured against, since a new encoder evaluated on the same snapshot with the same queries at the same cutoff is a real experiment and the same encoder evaluated on a corpus that has grown by 300,000 chunks is not.

None of this makes model choice unimportant, and there are corpora where a stronger encoder genuinely is the answer. The ordering is what matters. A controlled pair of arms costs an afternoon and returns a difference with one possible cause; a swap costs a week and, run first, is uninterpretable regardless of how it comes out, because the new model and the old formatting moved together and no arithmetic can separate them afterwards.

The on/off run answers whether formatting is worth something. It does not tell you what the ceiling is when the formatting is right — whether 0.79 is close to everything this encoder can do on your content, or whether something else is still leaving points on the table. For that you need a run in which the two sides cannot be mismatched at all.