recall@50 against precision@5 tells you which lever your evidence supports

Retrieval systems are usually watched by a small set of numbers, and the habit is to read them as a quality report — higher is better, and a low one means something needs fixing. A more useful habit reads them as coordinates that point at a stage. Recall@k is the fraction of queries for which at least one passage a human marked relevant appears in the top k results, and it ignores order entirely: anything inside the window counts the same whether it landed at rank 2 or rank 49. Precision@5 is the fraction of the top five results that are relevant, and it is a statement about ordering conditional on the candidate stage having succeeded.

Two interventions are available when those numbers are unsatisfying, and they cost very different amounts. Adapting the retriever to your domain means taking the embedding model that turns each chunk of text into a vector — trained by somebody else, on somebody else's corpus — and continuing its training on examples drawn from your own material, where each example is a question one of your users asked paired with the passage that answered it. That training changes the model's weights, so every document lands somewhere different when it is encoded again, and distinctions the model previously blurred can be pulled apart. A second and cheaper form of the same idea leaves the retriever alone and trains a reranker, a model that reorders the shortlist the retriever produced rather than deciding what goes into it. Both are commonly called fine-tuning, and both are what this topic covers.

The reason either exists is vocabulary. A published encoder learned what "these two texts mean the same thing" looks like from public text, and public text does not contain a controller board called MX-4471-B, a clinician's "SOB on exertion", or a language with little material on the open web. Where your terms were absent from its training data, the model has no reason to separate things your users need separated, and chunking documents differently or reweighting the search does not recover a difference the encoder cannot represent. Adapting the model is the only lever that reaches the encoder's own sense of similarity, and it is also the most expensive item on the list of things you could do instead, which is why the evidence for it has to be read carefully.

The two numbers form a diagnosis, not a score#

Start with a pair that settles the question without anybody opening a failing query: recall@50 of 0.93 and precision@5 of 0.41, measured over 240 judged queries against 1.8 million support and engineering passages.

Turn each figure into a count, because counts are easier to reason about than rates. Recall@50 at 0.93 says that for 223 of those 240 queries, at least one judged-relevant passage appears somewhere in the top fifty the retriever returns. Precision@5 at 0.41 says that of the five results a user actually sees, about two are relevant. Those two facts together describe a system whose candidate stage is doing its job and whose ordering is not.

The gap between them is exactly the territory a reranker owns, and the subtraction that measures it is worth doing explicitly. A cross-encoder reorders a list it is handed and never adds to it, so the best precision it could ever produce is bounded by the recall of the stage in front of it: it cannot promote a passage that was never retrieved. That makes 0.93 the ceiling and 0.41 the floor, and the distance between them, 0.52, is the whole of what any reordering technique in this pipeline could win. The candidate stage's contribution appears in both terms and cancels in the difference, which is why the residual is attributable to ordering alone. A reranker off the shelf can be in front of the evaluation harness by the end of the week. Adaptation moves the geometry of the vector space itself, which is a different and much more expensive intervention, and here it would be aimed at a stage that is not the one failing.

Reading the two metrics separately is what makes the diagnosis fall out in the general case as well:

The two low-recall rows both say "triage" rather than "adapt", and that is the substance of this chapter. A low recall@50 is a symptom with at least four common causes, and only one of them is an encoder that cannot represent your domain's distinctions. Fine-tuning is the correct response to that one cause and an expensive way of failing to fix the other three.

Which k you measure recall at decides whether the table means anything, and the right k is the one your pipeline actually passes downstream. A system that retrieves 200 candidates and reranks them should be reading recall@200; a system that puts the top 8 chunks in a prompt and nothing else should be reading recall@8, in which case the comfortable 0.93 above may be 0.61 and the diagnosis inverts. Report both — the recall at the candidate depth, which bounds what any later stage can achieve, and the recall at the depth the consumer sees, which is what users experience. The gap between those two numbers is the total value available from every reordering technique in the pipeline, and if it is small there is no reranker worth building.

Low precision@5 also has causes that reranking does not touch, which is why the top-right row of the table says "rerank" rather than "rerank and you are done". Near-duplicate chunks are the common one: if a versioned manual contributes five nearly identical passages, a good ranker will place all five in the top 5 because all five are relevant, and the user sees one answer repeated while four other useful documents are pushed out. That is a deduplication problem at result-assembly time, costs an afternoon, and shows up as a precision failure indistinguishable from a ranking failure until somebody reads the actual result lists. The other is judgment sparsity — a result in the top 5 that is relevant but unjudged scores as a miss — which stage 8.3's advice to judge queries completely exists to prevent, and which quietly depresses every precision number computed against a partially judged set.

What a recall@50 of 0.48 has actually told you#

Suppose the same harness returns 0.48 instead. Of 240 queries, 125 have no relevant passage anywhere in the top fifty. The tempting inference is that the encoder has never seen your vocabulary and is placing your documents somewhere unhelpful in its 768-dimensional space.

That inference may be true, and it is not yet supported, for a reason worth stating as a property of the instrument rather than as a caution. The metric records one bit per query: was a relevant passage inside the window or not. That bit reads the same when the encoder ranked the gold passage 4,000th, when the gold passage is not in the index at all, and when a chunk boundary went through the answer so the gold passage no longer contains it. Three quite different states of the world produce an identical reading, and a number that cannot vary across the states it would need to distinguish carries no information about which one you are in. Believing a low recall is evidence about the encoder is exactly that mistake, and it is tempting because the encoder is the component the metric is nominally about.

Those alternatives are cheap to separate, and the separation is mechanical. For each failing query, take its gold passage IDs from the judgments and ask four questions in order: does the passage exist in the index at all; if it does, does the chunk that carries the answer also carry the language that would retrieve it; is the judgment itself right; and does a lexical index find it when the dense one does not. The pair of words in that last question is worth fixing now, because the rest of the topic leans on it: a dense retriever is the vector-based one this topic is about adapting, and a lexical one — BM25 over an inverted index — scores documents by the words they literally share with the query, with no model in the path at all.

from collections import Counter
import re

CODE_LIKE = re.compile(r"\b(?:[A-Z]{2,}[-_]?\d{2,}|\d{3,}[A-Z]{1,})\b")

def triage(failing, gold, index, dense, bm25, chunks):
    """failing: [qid]; gold: {qid: {doc_id}}; index: set of doc_ids present."""
    buckets = Counter()
    detail = {}
    for qid in failing:
        q = queries[qid]
        golds = gold[qid]
        missing = [d for d in golds if d not in index]
        if missing:
            buckets["not_indexed"] += 1
            detail[qid] = ("not_indexed", missing)
            continue
        # Does the gold chunk still contain answer-bearing language?
        if all(len(chunks[d].split()) < 40 or chunks[d].startswith("...")
               for d in golds):
            buckets["chunk_suspect"] += 1
            detail[qid] = ("chunk_suspect", list(golds))
            continue
        bm_rank = min((bm25.rank_of(q, d) for d in golds), default=None)
        dn_rank = min((dense.rank_of(q, d) for d in golds), default=None)
        if bm_rank is not None and bm_rank <= 10 and (dn_rank or 10**6) > 50:
            buckets["lexical_finds_it"] += 1
            detail[qid] = ("lexical_finds_it", bm_rank, dn_rank)
        elif CODE_LIKE.search(q):
            buckets["code_token_query"] += 1
            detail[qid] = ("code_token_query", dn_rank)
        else:
            buckets["dense_blind"] += 1
            detail[qid] = ("dense_blind", dn_rank)
    return buckets, detail

The continue statements are the design decision inside that function, and they matter more than the regular expression. Each query lands in exactly one bucket, and the order of the tests decides which, so a query whose gold passage is both missing from the index and badly chunked is counted once, under the first test that fires. The tests are therefore ordered from most upstream to most downstream, so that a cause is attributed to the earliest thing that could have produced it. That choice is what makes the resulting counts add up to the number of failures rather than exceeding it, and it is also what makes the buckets actionable — the earliest cause is the one whose fix removes the others.

Running that over the 125 failures on a support and engineering corpus produced a distribution worth stating in full, because the shape of it is typical. Eighteen queries had gold passages that were not in the index at all, traced to an ingestion filter that skipped PDFs above 20 MB and had silently dropped 4,100 documents over eight months. Thirty-one had gold chunks whose text ended mid-sentence, where a 512-token fixed-width splitter had separated a symptom description from the remediation steps that answered it, so the chunk containing the answer no longer contained any of the language a person would use to ask for it. Twenty-two were judgment errors: passages marked relevant that a second reader disagreed with, most of them on queries where the labeller had marked one of several equally correct sources.

Twenty-nine of the remaining queries contained something code-like — part numbers such as MX-4471-B, error codes, an internal service name — and BM25 put the gold passage at rank 1 or 2 for twenty-six of them while the dense retriever had it below rank 500.

Add those four counts and they come to a hundred, exactly, leaving twenty-five of the 125 failures unexplained by anything other than the encoder. That the column closes with nothing left over is what makes the residue usable: it is not a rough estimate of how much of the problem is encoder-shaped, it is the failures that survived every other explanation, one query at a time. The whole argument of this chapter rests on that arithmetic, because a bucket that overlapped another bucket would let the same query count as evidence twice.

The code-like cluster is an argument about tokenisation before it is an argument about semantics, and the two retrievers disagreeing by more than two orders of magnitude on the same query is what says so. A subword tokeniser shatters MX-4471-B into five or six pieces that carry no meaning individually, and no amount of contrastive training on a few thousand pairs will teach an encoder to treat that string as an atomic identifier as reliably as an inverted index does by construction. Hybrid retrieval, meaning running the lexical and dense scorers over the same query and merging their two result lists, already solves it, costs a day of plumbing, and keeps solving it for part numbers that are minted after your fine-tune froze.

That leaves twenty-five queries out of the original 125 where the gold passage is present, intact, correctly judged, and lexically unremarkable, and the dense retriever still cannot find it. Fixing the ingestion filter, re-chunking on structural boundaries, correcting the judgments and adding BM25 to the mix lifted recall@50 from 0.48 to 0.79 in nine days of work by one engineer.

Set the achieved number against what the buckets would license and a second thing becomes visible. A hundred of the 125 failures had a named cause, and recovering all hundred of them would have put recall near nine in ten rather than at 0.79. The shortfall is the honest signature of partial fixes: re-chunking recovers a passage only if the new boundary happens to keep the answer with its retrieving language, and a corrected judgment only helps if the passage it now points at is one the retriever can reach. Diagnosing a cause and removing it are different achievements, and the gap between them is worth quoting alongside the improvement. The twenty-five survivors are the first real evidence that the encoder cannot see a distinction the domain depends on, and if that residue had been two hundred queries rather than twenty-five, the case for adaptation would be made.

Documents, chunks and labels a fine-tune cannot touch#

Adaptation does not add documents to your index, so no fine-tune recovers the 4,100 PDFs the ingestion filter dropped. It does not repair chunking, because the model can only score the text it is given and a chunk that omits the answer scores correctly when it scores low. It does not fix judgments, and it interacts with bad ones badly: the same wrong labels that depressed your recall become the training signal in the next chapter, so the fine-tune learns the labeller's mistakes and the evaluation that measures it shares them, which produces a number that improves while the system does not.

It also does not sit in opposition to reranking. Both raise precision, and a fine-tuned bi-encoder — the retrieval encoder itself, so called because it embeds query and document separately, which is what lets the documents be encoded in advance — generally raises precision@5 as well as recall@50, so the shallow argument that adaptation "lifts both metrics together" is factually true and still the wrong basis for a decision. The question a lever has to answer is what it buys per unit of cost against the alternatives. A reranker costs a day, changes no stored data, and can be removed by deleting a stage; the residue of failures it cannot address is bounded by the recall it inherits. Adaptation costs weeks, invalidates every vector in your index, and is the only intervention that raises that inherited ceiling. Spending it on a system whose ceiling is already 0.93 buys the part of the improvement you could have had for a day's work, at forty times the price.

The asymmetry in the cost of being wrong is what should settle the order of attempts. Building a reranker when adaptation was needed wastes a week and leaves you with a reranker, which is a component you were going to want anyway and which loses nothing; the evidence for adaptation is stronger afterwards, because a system with good ordering has failures that are unambiguously candidate-stage misses. Adapting when a reranker was needed wastes six weeks, invalidates the index, and produces a smaller improvement than the cheaper option would have. Cheap-and-partly-right before expensive-and-possibly-right is the general rule, and it holds here with unusual force because the cheap option is a strict prerequisite for measuring whether the expensive one is justified.

There is one honest case for adaptation with high recall, and it is worth naming so the rule does not get applied mechanically. If recall@50 is 0.93 but your latency budget cannot afford to retrieve fifty candidates and rerank them — a per-query budget of 40 ms, say, where reranking fifty pairs costs 35 ms on its own — then recall@5 is the number that governs, and it may be far below 0.93. Improving the first stage is then the only lever that fits in the budget. That is a resource constraint driving the decision rather than a retrieval failure, and it should be argued as one.

The twenty-five surviving queries are the entry condition for everything that follows. Having them means the geometry is the problem, and moving geometry means training, and training means labelled examples of what your users ask paired with the passages that answer them. Most teams stop at that sentence, because it sounds like an annotation project with a budget attached. It usually is not one, and the next chapter is about where those pairs already exist.