Retrievers, rerankers and recall

Everything after this chapter is a sequence of interventions on the list of passages that a prompt is assembled from — how many entries it contains, what order they arrive in, which ones get thrown away. The previous chapter held that list fixed and moved a single passage around inside it; from here on the list itself is the thing being changed. To argue about those changes precisely you need names for the pieces that produce the list, and for the two different quantities that people mean when they say retrieval is working.

Holding those two quantities apart is the substance of this chapter rather than its preamble, because a team with one word for "did it work" will spend weeks moving the wrong dial. A retrieval dashboard and a wrong answer can both be accurate reports about the same request. They are scoring different objects: one scores a list of chunks, the other scores a sentence a model wrote.

From a document store to a block of text in the prompt#

The stages are few and each one is a place where a decision has been made, usually by whoever wrote the ingestion script eighteen months ago.

A chunk is a contiguous slice of one source document, sized so that a few of them fit comfortably in a prompt and each is self-contained enough to be understood alone. Typical production sizes are 200 to 800 tokens with some overlap between neighbours; the number matters here mainly because it converts a chunk count into a token count, and everything later in this topic is really about tokens. Eight chunks at 350 tokens is 2,800 tokens of evidence; fifty is 17,500, which is a different prompt entirely.

Both of those figures are one multiplication, and the multiplication is worth saying out loud because of what it assumes. Chunks are pasted into a prompt end to end, so each contributes its own tokens and shares none with any other: eight chunks of 350 tokens is 8 × 350 = 2,800, and fifty of the same chunks is 50 × 350 = 17,500. Nothing cancels and nothing is deduplicated along the way, so a chunk repeating a sentence that already appears in another chunk still costs those tokens a second time. That is also why chunk count is the wrong unit to hold this conversation in. The range quoted above spans a factor of four, so a chunk is a variable quantity of prompt, while a token is what price, latency and every effect in this topic are measured in — which leaves "we retrieve eight" saying nothing at all until somebody multiplies.

The first-stage retriever scores every chunk in the index against the query and returns the best n of them in descending score order. In a dense setup that score is cosine similarity between the query's embedding and the chunk's — an embedding being a fixed-length list of a few hundred numbers, produced by a small model, arranged so that two texts about the same subject come out close together and the angle between them is a usable measure of relatedness. Comparing the query against every chunk exactly would mean 612,000 of those comparisons per request, so the work is done approximately by an index such as HNSW, which walks a graph of near neighbours and returns almost the same answer in single-digit milliseconds. In a lexical setup it is BM25 or a similar term-weighting function over an inverted index, which is a map from each term to the documents containing it. Hybrid systems run both and fuse the two orderings. The property that all of them share, and the one that matters for this topic, is that the query and the chunk are scored independently — the chunk's vector was computed at ingestion time without any knowledge of the query, which is what makes the search fast and what limits how precise it can be.

Chunking deserves one paragraph of attention because it silently sets the units for everything else. Splitting on a fixed token count with a sliding overlap is the default and it cuts through tables, code blocks and the middle of numbered clauses; splitting on document structure — headings, list items, table rows — produces chunks whose lengths vary by an order of magnitude, which makes "eight chunks" a much less predictable number of tokens. Neither is wrong, and the practical consequence is that when you compare two configurations later in this topic you have to compare them at matched token counts rather than matched chunk counts, or you will attribute to the cutoff what actually belongs to the splitter.

The output is a ranked list: chunk at rank 1, rank 2, and so on. That rank is not a probability and is not calibrated. A score of 0.83 for the top hit tells you very little about whether the answer is in it; it tells you only that 0.83 beat everything else in the index. Prompt assembly then takes the top k of that list and serialises them into the request, and the gap between n, the number of candidates the first stage produces, and k, the number that ends up in the prompt, is where every technique in the next two chapters lives.

Two numbers scored on two different objects#

Recall@k is measured over a labelled evaluation set where somebody has marked which chunk contains the answer: it is the fraction of queries where that chunk appears anywhere in the returned list. It is a property of the retriever and it is blind to everything downstream — the ordering within the list, the other nineteen chunks, the prompt, the model.

Answer accuracy is a property of the generated response. Whether it is exact match against a short gold answer, an LLM judge — a separate model call that reads the response alongside the gold passage and scores whether the one is supported by the other — or a human rubric, it is scored on the sentence the user sees.

def recall_at_k(results, gold_chunk_id, k):
    """Property of the retriever's list. Ignores order beyond the cutoff."""
    return any(r.chunk_id == gold_chunk_id for r in results[:k])

def answer_correct(response_text, gold_answer):
    """Property of the model's output. Ignores the list entirely."""
    return normalise(gold_answer) in normalise(response_text)

Holding both is not pedantry, because the two can move in opposite directions in response to the same change. Raising k can only ever increase recall or leave it unchanged — a longer list cannot contain fewer things — while answer accuracy has no such guarantee, and the next chapter is the demonstration that it routinely falls. Reordering a fixed list of twenty chunks leaves recall@20 exactly where it was, since the same chunks are present, while moving answer accuracy by the spread the position probe of the previous chapter measured. A dashboard built only on recall is structurally incapable of showing you either effect, which is why it stays green while the product gets worse.

The one-way guarantee in that paragraph is worth deriving rather than accepting, because it is the reason the two numbers can be relied on to disagree. Recall asks a membership question: is the gold chunk among the first k. The first twenty entries of a ranked list are contained in the first fifty entries of that same list, and a set containing another set cannot be missing something the smaller one had. Raising k can therefore add a hit and can never take one away, whatever the retriever happens to do.

Answer accuracy is not a membership question and gets no such guarantee. It is scored on a sentence a model wrote after reading the whole set, so the set is an input to the thing being scored rather than a container being searched. Adding an element changes what the model was choosing among, and a change of that kind can move the score in either direction. The two numbers are not two views of one quantity, and once that is settled, a change that raises one while lowering the other stops looking like a measurement error and starts looking like the trade it is.

The belief all of this guards against is that a green retrieval dashboard means retrieval is working, and it is worth saying why that belief is reasonable before saying why it fails. Recall genuinely is a property of the retriever, it genuinely is measured correctly, and it genuinely does fall when the retriever gets worse. Nothing about the instrument is broken. What is wrong is the inference drawn from it: the dashboard's object is a list of chunks and the thing a user complains about is a sentence, so a panel made entirely of properties of the list cannot report on a step that happens after the list is handed over. It will hold steady through every failure that occurs there.

Two smaller distinctions are worth keeping in the same drawer. Precision@k is the fraction of the top k that are relevant, which is the number that falls as you widen the list even while recall rises. And a gold chunk label is an annotation someone made, so recall is only as trustworthy as that labelling; in document collections where the same fact appears in several places, single-gold labelling systematically understates recall and makes the retriever look worse than it is.

The second pass exists because the first one is fast#

The reason the first stage cannot simply be made more accurate is structural. Its scoring function must be decomposable — chunk embeddings are computed once at ingestion and reused for every query — because scoring 612,000 chunks per query any other way is not a system anyone can afford to run. That independence is exactly what costs it precision: the embedding of a chunk about parental leave was computed without knowing whether the query would ask about accrual rates, eligibility, or notice periods, so a single vector has to stand in for the chunk's relevance to all of them.

That paragraph compresses four steps, and each of them can be checked on its own. A query has to be scored against 612,000 chunks and an answer has to come back in milliseconds, so nearly all of the work has to have been finished before the query arrived. Work finished before the query arrived cannot depend on the query; that is the ordering of events and nothing more. Whatever is stored per chunk is therefore fixed at ingestion time, and what is stored is one vector for each chunk.

The consequence is a representation obliged to answer for every question anyone will ever ask of that chunk. One vector cannot lean toward accrual rates and toward notice periods at the same time, because it is the same numbers doing both comparisons, so it settles somewhere that serves all of them adequately and none of them well. The ceiling on first-stage precision is therefore not an engineering budget that a better-funded team would spend differently. It follows from the query being absent at the moment the representation was built, and no amount of work done at that moment can put it there.

The arrangement will look familiar to anyone who has worked on a search product, because it is the same one web search settled on decades ago. An inverted index produces a few thousand candidates in milliseconds using a cheap scoring function, and a progressively more expensive cascade of models reorders successively smaller slices of that list, ending with a model that would be ruinous to run over the whole collection and is affordable over forty documents. What changed with retrieval-augmented generation — the arrangement where a search runs first and its output is pasted into the prompt of a generation call — is only what sits at the end of the cascade: instead of ten blue links for a human to skim, the top few go into a prompt, and the consumer has the position sensitivity that the previous chapter measured. The cascade's economics are unchanged.

A reranker removes that constraint over a small candidate set. A cross-encoder reranker takes the query and one chunk concatenated as a single input and produces a relevance score with full attention between the two, which lets it notice that the query says "accrual" and the chunk's only mention of accrual is a cross-reference to another document. It cannot be precomputed, so it runs once per (query, candidate) pair at request time, and its cost is linear in the number of candidates you hand it.

Each clause of that sentence follows from the one before it. The score depends on the query and the chunk jointly, so it cannot be computed until both are known, and the query is known only at request time. Nothing can be cached against the chunk alone either, because the same chunk scores differently against different queries, and that variation is the entire reason the second pass is worth running. What remains is one model pass per candidate: a hundred candidates is a hundred passes. The first stage bought its speed by paying once per chunk and reusing the result forever, and the reranker buys its precision by giving that up and paying again on every request.

The latency figures are medians — p50, the time under which half of requests complete, which is the figure a ranking stage is usually specified in — and they assume a small self-hosted cross-encoder batched on a single GPU; hosted rerank endpoints are in the same order of magnitude and price per search rather than per token. The important column is the third one, because the cost of the reranking stage is set by how many candidates you feed it, and that is a dial you control independently of how many chunks you eventually put in the prompt.

How that dial behaves is worth reading off the figures rather than assuming. Latency does not rise in proportion to the candidate count, because part of what a request costs is paid once whatever the count is — the query is encoded, a batch is assembled, the call is made — and the per-candidate work runs across a batch rather than one candidate after another. The table's own numbers show it: moving from 50 candidates to 200 is a four-fold rise in work and carries the median from 90 ms to 210 ms, well short of four times. The marginal candidate therefore costs less in latency than the average one, so widening the rerank window hurts the clock less than the ratio of the counts suggests. That relief applies to latency alone: the money in the row above is charged per candidate scored, and it stays linear.

candidates = index.search(query, n=100)          # cheap, recall-oriented
scored = reranker.rank(query=query, documents=[c.text for c in candidates])
chunks = [candidates[s.index] for s in scored[:8]]   # precision-oriented cutoff

Those three lines encode the shape of the whole arrangement: retrieve wide, rescore, then keep few. The first stage is asked for recall because it is cheap enough to be generous. The reranker is asked for precision because it is expensive enough that you only run it on a hundred things. The prompt gets eight because — for reasons that are the subject of the next two chapters — the number that goes into the prompt is not the number that maximises recall.

What a reranker cannot repair#

A reranker reorders and truncates a candidate list. It does not put anything into the list that the first stage failed to return, so a reranker cannot repair recall; if the answer-bearing chunk is at rank 340 of the dense search, reranking the top 100 will never see it. Fixing that is chunking, embedding model choice, query rewriting or hybrid search — a different problem with a different budget, and one of the more common ways to spend a month on the wrong thing is to add a reranker to a system whose recall@100 is 71%.

The belief being spent against there is that a reranker will fix recall, and it is tempting for a respectable reason: a reranker is the more expensive model, it is the more accurate model, and reaching for the more accurate model when a number is too low is usually right. The step being skipped is what kind of function a ranker is. A ranking function takes a set of items and returns an ordering of that same set, so its output is a rearrangement of its input and there is no slot in it for an item that was never handed in. A cross-encoder that never sees the chunk at rank 340 cannot promote it, cannot score it, and has no way to learn that it exists.

That is what makes recall@100 of 71% a number that closes the argument rather than opening it. Whatever the reranker does with those hundred candidates, it is working inside that 71%, and the queries whose answer sits outside the hundred were already outside it when the reranker was called. The spend buys a better ordering of a list that does not contain the answer.

Neither does a good ranking guarantee a good answer. The chunks can be perfectly ordered and the model can still ignore the top one, which is the effect the position probe of the previous chapter measured. Ranking quality and use-of-context are separate failure modes with separate fixes, and the whole point of holding recall and answer accuracy apart is to be able to tell which one you have.

The vocabulary is also silent on ordering after the cut. Rerankers return their top k in score-descending order and nearly every pipeline pastes them in that order, but nothing about the retrieval stage requires it, and given the U-shaped position curve there is an obvious question about whether descending is the right arrangement at all. That question only becomes answerable once you know how many chunks belong in the prompt.

The names in this chapter are worth the space because they turn arguments into measurements. A retriever produces a ranked list of chunks and is scored by recall; a reranker rescores the top of that list and is scored by precision at small k; the model is scored by whether its answer is right, and that last number is the only one a user ever experiences. Keeping all three on the same dashboard is what lets you notice that a change bought recall and sold accuracy, which is the trade the rest of this topic is mostly about.

The gap between n and k is where that trade lives, and the case for k being much smaller than n starts with a concrete question about what is actually sitting at ranks 21 through 50 of a real search. The next chapter opens those thirty candidates and reads them.