Choosing retrieve-k and rerank-to-n means reading two curves against each other

A two-stage search pipeline — a cheap retriever running over an entire corpus, followed by a slower model that reorders what the retriever returned — contains two integers, and configuration files usually expose them as one "top-k" setting. The first, retrieve-k, is how many candidates the first stage hands over, which is also how many query-document pairs the second stage has to score. The second, rerank-to-n, is how many of those candidates survive the reordering and reach whatever consumes the list: a results page, or the retrieved-context block of a prompt.

Treating the two as one knob is the mistake this chapter is about, and it is a mistake with a price attached. Retrieve-k governs a forward pass per candidate and is the expensive parameter in the system. Rerank-to-n governs a slice taken after all that work has finished and costs nothing at all. They are also constrained by different things: one by a recall curve and a latency budget, the other by the size of a prompt.

Depth carries one further surprise that neither of those constraints predicts. The recall curve and the cost curve have different shapes, which is ordinary enough, and the ordering quality that sits between them is not monotonic in depth at all. More candidates can make the final ranking worse, and the reason is specific enough to be measured rather than guessed at.

One curve bends and the other does not#

Recall against depth saturates because the documents you are still missing at depth 50 are missing for reasons that more depth does not address. The first fifty candidates capture the queries where the first stage was roughly right; the next hundred and fifty capture the ones where it was nearly right; the queries where it was wrong about what the question meant are not in the top thousand either. On the hybrid pipeline from chapter 2, where a lexical arm was fused with the dense retriever so that both contribute candidates, measured over the same 200 judged queries, the curve looked like this.

Start with the last column, because it is the one with no shape in it at all. Every entry is the depth multiplied by the per-pair inference time in the column header: ten pairs at seven-tenths of a millisecond is seven milliseconds, fifty pairs is thirty-five, a thousand pairs is seven hundred. Multiplication is the right operation here because the second stage does one forward pass per candidate and each pass is independent of the others, so there is nothing to share between them and nothing that gets cheaper as the batch grows. The column has no intercept either — no fixed startup charge that a larger depth could spread out — which is why the cost line runs straight through the origin and never bends. Doubling the depth doubles the bill, at every depth, forever.

The recall column is the opposite shape, and the way to see it is to read the gaps between rows rather than the levels. Between the first two rows, fifteen extra documents buy fifteen thousandths of recall, so a single extra document is worth a full point. Between 25 and 50, twenty-five documents buy ten points, or one point for every two and a half documents. From 50 to 100, fifty documents buy four points: one point per twelve and a half documents. From 100 to 200 it is one point per forty documents, from 200 to 500 one point per hundred and fifty, and from 500 to 1000 one point per five hundred documents. The yield per document collapses by a factor of five hundred across the table while the price per document does not move at all.

Multiplying those two readings together converts the whole table into a single price. At seven-tenths of a millisecond a document, the first step charges under a millisecond for a point of recall, the 25-to-50 step charges a shade under two milliseconds, the 100-to-200 step charges twenty-eight, and the 200-to-500 step charges a hundred and five. By the last row a point of recall costs three hundred and fifty milliseconds of inference, which is the entire rerank bill for a depth-500 configuration spent on one point. The third column in the table converts the same shape into a rate, and these per-point prices are what that rate is made of.

The prose version of the same arithmetic is short. Between 25 and 50 you buy ten points of recall for 17 ms; between 200 and 500 you buy two points for 210 ms. Those are the same purchase at a price ratio of about a hundred to one — a hundred-odd milliseconds for a point of recall at the deep end of the table against a millisecond or two at the shallow end — and the bend that separates them sits somewhere between 50 and 100 in this corpus.

There is a cheap way to find yours without running the reranker at all, since recall@k only needs the candidate lists and the judgments.

def recall_curve(runs, qrels, depths=(10, 25, 50, 100, 200, 500, 1000)):
    out = {}
    for k in depths:
        hits = []
        for qid, ranked in runs.items():
            relevant = {d for d, g in qrels[qid].items() if g > 0}
            if relevant:
                hits.append(len(relevant & set(ranked[:k])) / len(relevant))
        out[k] = sum(hits) / len(hits)
    prev_k, prev_r = None, None
    for k, r in out.items():
        if prev_k:                       # recall bought per extra candidate
            print(f"{prev_k:>4}->{k:<4} +{r - prev_r:.4f}  "
                  f"({(r - prev_r) / (k - prev_k) * 1000:.2f} per 1000 docs)")
        prev_k, prev_r = k, r
    return out

The division in that last print statement is what turns a raw increment into something comparable across uneven steps. A jump from 200 to 500 covers six times as much ground as a jump from 25 to 50, so their raw increments are not the same kind of number until each is divided by the width of its own step. Retrieval depth is free in the first stage — an ANN index returning 500 neighbours instead of 50 costs a few milliseconds more, and BM25 barely notices — so the whole cost of depth lands on the reranker, which is why the last column of the table is per-pair inference and nothing else.

The other direction to approach k from is the budget, working backwards. If the product's service-level objective for search, the SLO the team has committed to, is 400 ms at p95 — the latency that 95% of requests come in under, so one request in twenty is slower — and the first stage plus fusion plus serialisation already consume 180 ms, then 220 ms is what remains. Dividing that remainder by the per-pair cost of seven-tenths of a millisecond gives a little over three hundred pairs, so at a batch of 64 on a warm endpoint the arithmetic permits roughly 300 candidates before anything else goes wrong.

Two steps in that calculation are decisions rather than facts, and it is worth being able to point at them. Subtracting the first stage's 180 ms is forced: that time is spent whatever the depth. Spending the entire remaining 220 ms on rerank pairs is not forced, and a system that wants headroom for a slow day should spend less of it. That figure is an upper bound rather than a target, and the recall curve usually says to sit well beneath it. Where the two disagree — where the bend sits above what the budget allows — the answer is a cheaper scorer or a shorter passage, since neither more depth nor a better model can be bought with time that does not exist.

Deeper candidates are harder negatives, not neutral ones#

The recall curve says depth is worth less and less. It does not say depth is ever harmful, and the measured behaviour of a real reranker is that it can be. The mechanism runs in three steps. Documents at ranks 51 to 200 were retrieved, so they are topically close to the query. They were not retrieved in the top 50, so the first stage judged them worse than fifty other things it found. A document that is topically close and ranked just outside the cut is, by construction, a passage about the right product, using the right vocabulary, answering a question adjacent to the one asked — which is the definition of a hard negative, the category of wrong answer that is hardest for a scorer to reject.

A concrete instance from the same evaluation: the query was "how long do we keep audit logs for deleted workspaces", the correct passage was a retention table stating 90 days, and at candidate rank 137 sat a passage reading "audit logs for active workspaces are retained for 400 days and are exportable at any time". The cross-encoder scored the rank-137 passage at 6.2 and the correct one at 5.8, and at depth 50 the wrong passage was never in the room.

Read that pair carefully, because the specimen contains the whole failure. The two passages differ in one word — deleted against active — and agree on everything else: the subject is audit logs, the property is a retention period, the format is a number of days. Nearly everything a scorer can use as evidence points the same way for both, and the four-tenths that separates 6.2 from 5.8 is the model's entire opinion about which one the query asked for. A margin that thin flips on noise. Every additional hundred candidates draws more samples from this distribution, and the reranker only has to lose a few of those coin flips per query for precision@10 to move.

Quality peaks at 100 and precision peaks at 50, and the difference between them is 35 ms of inference for one point of nDCG@10 — the cost column rising from 35 to 70, the quality column rising from 0.710 to 0.720. Neither peak is the ceiling from the oracle rerank in chapter 2, where the candidate list is reordered by the human judgments to give the best score any reranker could possibly reach. At depth 100 that oracle scores 0.90, and the reranker scores 0.72; the gap between them is the reranker's error rate, and it widens with depth because the problem it is being handed gets harder.

The oracle figure at that depth is worth pausing on, because it is the same number as recall@100 in the second column, to three decimal places and by no coincidence. Chapter 2 gave the reason: when most queries have a single relevant document, a perfect reranker either has that document in the candidate list and puts it at rank 1, scoring one, or never had it and scores zero, so averaging over queries reproduces the fraction of queries where the document was present. Two instruments — a recall counter that never looks at an ordering, and an nDCG calculation over a perfectly ordered list — land on one value. What that licenses is a shortcut: at any depth where the single-relevant-document assumption roughly holds, the recall column already tells you the ceiling, and the oracle experiment is a confirmation rather than a discovery. It also tells you which lever moves the ceiling, since the only way to raise a number that is counting presence is to put more correct documents into the list.

That widening gap between oracle and achieved is the mechanism behind the general result reported in the study Do Reranking Models Actually Improve RAG?, where the measured gains land well below what depth-based intuition predicts.

Cascades, and why the industry is drifting toward them#

The tension between depth and cost has an escape that is now standard in large systems: stop treating the second stage as a single stage. A cheap scorer runs over 500 candidates, a strong cross-encoder runs over the 40 that survive, and total inference cost is 500 cheap passes plus 40 expensive ones rather than 500 expensive ones. With a distilled 6-layer model at 0.15 ms per pair and a 12-layer model at 0.7 ms, that is 75 ms plus 28 ms against the 350 ms of a single-stage depth-500 run, for a candidate ceiling of recall 0.945.

Both terms in that sum are the same multiplication as before, run at two prices. Five hundred candidates at fifteen-hundredths of a millisecond is seventy-five milliseconds; forty candidates at seven-tenths is twenty-eight; the two together are a little over a hundred milliseconds. The 350 ms they are being compared against is not a separate estimate — it is the depth-500 row of the first table, unchanged. So the cascade reaches the recall ceiling of a depth-500 run for under a third of its inference time, and the saving comes entirely from the fact that the expensive model's multiplier fell from five hundred to forty while the cheap model's price is small enough that its own multiplier of five hundred barely registers.

Cascades are also how a system keeps a high ceiling while holding precision, because the cheap stage's job is to discard the obvious rather than to order the subtle, and the expensive stage never sees the tail where it was going to make mistakes. The cost is another model to train, monitor and version, plus a second set of depth parameters, so the pattern earns its complexity at scale and rarely below it. Two stages are the right default; three is a decision that needs a number behind it.

n is not the same kind of parameter as k#

Cutting the reranked list to n costs nothing at inference time, because all k candidates were already scored — n is a slice taken after the work is done. That makes it a decision about the consumer rather than about the model, and the consumer is almost always a generation step with a context budget and a tolerance for noise.

Passages in this system average 220 tokens. A prompt that reserves 3,000 tokens for retrieved context therefore fits thirteen of them, since dividing the reservation by the average passage length leaves a remainder too small for a fourteenth. At n = 10 the retrieved block is about two thousand two hundred tokens, leaving the best part of a thousand spare, which is the headroom that absorbs a long passage or two without the prompt overflowing.

Pushing n to 20 fills 4,400 tokens, adds latency and cost to every generation call, and hands the model ten more passages whose average relevance is lower than the first ten by construction. The arithmetic there is worth reading twice: 4,400 tokens is more than the 3,000 the prompt reserved, so n = 20 is not a larger slice of the same budget but a decision to enlarge the budget, and it has to be argued as one. Long-context models blunt this without removing it: attention over retrieved context degrades toward the middle of a long block, and irrelevant passages measurably raise the rate at which a generator answers from the wrong source.

Where the bend moves, and what moves it#

Where the bend sits is a property of the corpus and the first stage rather than a constant. A 400,000-document corpus with a strong hybrid retriever bends earlier, around 20 to 30, and a noisy 50-million-document corpus with lexical-only retrieval may not bend until 300. Two systems can therefore read the same table shape and land on depths an order of magnitude apart without either of them being misconfigured, which is why the curve has to be measured rather than inherited from somebody else's blog post.

Passage length interacts with this, and it interacts sharply. Cross-encoders truncate their input to a fixed sequence length, commonly 512 tokens including the query, so a 900-token passage is scored on its first two thirds and a relevant sentence in the final third is invisible to the model. At depth 50 that costs you a few queries; at depth 500, where the correct document is more often a long one that the first stage ranked poorly precisely because its relevant span is buried, truncation and depth compound. Chunking passages to sit comfortably inside the reranker's window is a cheaper fix than either more depth or a longer context reranker, and it improves the first stage at the same time.

A reranker's tolerance for depth is partly a training artefact. Cross-encoders trained on MS MARCO saw negatives sampled from a BM25 top-1000 list, so they have seen hard negatives of a particular flavour; a reranker fine-tuned on your own judged pairs with negatives sampled only from your top-20 will degrade faster at depth than one whose negatives came from the top-500. If you plan to run at depth 200, the negatives in the training set should come from depth 200.

The tempting inference from all of this is that some depth is safe, and that finding it once settles the question. It does not, because three of the inputs move on their own schedule. Re-chunking shifts the truncation exposure, a first-stage change shifts where the bend sits, and a reranker upgrade shifts how well the model survives hard negatives. A depth chosen against one measurement of those three is a depth that was correct on the day it was measured.

The two parameters therefore answer different questions. Retrieve-k is set by where the recall curve bends and what the latency budget permits, and it is the expensive one. Rerank-to-n is set by how many passages the downstream step can actually use, and it is free. Systems that tune them together, as one "top-k" knob, end up choosing a depth to suit a prompt size, which is the wrong constraint driving the wrong parameter.

Setting n to a constant carries an assumption that is easy to miss: that every query has n passages worth showing. The reranker's scores are the only evidence about whether that is true for a given query, and reading them as evidence rather than as an ordering requires knowing what the numbers actually mean.