Reranking

Read these first

• Hybrid search
• Retrieval evaluation

What you will know

• Add a reranker and measure the quality gain and the latency cost

  Use a labeled retrieval set and production-like traffic so relevance improvement, p95 latency, and added inference spend are measured together.

• Run the same candidate set through a cross-encoder and a bi-encoder, then compare their ranking quality and throughput

  Joint query-document attention captures interactions independent embeddings miss, but its compute grows with the number of candidates scored.

• Choose retrieve-k and rerank-to-n with numbers behind both

  The useful cutoffs depend on recall loss, corpus size, score calibration, and the latency tail rather than top-line relevance alone.

• Show a case where the reranker was not worth it

  Strong first-stage recall, short documents, tight SLOs, or a weakly trained model can leave too little gain to justify another model invocation.

• Trace a reranked request and attribute its p50/p95 latency and per-query cost to candidate retrieval, model inference, batching, and serialization

  Queueing and cold starts can dominate nominal model time, while scoring every query can multiply spend even when average latency looks acceptable.

Chapters

1. 1

   A cross-encoder reads the pair together; a bi-encoder never sees them at once

   The bi-encoder embeds query and document independently, so every document vector can be computed once, months before the query exists. The cross-encoder concatenates them into a single forward pass and attends across both, capturing interactions the separate encodings cannot represent — and forfeiting precomputation entirely. That is why it cannot be first-stage over five million documents: not size, but one pass per document per query. Read the objection carefully and it is about the candidate universe rather than the model, so a cross-encoder can be your first stage whenever that universe is small and bounded by something other than search — a few hundred documents in total, one customer's contract set, a per-user collection, or a slice a hard metadata filter has already cut down.

   Read chapter
2. 2

   The first stage sets a ceiling the reranker cannot lift

   A reranker reorders candidates; it cannot invent one. With recall@50 at 0.62, more than a third of queries have no correct document in the set at all, and the most accurate cross-encoder in the world will rank fifty wrong answers beautifully. Before buying a second model inference, read your candidate recall and decide whether the money belongs in retrieval instead.

   Read chapter
3. 3

   Choosing retrieve-k and rerank-to-n means reading two curves against each other

   Candidate recall climbs with depth and flattens; cost climbs linearly and never does. Going from 50 candidates to 200 quadruples per-query inference and can lower precision@10, because deeper candidates are plausible-looking wrong documents that a reranker sometimes promotes. Find where the recall curve bends, then set n from how many passages your generation step can actually use.

   Read chapter
4. 4

   A reranker score orders candidates without claiming any of them are good

   A cross-encoder emits a learned relevance score on an arbitrary scale, not a probability, so the number that ranks candidate three above candidate four says nothing about whether either belongs in the prompt. That is what makes a fixed n a decision rather than a default: returning exactly n every time pushes irrelevant passages into generation on precisely the queries where nothing was relevant, which is the most common way a reranker adds a failure mode while improving the average. Calibrate the score against judged pairs, set a bar, and let a query that clears it for only two candidates return two — or none.

   Read chapter
5. 5

   Quality gain and latency cost are one measurement, not two

   Run the labelled retrieval set and production-like traffic in the same experiment, and report the relevance delta, p50 and p95 latency, and per-query inference spend side by side. Reported separately, a team celebrates nDCG in a notebook while the tail regression lands in a different dashboard owned by someone else, and nobody ever computes the exchange rate between them.

   Read chapter
6. 6

   When p95 explodes and p50 barely moves, the time is not in the model

   A vendor quoting 30 ms for fifty documents and a p95 that jumps from 210 ms to 1.9 s are both true: nominal inference is not where the tail lives. Trace the request and attribute it to candidate retrieval, serialising the candidates' text over the wire, queueing behind other requests, batch formation waits, and cold starts on a scale-to-zero endpoint. Each has a different fix, and a smaller model addresses none of them.

   Read chapter
7. 7

   Short passages and a high precision@5 leave a reranker nothing to win

   Reranking pays when the first stage returns the right documents in the wrong order. If passages are already short and focused, precision@5 is high, and the latency budget is tight, there is barely any reordering left to do and a weakly trained reranker can make it worse. Write down the case where you measured the gain and declined to ship it — that is the argument that survives contact with enthusiasm.

   Read chapter
8. 8

   An LLM reranker is a different bargain from a trained cross-encoder

   It earns its place with no training pairs, a novel domain, or criteria expressed as instructions rather than relevance labels. You pay in per-query tokens, in latency measured in hundreds of milliseconds, and in non-determinism. The experiment flatters itself in a notebook through position bias in the candidate list and through grading with the same model family that did the ranking, so both effects have to be controlled before the number means anything.

   Read chapter