The retrieval problem

Read these first

• Knowledge cutoff

What you will know

• Write a task-specific relevance rubric and label a representative set of query-document pairs with it

  A useful judgment may require answerability, authority, recency, or coverage rather than topical similarity, and different tasks can produce different labels.

• Compare lexical and embedding-based rankings on the same labeled queries and report where each method wins or fails

  Exact terms, identifiers, negation, spelling variants, and paraphrases expose different blind spots that a single aggregate score can hide.

• Run retrieved passages through a test prompt and trace an incorrect answer to missing evidence, bad ranking, or model misuse

  A fluent answer can conceal retrieval failure, while irrelevant context can also distract a model from evidence that was present.

• Construct a query for which a highly similar passage is irrelevant, then add an evaluation case that catches the failure

  Similarity does not encode the task’s constraints, such as entity, time period, permission, or requested relationship.

• Demonstrate a question the corpus cannot answer and a question requiring synthesis across multiple documents, then record why retrieval alone is insufficient

  Better ranking cannot create absent facts or perform the cross-document reasoning and aggregation the task requires.

Chapters

1. 1

   Writing a relevance rubric

   A 2021 refund policy page scoring cosine 0.91 is a perfect topical match and a wrong answer, because the task carries a recency constraint no similarity score encodes. This chapter names the five dimensions a rubric has to state out loud — answerability, authority, recency, coverage, entity — gives each one a test an annotator can apply, and shows why a three-value scale with a mandatory decided_by field beats a five-point one. Two tasks over the same corpus can legitimately produce opposite labels for the same document.

   Read chapter
2. 2

   Annotator disagreement

   Two labellers agreeing on 169 of 200 pairs looks like a successful exercise until you notice that 27 of the 31 disagreements come from one query class. Majority vote resolves them and destroys the only evidence that the rubric was underspecified; naming the missing criterion and relabelling that class took fifty minutes each and dropped the shipped retriever's measured recall@5 from 0.81 to 0.66. This chapter also sizes the smallest labelled set worth having, at roughly a hundred queries and one afternoon.

   Read chapter
3. 3

   Constraint classes and trap passages

   Embedding similarity measures topical proximity, and topical proximity is blind to the constraints a query carries: which polarity, which entity, which direction of a relationship, which permission scope. A query whose nearest neighbour states the complement of the fact requested — a covered-regions list at 0.89 against an exclusions appendix at 0.71, eleven places below it — is worth more than any number of examples where retrieval merely returns something weak. This chapter builds one deliberately and pins it as a regression case with guard documents.

   Read chapter
4. 4

   BM25, bi-encoders and recall@k

   Term matching scores a document by which of the query's words it contains and how rare those words are; embedding retrieval scores by nearest neighbours in a learned space, at about 10 ms for an exact scan over 12,000 passages and around 4 seconds at 5 million. Recall@k is the other half — the fraction of labelled queries with at least one relevant document anywhere in the top k — and it counts queries rather than positions, which is what makes the next chapter's tie interpretable. Working definitions, in enough detail to produce the numbers yourself.

   Read chapter
5. 5

   Stratifying by query shape

   BM25 and a dense retriever both scoring recall@10 = 0.74 over the same 300 labelled queries failed on 78 queries each and shared only 31 of them, so an oracle picking the better ranker per query would score 0.897. Splitting the set by mechanism — exact identifiers, short keywords, misspellings, paraphrase, multiple entities, negation — turns one uninformative average into six rows, three of which decide something, and one of which shows both systems failing where neither routing nor fusion can help.

   Read chapter
6. 6

   Triaging a wrong answer

   A fluent, well-cited wrong answer is in one of three states: the evidence was never retrieved, it was retrieved and never reached the prompt or was outcompeted by distractors, or it sat in context and the model used it wrongly. A string search over the retrieval dump costs no model calls and settles the first branch; a gold-only ablation settles the rest. On twenty complaint queries the split was 3, 9, 6 and 2 — and the fix everyone proposed first addressed two of the twenty.

   Read chapter
7. 7

   Unanswerable queries and abstention

   Two shapes of question defeat retrieval however good the ranker gets: the fact the corpus does not contain, and the query needing an aggregate over 70 documents when retrieval returns the top five. Neither is a ranking bug, and both belong in the eval set with an empty relevant-document list, a stated expected behaviour and a separate pair of numbers — because the system that wins recall@5 by two points can be the one that answers eleven of twelve unanswerable questions with a confident invention.

   Read chapter