Adapting the retriever to your domain

Read these first

• Retrieval evaluation
• Embedding models
• Reranking

What you will know

• Decide from your own retrieval evaluation whether adaptation is the right lever, or whether hybrid search and reranking still have room

  Adaptation is the expensive answer and it is often not the needed one. If recall@50 is already high and precision@5 is not, a reranker fixes it for a day's work. Adaptation earns its keep when the right document is not in the top 50 at all, which means the encoder cannot see the distinction your domain depends on.

• Assemble a training set of query–passage pairs from something you already have: click logs, support tickets and their resolutions, or questions generated from your own documents

  Hand-labelling is what stops most teams, and it is usually unnecessary. Any log where somebody asked and something answered is a positive pair. Generating a question per chunk with a model is a legitimate cold start and gets within reach of hand-labelled data at a fraction of the cost.

• Fine-tune a base embedding model with a contrastive objective, and explain what the loss does to the vector space

  Multiple-negatives ranking loss pulls a query toward its passage and pushes it away from every other passage in the batch — which is why batch size matters more here than in most training, and why a batch full of near-duplicates teaches the model nothing.

• Mine hard negatives and measure what they buy you against random negatives

  A random negative is trivially separable and the model learns nothing from it. A hard negative — retrieved by the current model, ranked high, and wrong — is where the decision boundary actually is. This is usually the single largest difference between a fine-tune that works and one that does not.

• Train a cross-encoder reranker on the same pairs and compare it against the fine-tuned bi-encoder on cost, latency and quality

  They are not alternatives so much as different points on the same curve: the bi-encoder is precomputed and cheap and has to be right about everything at once, the cross-encoder sees the pair together and is far more accurate over the handful it can afford to score. Most production systems end up wanting both.

• Re-run the stage 8.3 evaluation against the frozen baseline, and decide whether the gain survives the re-embedding cost of your whole corpus

  A new embedding model invalidates every vector you have stored. On a large corpus that is a real migration with real money attached, and a two-point gain on nDCG does not pay for it. Deciding no here is a correct outcome, not a failed project.

Chapters

1. 1

   recall@50 against precision@5 tells you which lever your evidence supports

   With recall@50 at 0.93 and precision@5 at 0.41, the right document is almost always in the candidate set and merely ordered badly — a reranker fixes that in a day. A recall@50 of 0.48 is not yet evidence that the encoder is blind, and reading it that way is the shallow answer pointed in the opposite direction: sample the failing queries and confirm the gold passage is indexed and intact, that chunking did not split the answer away from the language that would retrieve it, and that the judgments are right. Check also whether the misses cluster on part numbers, codes and domain shorthand, because unlearned tokens are an argument for putting BM25 in the mix before they are an argument for moving the vector space. Adaptation is the expensive answer, and it is earned only by failures that survive all of that.

   Read chapter
2. 2

   Your training pairs already exist in logs somebody is currently deleting

   Any record of a person asking and something answering is a query-passage positive: search click logs, support tickets paired with the resolution that closed them, chat transcripts where a document was cited. Each source has a bias worth naming — clicks favour whatever ranked first, ticket resolutions are written in agent vocabulary rather than customer vocabulary — but hand-labelling is what stops most teams, and it is usually unnecessary.

   Read chapter
3. 3

   A generated question is written from the chunk it is supposed to find

   Asking a model for one question per chunk is a legitimate cold start with a specific weakness: the question reuses the chunk's own vocabulary, so it is already retrievable by the model you are trying to improve. You train on the easy cases and evaluate on them too. Mitigate by rewriting questions in user vocabulary, sampling real phrasings, and validating against a small set of genuine queries before spending on a fine-tune.

   Read chapter
4. 4

   Multiple-negatives ranking loss pulls one pair together and pushes the batch apart

   For each query the objective raises similarity with its paired passage and lowers it against every other passage in the same batch. That is what moves the vector space rather than the ranking on top of it. It also explains why batch size carries unusual weight here: the batch is not just a throughput setting, it is the supply of negatives, and a bigger one gives the loss more to push against.

   Read chapter
5. 5

   Batches of consecutive chunks train the model to reject relevant text

   If your loader walks documents in order, a batch contains several chunks of the same source, and the objective dutifully pushes the query away from passages that genuinely answer it. The gradient is confidently wrong, the loss curve looks fine, and the model gets worse at exactly the neighbourhood you care about. Shuffle globally, cap same-document chunks per batch, and check what a sampled batch actually contains.

   Read chapter
6. 6

   A hard negative is one the current model already ranks high and is wrong about

   Random negatives are trivially separable and teach nothing; the decision boundary lives among passages the model retrieves confidently and should not. Mining them is usually the largest single difference between a fine-tune that works and one that does not. But taking ranks 10 to 50 from the current model harvests genuine positives that simply were not the labelled one, so filter mined candidates with a cross-encoder or an LLM judge and use a score margin rather than a rank window.

   Read chapter
7. 7

   A cross-encoder cannot use in-batch negatives, so it needs a different training set

   There is no separate query embedding to contrast against the other passages in the batch, so the recipe you just learned does not transfer. Every example is an explicit (query, passage, label) row, which means each query's own mined negatives have to be written into the training data before a single step runs, and 20k positives become a much larger labelled file. The objective is pointwise or pairwise rather than multiple-negatives ranking, and batch size drops back to being an ordinary throughput knob. Mined-negative quality now determines the labels directly instead of the sampling, so a rank-window mine that swept in genuine positives teaches this model that they are wrong rather than merely diluting the contrast.

   Read chapter
8. 8

   The bi-encoder and the cross-encoder are two points on one curve

   The same 20k pairs feed either model, though not in the same shape. The bi-encoder is precomputed, costs nothing per query, and has to be right about the whole corpus at once; the cross-encoder sees each pair together and is far more accurate over the handful it can afford to score. Most production systems end up wanting both, which makes the question one of order — and the reranker ships without touching a single stored vector.

   Read chapter
9. 9

   A new encoder invalidates every vector you have already stored

   Re-run the stage 8.3 evaluation against the frozen baseline on your own judgments, and read it per query class as well as in aggregate: a domain fine-tune routinely lifts the queries it was trained for while regressing ones that already worked, and the aggregate is the comparison that hides it. Then set the gain beside what shipping it costs: six weeks and forty thousand dollars of re-embedding for three nDCG points is a bad trade, and it stays a bad trade because you championed the project. Deciding no here is a correct outcome. Name in advance what result would have justified it.

   Read chapter