A hard negative is one the current model already ranks high and is wrong about

Three runs, same 6,200 pairs, same base model, same batch of 128, same three epochs, differing only in what each query was contrasted against. In-batch negatives alone reached recall@50 of 0.79. Adding four mined negatives per query, taken as ranks 10 through 50 from the current model, reached 0.74 — worse than mining nothing. Adding four mined negatives per query that had been filtered by a cross-encoder reached 0.87. The spread between the second and third runs is larger than the spread between using a base model and fine-tuning at all, and the only difference between them is which passages were labelled wrong.

A random negative is a passage about duplicate invoices contrasted against a query about thermal derate. Read that back against the similarity matrix from chapter 4 and the consequence is arithmetic: the off-diagonal entry is 2.2 against a diagonal of 12.4, the softmax already assigns 0.9 to the correct passage, and the gradient contribution is close to nothing. The model separated those two before training started, and it will separate them after. A hard negative is a passage the current model ranks in the top 10 for this query and that does not answer it — the warranty clause that shares every technical term with the thermal derate spec but covers a different product generation. That comparison sits where the model's decision boundary actually is, which is the only place a gradient can move it.

Mining is retrieval against your own labels#

The mechanism is to run the retriever you are trying to improve over your training queries and harvest what it returns.

The two decisions in that diagram are where the three runs diverged. The failing run used a rank window — take ranks 10 to 50, assume the positive is above rank 10 and everything below it is negative — and the assumption is false in a way that is easy to state and easy to miss. Your labels are sparse. A click log records the one passage a user opened, not the seven others that would have answered them; a generated question is attached to the chunk it was written from, not to the four other chunks covering the same procedure. Rank 12 for a query is very often a genuinely correct answer that nobody labelled.

The step being skipped there is the one chapter 2 named and this chapter has to use. A positive pair asserts that a passage answers a query. It does not assert that no other passage does, and nothing in the way pairs are collected could make it assert that — a click log records one choice out of a list, and a generated question records one chunk out of a corpus. So "not labelled positive" and "not relevant" are different statements about a candidate, and a rank window treats them as the same statement. Whether that treatment is safe depends entirely on how densely your corpus answers your queries, which is a property of the corpus that nobody in this pipeline has measured.

Hand-checking 300 negatives mined by rank window from this corpus put 19% of them as relevant to their query under a second reader. That error rate is worse than it looks next to the 4% of suspect in-batch negatives from chapter 5, for two reasons. Explicit negatives are supplied once per query and always present, rather than appearing by chance in one batch out of many, so a wrong one is applied at every epoch. And they are by construction the highest-scoring off-diagonal entries, which is where the softmax concentrates its gradient, so each wrong label carries more weight than an ordinary comparison. Mining amplifies whatever it harvests, which is why it is simultaneously the largest available gain and the most efficient way to make the fine-tune worse.

Both of those reasons are multipliers, and putting them side by side shows why the two error rates are not comparable quantities at all. A suspect in-batch negative is a coincidence of shuffling: this passage happened to share a batch with this query on this epoch, and next epoch it will not. A mined negative is written into the file, so it is present in every batch that row appears in, for every epoch of the run. That is the first multiplier, and it scales with the number of epochs. The second is positional. An in-batch negative lands wherever the encoder happens to score it, usually far from the positive and therefore carrying almost no gradient; a mined negative was selected precisely because it scores close to the positive, which is where the softmax puts most of the gradient it has. So 19% of a quantity that is applied every epoch at maximum weight and 4% of a quantity that is applied occasionally at minimum weight are not four and a bit times apart. The ratio of their effects is much larger than the ratio of their rates, and nothing in either percentage says so.

A margin is the right filter; a rank is a proxy for one#

Rank positions carry no information about how close a candidate came. Rank 12 might score 0.71 against a positive at 0.73, or 0.31 against a positive at 0.88, and those two cases call for opposite treatment. Filtering on the score gap directly says what you mean.

def mine(query, positives, retriever, cross_encoder, k=50,
         floor=0.05, ceiling=0.30, n_neg=4):
    """Return hard negatives whose score sits in a band below the positive."""
    pos_score = max(retriever.score(query, p) for p in positives)
    cands = [c for c in retriever.search(query, k=k)
             if c.chunk_id not in positives
             and c.doc_id not in {p.doc_id for p in positives}]

    kept, ambiguous = [], []
    for c in cands:
        gap = pos_score - c.score
        if gap < floor:
            ambiguous.append(c)          # indistinguishable from the positive
        elif gap <= ceiling:
            kept.append(c)               # the useful band
        # gap > ceiling: too easy, contributes no gradient

    if ambiguous:
        scores = cross_encoder.predict([(query, c.text) for c in ambiguous])
        for c, s in zip(ambiguous, scores):
            if s < 0.0:                  # ms-marco cross-encoders: logit, 0 ≈ boundary
                kept.append(c)
            # otherwise: silently relevant — promote it to a positive instead
    return sorted(kept, key=lambda c: -c.score)[:n_neg]

Why a gap rather than a score is the right quantity to band on follows from the loss the negatives will be fed to. The softmax that consumes them depends only on differences between scores within a row, so a candidate's effect on training is determined by how far below the positive it sits and not by where either of them sits. Two queries can have positives at very different absolute similarities — one query is phrased in the document's own words and one is not — and a fixed score cutoff would band those two queries' candidates by an accident of phrasing. Subtracting the positive's own score removes exactly that accident, which is why pos_score - c.score is computed per query rather than once for the corpus.

The floor exists because a candidate scoring within 0.05 of the labelled positive is, as far as the model can tell, the same thing; labelling it negative is teaching an arbitrary preference between two passages the model has no basis to separate. Those are exactly the candidates a cross-encoder should adjudicate, and adjudicating them is cheap: a run over 6,200 queries with an average of 6 ambiguous candidates each is 37,000 pairs, about 25 seconds on one GPU at 1,500 pairs per second. Sending every candidate to an LLM judge instead would be 310,000 calls, so the sensible arrangement uses the cross-encoder as the bulk filter and reserves a judge for a sampled few hundred to check the cross-encoder's own agreement with a human.

Both of those counts close exactly, and the closure is what makes the design decision obvious rather than a matter of taste. Six thousand two hundred queries with about six ambiguous candidates apiece is the thirty-seven thousand pairs quoted, and thirty-seven thousand divided by fifteen hundred a second is the twenty-five seconds quoted, with nothing left over. The judge's figure closes the same way: six thousand two hundred queries at fifty retrieved candidates each is the three hundred and ten thousand calls. So the two options differ by the ratio between fifty candidates and six, before any difference in price per call is considered. The cross-encoder is not cheaper because it is a smaller model. It is cheaper because the margin band has already thrown away the candidates that needed no adjudication, and the judge would have been asked about all of them.

The ceiling matters less and still earns its line. Candidates far below the positive produce near-zero gradients and displace budget that could hold a useful negative; keeping four negatives per query means those four slots should all be in the band.

The doc_id exclusion carries the lesson of chapter 5 into mining. A different chunk of the very document that contains the labelled positive is the single most likely candidate to be relevant-but-unlabelled, and excluding the whole document costs almost nothing because the corpus has 200,000 others.

How many negatives to keep per query is a smaller decision than it looks, because of how the training framework consumes them. A row of (query, positive, negative) contributes its explicit negative to the batch's passage pool, where it becomes a candidate for every other query in the batch as well. Four negatives per query at batch 128 therefore yields a pool of 640 passages rather than 128, which is most of the benefit of a much larger batch and part of why mining and batch size are partly interchangeable. It also means a mislabelled negative is not a local error: a passage wrongly marked negative for query 7 sits in the pool as a distractor for queries 0 through 127, and if it is genuinely relevant to query 43 as well, the damage propagates.

That pool figure is one multiplication and it generalises without any further measurement. Each row brings its own positive and its own explicit negatives, so a batch of B rows carrying n negatives apiece holds B times one-plus-n passages; a hundred and twenty-eight rows at four negatives apiece is the six hundred and forty quoted. The consequence worth carrying is that explicit negatives widen the pool for every query in the batch and not only for the query they were mined for — which is why they buy some of what a larger batch buys, and equally why a wrong one is not contained to the row it was written on. Between two and eight negatives per query is the normal range, and the number matters much less than the filtering does.

Diversity across mining sources is worth one line of code and buys the same thing diversity always buys. Taking four negatives all from the dense retriever's top 20 gives four passages that fail in the same way; taking two from the dense model, one from BM25 and one from a different embedding model entirely gives the loss four different kinds of confusion to correct. The union is also a cheap check on your own pipeline, because a candidate that BM25 ranks first and the dense model ranks 400th is either a lexical match with no semantic relevance, which is a perfect hard negative, or a genuine answer your retriever cannot see, which is the failure this whole topic is about.

Random explicit negatives sit exactly where in-batch negatives sit, which is the clearest statement of the chapter's premise: a negative that is easy to reject is not training data, it is arithmetic the model performs correctly and learns nothing from. BM25 mining lands between the two and is worth knowing about for a specific reason — it finds candidates the dense model does not rank highly, so it does not inherit the dense model's blind spots, and mining with both retrievers and unioning the results is why the last row edges ahead.

The first two rows landing on the same pair of numbers is worth more than the sentence above gives it. Those are two mechanisms with no step in common. One supplies negatives by accident of batch membership, from a pool the sampler assembled; the other writes an explicit negative column into the file, chosen at random from the corpus. They differ in code path, in file format and in how many comparisons each query gets. They agree to the second decimal on both metrics. Two procedures that share no machinery landing on one number is what licenses reading the difference between them as zero rather than as unmeasured, and it rules out the alternative explanation that explicit negatives help simply by being explicit. What matters is which passages they are.

The rest of the table then separates into two forces, and it separates cleanly because the false-negative column varies independently of the hardness. Compare the random row with the filtered margin-band row: the false-negative rate is near zero in both, and recall moves from 0.79 to 0.87, so eight points is what hardness buys when correctness is held roughly fixed. Now compare the unfiltered margin band with the filtered one: hardness is the same procedure in both, the false-negative rate falls from 12% to 3%, and recall moves from 0.84 to 0.87, so three points is what correctness buys when hardness is held fixed. The rank window is where both forces point the wrong way at once, and it lands five points below doing nothing at all. Recall is therefore not a function of either column alone, which is why the table needs both and why a summary that quotes only the false-negative rate says nothing about what to do.

The s < 0.0 threshold in the filter is a default that has to be calibrated rather than trusted, because a cross-encoder's logit — the raw unbounded score it emits before anything squashes it into a probability — is on a scale set by its own training data rather than by your corpus. Calibrate it once: take 200 ambiguous candidates, have a domain reader label them relevant or not, and plot the filter's decisions against those labels at several thresholds. On this corpus the ms-marco checkpoint's zero point admitted 14% of relevant passages as negatives, and moving the threshold to −1.5 cut that to 3% while discarding a third more candidates — an exchange worth making, since candidates are abundant and wrong labels are not recoverable. The same 200 labels give you the filter's error rate as a number to quote when somebody asks how much of the training set is wrong, which is a better answer than an assurance that a model checked it.

That threshold is this chapter's stumble, and it is a belief that a defence works. Adding a cross-encoder filter feels like the step that makes mining safe, and it is the step that makes mining safe — at a threshold somebody chose. The zero point is tempting because a logit of zero reads like a decision boundary, and for the corpus the cross-encoder was trained on it very nearly is one. It is a constant. Your corpus is not, and neither is the vocabulary, the chunk length or the density with which your documents answer your queries. So the filter's admitted-relevant rate varies across corpora while the number in the source stays where it was, which is the shape to watch for everywhere: a fixed quantity standing in for a property that moves. Two hundred hand labels is what converts the constant back into a measurement, and the same two hundred labels are the only honest answer to how much of the training set is wrong.

The pool goes stale as the model improves#

Mined negatives are defined relative to a model, and after an epoch the model is a different one. Candidates it used to rank at position 8 have moved to position 300, which turns your carefully mined hard negatives back into easy ones and flattens the second half of training. The standard response is to re-mine periodically — after each epoch, or once at the midpoint — refreshing the negative pool against the current checkpoint. This is the idea behind approximate nearest neighbour negative contrastive learning, published as ANCE, where the negative index is rebuilt asynchronously during training. A full re-index is not usually necessary at this scale: re-mining 6,200 queries against a freshly encoded 200,000-chunk corpus is about eight minutes, and doing it once between epoch one and epoch two recovered most of the benefit that continuous refresh would give, moving recall@50 from 0.87 to 0.885 on this data.

The one-shot alternative, when re-mining is inconvenient, is to mine from a stronger model than the one being trained — a large general-purpose encoder, or the cross-encoder itself over a BM25 candidate pool. Those negatives stay hard for longer because the student never catches up to the miner within the run.

Where the difficulty lives#

A hard negative is not a difficult query. The difficulty lives in the pair, so the same passage can be a trivial negative for one query and an excellent one for another, which is why mining is per-query and why a global pool of "confusing documents" does not work.

That is not a stylistic preference and it follows from the margin the miner bands on. The quantity being thresholded is the difference between a candidate's score and the positive's score for one particular query, and both terms of that difference change when the query changes. A passage that sits just below one query's positive can sit far below another's, and there is no property of the passage alone that survives the subtraction. A global pool would have to threshold on the candidate's score by itself, which is the first term without the second.

Mining is not a reranking decision either, though a cross-encoder appears in both places. Here the cross-encoder is a labelling instrument used offline against a few tens of thousands of pairs to decide what goes into a file; at serving time it would be scoring a live candidate list under a latency budget. Using the same model for both is convenient and slightly incestuous, since a bi-encoder trained on labels the cross-encoder produced will inherit that cross-encoder's errors and then be reranked by it, which suppresses exactly the disagreements that would have shown you the problem. Checking a few hundred of the filter's decisions against a human is what keeps that honest.

And mining does not substitute for positives. Four excellent negatives attached to a query whose positive is wrong teaches the model to retrieve the wrong passage more confidently. Everything in this chapter multiplies the quality of the pairs from chapters 2 and 3; it does not add to it.

The training set now has a shape it did not have before: each row is a query, one positive, and four negatives chosen because a particular model found them confusing. That file will train a reranker as well as it trains an encoder, and the cross-encoder that has been doing the filtering is an obvious candidate to train on the same data. It cannot be trained by the recipe of the last three chapters, though, because the machinery that made those chapters work has no equivalent in an architecture that never produces a query embedding at all.