A cross-encoder cannot use in-batch negatives, so it needs a different training set

The bi-encoder training file was 6,200 rows and 9 MB: one query, one passage, one row. Converting the same data for a cross-encoder produced 31,000 rows and 48 MB, because every mined negative becomes its own row with its own copy of the query text, and every passage that appeared once now appears once per query it was mined for. At the 20,000-positive scale the difference is starker — 20,000 rows become 180,000 at eight negatives per positive, and the passage text, which the bi-encoder file stored once, is stored nine times.

That inflation is a symptom rather than an inconvenience. A cross-encoder concatenates query and passage into one sequence and emits a single scalar, which means there is no query vector and no passage vector, and therefore nothing to multiply into a B×B similarity matrix. The construction that made chapter 4's loss work — B examples yielding B×(B−1) free comparisons — has no counterpart here. Every comparison the model is ever going to learn from must be written down, by you, before the first step runs.

The row counts are the same arithmetic twice and it is worth doing once properly, because the general form tells you the size of any file you are about to build. Each positive becomes one row of its own plus one row for each of its negatives, so a set of P positives at n negatives apiece becomes P times one-plus-n rows. Six thousand two hundred at four negatives apiece is the thirty-one thousand quoted; twenty thousand at eight apiece is the hundred and eighty thousand. The file therefore grows linearly in the number of negatives you chose, and every one of them is a labelling decision somebody has to be right about. The byte count grows faster than the row count — from nine megabytes to forty-eight, which is more than the fivefold the rows imply — because the query string is copied onto every row as well, and query text is short but not free.

Every example is a labelled row#

The unit of training becomes a triple: query text, passage text, and a target. With pointwise training the target is a number, usually 1 for relevant and 0 for not, and the loss is binary cross-entropy over the model's single logit.

import json

def build_ce_rows(pairs, mined_negatives, out_path):
    """pairs: [(query, positive_text)]; mined_negatives: {query: [text, ...]}"""
    n_pos = n_neg = 0
    with open(out_path, "w") as f:
        for query, positive in pairs:
            f.write(json.dumps({"query": query, "passage": positive, "label": 1}) + "\n")
            n_pos += 1
            for neg in mined_negatives[query]:
                f.write(json.dumps({"query": query, "passage": neg, "label": 0}) + "\n")
                n_neg += 1
    return {"rows": n_pos + n_neg, "pos": n_pos, "neg": n_neg,
            "ratio": round(n_neg / n_pos, 1)}

Training over those rows is a short script, and the parts that differ from chapter 4 are worth reading closely.

import torch
from datasets import load_dataset
from sentence_transformers.cross_encoder import (CrossEncoder, CrossEncoderTrainer,
                                                 CrossEncoderTrainingArguments)
from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", num_labels=1, max_length=512)
ds = load_dataset("json", data_files="ce_train.jsonl")["train"]

args = CrossEncoderTrainingArguments(
    output_dir="out/ce-derate-v1",
    num_train_epochs=2,
    per_device_train_batch_size=64,     # throughput only; no effect on the objective
    learning_rate=2e-5,
    warmup_ratio=0.1,
    bf16=True,
)

CrossEncoderTrainer(
    model=model,
    args=args,
    train_dataset=ds,
    loss=BinaryCrossEntropyLoss(model, pos_weight=torch.tensor(4.0)),
).train()

per_device_train_batch_size=64 is now doing what batch size does everywhere else in deep learning: filling the GPU. Doubling it changes step count and memory and gradient noise, and it changes nothing about what any example is contrasted against, because the examples are independent. The eight-point recall swing from chapter 4 has no analogue here, and a team that carries the "batch size is the negatives" instinct across from the bi-encoder will spend a day tuning a knob that is not connected to anything.

That instinct is worth naming as the chapter's stumble, because it is a belief about a mechanism rather than a slip. In the bi-encoder, batch size was load bearing for a specific structural reason: the loss built a matrix whose width was the batch, so widening the batch widened the multiple-choice question. Here the loss reads one row, produces one logit and compares it against one label. No quantity in that computation depends on what else is in the batch. So the name of the setting, its units and its position in the configuration file are identical across the two architectures while the thing it governs has changed completely — which is exactly the shape that makes a transferred habit expensive. The check is not whether the knob exists but whether the loss for one example reads any other example.

pos_weight=4.0 compensates for the class imbalance the file's construction created. At four negatives per positive the training set is 80% zeros, and an unweighted binary cross-entropy will happily reduce loss by predicting low scores for everything; weighting the positive class by the negative-to-positive ratio restores the balance. The ratio itself is a decision the bi-encoder never asked you to make. More negatives per positive sharpens the boundary and costs proportionally more training time and labelling risk; between 4 and 8 is the usual range, and going past about 12 mostly buys duplicated passage text.

Why the weight should be the ratio and not some other number is one line of algebra and it closes exactly. Take a file with n negatives for every positive. Summed over the file, the negative rows contribute n units of loss weight per positive row's one, so the two classes enter the objective in a ratio of n to one. Multiplying every positive row's loss by n makes its contribution n and leaves the negatives' at n, and the two are then equal — the n cancels against the imbalance that produced it. Nothing about the corpus enters that argument, so the rule holds at any ratio: set pos_weight to whatever build_ce_rows reported as its ratio field, which is why the function returns it.

Starting from cross-encoder/ms-marco-MiniLM-L-6-v2 rather than from a raw bert-base checkpoint is the other decision embedded in that script, and at this data scale it is close to decisive. The ms-marco checkpoint has already learned what relevance looks like from half a million judged pairs, so your 31,000 rows are teaching it your domain rather than teaching it the task; the raw checkpoint has to learn both from the smaller signal. Trained identically on this data, the ms-marco start reached nDCG@10 of 0.79 and the raw BERT start reached 0.64, below the untrained ms-marco model's 0.72. The threshold where starting from scratch becomes viable is somewhere in the hundreds of thousands of labelled pairs, which is not where a domain adaptation project lives.

Those three figures are a closed comparison and the ordering in them carries the argument. The untrained ms-marco checkpoint at 0.72 is the do-nothing option. Fine-tuning it lands seven points above that. Fine-tuning a raw BERT on the identical rows lands eight points below it, so the raw start does not merely learn more slowly — it finishes behind the model that was never trained on this domain at all. That is what "teaching it the task as well as the domain" costs when the data is a few tens of thousands of rows, and the do-nothing row is what makes the cost legible. Any comparison of two fine-tunes that omits the untrained checkpoint cannot distinguish a small gain from a loss.

The pointwise objective has a known weakness, which is that it asks the model to produce an absolute relevance score when the task at serving time is ordering. Pairwise and listwise losses target the ordering directly: a pairwise objective takes (query, positive, negative) and penalises the model when the negative's score exceeds the positive's by less than a margin; a listwise one takes a query with its whole candidate list and optimises a ranking metric. Sentence Transformers exposes both families for cross-encoders, and listwise training on a query with one positive and eight negatives in a single row is the current default recipe for strong rerankers because it uses graded information the pointwise version throws away. Pointwise remains the right starting point when your labels are binary, which log-derived and generated labels usually are.

The same mining, with the stakes moved#

Chapter 6's mined negatives now function as labels rather than as sampling. That distinction is not academic, and the run that demonstrates it used the rank-window mine whose 19% false-negative rate was measured there. Trained as a bi-encoder, that data produced recall@50 of 0.74, five points below the in-batch-only baseline. Trained as a cross-encoder, the same data produced nDCG@10 of 0.66 against 0.72 for the untouched off-the-shelf reranker, so the fine-tune was worse than doing nothing at all.

The mechanism behind the larger regression is the explicitness of the label. In the bi-encoder, a relevant passage sitting in the batch as a negative competes in a softmax with the true positive; the loss wants the positive to win, and the wrongly-included relevant passage suppresses the gradient rather than reversing it — the model is pushed to prefer one correct answer over another correct answer, which is wasteful and partly self-cancelling across batches where the roles reverse. In the cross-encoder, the row says this pair scores 0, and binary cross-entropy drives the logit for that specific query-passage combination down until it does. There is no competition and nothing to cancel it, and if the same passage appears as a positive for a near-identical query elsewhere in the file, the model receives two flatly contradictory instructions about nearly the same input.

The self-cancellation in the bi-encoder case is worth making concrete, because it is the whole reason the two architectures respond differently to one file. In a softmax the scores are compared within a row, so pushing a wrongly-labelled relevant passage down is only ever a statement about its rank against the positive in that batch. Draw a later batch in which the roles are swapped — the other passage is the positive and this one is a negative — and the loss issues the opposite instruction. Over many batches those two partly annul, leaving a model that is confused about the ordering of two correct answers. The cross-encoder's row asserts an absolute value for one exact pair, and an absolute assertion has no counterpart that could annul it unless the identical pair appears elsewhere with a different label. The difference is not that one loss is more sensitive; it is that one loss is relative and the other is not.

The last two rows are the ones that decide project sequencing, and chapter 8 is about them. The training-time rows are the ones that decide whether you can start today: cross-encoder training needs a completed mining pass as an input, whereas a bi-encoder fine-tune can run on pairs alone and be improved by mining afterwards. If negatives are not mined yet, the bi-encoder path has a working first checkpoint several hours earlier.

There is a second reason the mined negatives matter more here, and it is about distribution rather than correctness. A reranker only ever sees what the first stage retrieves, so its training set should be drawn from that same distribution: mine the negatives with the retriever that will actually be in front of it, at the depth it will actually serve. Training on negatives mined from BM25 and then deploying behind a dense retriever gives the model a candidate distribution at serving time that it never saw during training, and the errors it learned to correct are not the errors it will encounter. When the first stage is itself about to change — because the bi-encoder fine-tune from chapters 4 to 6 is going to ship — the reranker's negatives should be re-mined against the new retriever afterwards, which makes the reranker the second thing to retrain rather than the first.

Evaluating the result follows from the same property. A reranker's score is only meaningful over a fixed candidate list, so the evaluation retrieves once with the frozen first stage, caches the top k, and then compares orderings of that cached list. Reporting nDCG@10 over the reranked top 50 against nDCG@10 of the first stage's own top 50 is the comparison that isolates the reranker's contribution; recall@50 is unchanged by construction and reporting it as evidence of anything is a category error.

That "by construction" is worth unpacking once, because it is the ceiling the next chapter is built on. Recall at depth k asks what fraction of the gold passages appear anywhere in the top k. A reranker receives the top k, permutes it, and returns k items — the same k items. A permutation does not change set membership, so the count of gold passages in the set is identical before and after, and no property of the reranker can alter it. The quantity is therefore fixed at exactly the value the first stage set, whatever the reranker does, which makes it useless as evidence about the reranker and decisive as evidence about the retriever.

The depth also has to match the deployment: a model that gains four points when reranking 200 candidates may gain one when reranking 20, because most of its work was rescuing documents from ranks 50 to 200 that a k of 20 never hands it.

Graded labels are available more often than teams use them, and binary cross-entropy accepts them without modification. A query-chunk pair with 40 clicks from 40 distinct sessions is stronger evidence than one with two, and writing 0.9 instead of 1.0 for the weak pairs, or 0.1 instead of 0.0 for a mined negative the filter was unsure about, gives the loss a way to be less confident where the data is. The effect is modest — about a point of nDCG@10 here — and its main value is preventing the model from becoming extremely confident about labels that were guesses.

Sequence length is the other practical difference, and it is a shared budget rather than two separate limits. The cross-encoder must fit query and passage into one 512-token window, so the space available to the passage is the window minus the query minus the model's separators: a 60-token query leaves the passage a little under four hundred and fifty tokens, and a 400-token passage fits inside that with very little to spare. Lengthen the query, or the passage, and the passage is truncated from the right — which is where the answer often is, since chunk text tends to lead with context. Bi-encoders truncate query and passage independently against the same limit and never compete for it, so the same pair that overflows a cross-encoder is comfortable in a bi-encoder. Checking the truncation rate over the training file takes one pass with the tokeniser, and a rate above a few percent argues for either shorter chunks in the reranking stage or a model with a longer window.

The supervision is exactly what the file contains#

This is not a different way of using the same file. Handing a bi-encoder file directly to a cross-encoder trainer produces a model trained only on positives, which learns to output a high score for everything and reports an excellent training loss while being useless. The conversion in build_ce_rows is mandatory, and the negatives it consumes are the expensive part.

It is not a replacement for the mining discipline either, and the temptation runs the other way: because the cross-encoder sees pairs jointly, it is easy to assume it will simply notice that a mislabelled negative is relevant and ignore the row. It has no mechanism to do so. The model has exactly the supervision the file contains.

That temptation is the second belief-in-a-defence worth naming, and it is tempting for a real reason. A cross-encoder is genuinely better at judging relevance than a bi-encoder, and it is reading the query and the passage together, so it looks like the kind of model that could tell. What it lacks is anywhere for that judgement to go. The loss compares its output against the label; a disagreement between them is precisely what the gradient is computed from, and the gradient's direction is towards the label. So the model's superior judgement about a wrong row is not evidence the training can act on — it is the error signal, and it is used to remove the judgement.

And a trained cross-encoder still cannot retrieve. It has no encode method, produces nothing storable, and scores one pair per forward pass, so it depends on a first stage to hand it candidates — the property established in stage 8.14 and unchanged by anything in this chapter. Training it on your domain raises the quality of the reordering it performs over whatever it is given, and leaves the ceiling set by the retriever exactly where it was.

Both models are now trainable from the same afternoon's data collection, with different files and different objectives, and each addresses a different part of the failure the evaluation in chapter 1 diagnosed. Which one to build first is a question with a real answer, and it turns less on which model is more accurate than on what each one costs to put in front of users.