A cross-encoder reads the pair together; a bi-encoder never sees them at once

Every search system has to answer one question about a query-document pair: how well does this document answer this query. There are two architectures for asking it, and their names describe how each one reads such a pair. They differ in exactly one respect — whether the model is allowed to look at both texts at the same time. That single difference decides how accurate the model can be. It also decides, quite separately, how many documents the model can afford to look at. Those two consequences pull in opposite directions, which is why a production search system usually contains both architectures rather than picking one.

A bi-encoder encodes the query into a vector and each document into a vector separately, and scores the pair by the distance between those two vectors. Because the document side never looks at the query, every document vector can be computed in a batch job and stored in an index long before anyone searches. A cross-encoder takes the query and one document as a single joined input, runs them through the network together, and emits one relevance score for that pair; nothing can be prepared in advance, because the model never sees a document without a query attached to it. The arrangement that uses both has a name. Reranking is a second pass over search results: a cheap retriever runs over a whole collection and returns a short candidate list — fifty documents out of several million, say — and a slower, more accurate model then scores each of those candidates against the query and puts the list back in order. The first stage is chosen for being cheap enough to touch every document, the second for being accurate enough to be worth running on what the first one handed over. Search systems use the first to narrow the field and the second to order whatever survives the narrowing, and the arrangement exists because no single model is good at both jobs at once.

The pattern turns up under several labels. In a retrieval-augmented generation (RAG) system it is the step between the vector search and the prompt, cutting fifty retrieved passages down to the ten a generator will actually read. Cohere, Voyage and Jina sell it as a hosted endpoint that accepts a query and a list of document strings and returns a score per document. In an open-source stack it is the CrossEncoder class in sentence-transformers, or a rerank node in a retrieval framework's pipeline configuration, sitting immediately after the retriever.

One vector per document, or one forward pass per pair#

The bi-encoder computes a score as $\mathrm{sim}(f(q), f(d))$, where $f$ is the same encoder applied twice and $\mathrm{sim}$ is usually cosine or a dot product. The important property of that expression is that $f(d)$ contains no reference to $q$. Nothing about the document's representation depends on what anyone will ask, which means it can be computed at index time and stored — and once it is a vector in a fixed space, an ANN index — approximate nearest neighbour, the index structure that finds close vectors without comparing against all of them — can search millions of them without touching the model again.

That property has a name in ordinary programming terms, and the name is worth having, because it is the entire architectural difference stated once. $f(d)$ is loop-invariant. Scoring a query against a collection is a loop over documents, and any expression inside a loop that does not mention the loop variable can be hoisted out of it and computed once. The query is the loop variable here. Since $f(d)$ never mentions $q$, it lifts cleanly out of the per-query loop and into a batch job, and what remains inside the loop is a comparison between two vectors that are already sitting in memory.

The cross-encoder computes $g([\mathrm{CLS}]\ q\ [\mathrm{SEP}]\ d\ [\mathrm{SEP}])$, a single sequence through a single transformer, with a classification head on top emitting one scalar. Every attention layer in that stack can put query tokens and document tokens in the same attention matrix, so the representation of the word "automatically" in the query is computed partly from the phrase "unless a card is on file" in the document. That score does not factor into a query part and a document part. There is no intermediate value you could have stored, because every intermediate value in the network already depends on both inputs.

Read that as the same loop argument running the other way. There is no subexpression to hoist, because there is no subexpression that fails to mention $q$ — attention has made every one of them mention it. So the whole forward pass stays inside the loop, and the work of answering one query over a collection of $n$ documents is $n$ complete forward passes with nothing shared between them. The passes are independent and identical in cost, which is what licenses multiplying rather than doing anything cleverer: total work is the count times the unit cost. Batching many pairs into one GPU call changes the unit cost by some constant factor, and it does not change the shape of the expression.

from sentence_transformers import SentenceTransformer, CrossEncoder

bi = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

query = "does the trial convert to a paid plan automatically"
passages = [
    "At the end of the fourteen-day trial the workspace is downgraded "
    "to the free tier unless a card is on file.",
    "Paid plans are billed automatically on the first of each month.",
]

# Bi-encoder: two independent encodings. doc_vecs could have been
# computed and stored at index time, months before this query existed.
doc_vecs = bi.encode(passages, normalize_embeddings=True)   # (2, 384)
q_vec = bi.encode([query], normalize_embeddings=True)       # (1, 384)
bi_scores = (q_vec @ doc_vecs.T)[0]                         # [0.41, 0.55]

# Cross-encoder: the pair is the input. Nothing here is reusable
# for the next query.
ce_scores = ce.predict([(query, p) for p in passages])      # [7.9, -3.2]

The two passages are the point of that snippet. The second one contains "automatically" and "paid plan" and scores higher under the bi-encoder, at 0.55 against 0.41, because the document vector for the first passage had to compress a whole sentence about trial expiry into 384 dimensions without knowing that someone would eventually care about the conditional. The first passage actually answers the question, and it answers it through a negation — "unless a card is on file" — which is precisely the kind of structure that survives joint attention and does not survive independent summarisation. The cross-encoder puts the two in the opposite order and does it emphatically, at 7.9 against −3.2. Negation, conditionals, numeric qualifiers and questions whose answer depends on which of two entities in the passage is the subject are the standard cases where the two models disagree.

It is worth being precise about why compression is the mechanism rather than model size, because the two are easy to conflate. The bi-encoder is asked to write down everything about that passage that any future query might need, before any query exists, in 384 numbers. The word "unless" only matters when something in the query turns on the condition it introduces, and there is no way for the encoder to know at index time that this is one of the sentences where it will. The cross-encoder faces no such problem, because by the time it runs, the query is sitting in the same attention matrix as the word "unless". It is not storing a better summary. It is not storing a summary at all.

This arrangement is older than the current generation of embedding models and it arrived from the opposite direction. The 2019 paper that put BERT on MS MARCO — a public benchmark of real Bing queries paired with judged passages, and still the training set behind most off-the-shelf rerankers — fed the model a query and a passage as one sequence and read a relevance probability off the [CLS] token, which lifted MRR@10 on the passage task from BM25's 0.19 to around 0.36 and was, at the time, simply how you used a transformer for search. Nobody proposed running it over the eight-million-passage collection. The benchmark handed out a top-1000 candidate list from BM25, and the model's job was defined as reordering that list, so the question of what a first stage would cost never came up. Dense bi-encoders arrived afterwards as the answer to that question, and the two-stage shape everyone now draws is the residue of the sequence: the accurate model existed first, and the cheap retrievable one was built to feed it.

Size is not what makes it undeployable#

The tempting explanation for any latency problem in machine learning is that the model is too big, and it is tempting here for a good reason: it is usually right. Nearly every other serving problem does yield to a smaller checkpoint, a quantised weight matrix or a faster kernel. That instinct fails in both directions here, and the table makes the failure concrete. The relevant axis is not parameter count but when the compute can happen.

Sort those rows by the parameter column and the verdict column scrambles. Two rows sit at exactly 22.7M and land on opposite verdicts, and the largest model in the table, at 7.1B, is one of the Yeses. Sort the same rows by the query-time column and the verdict column sorts with it perfectly — every row under a hundred milliseconds is viable, both rows above it are not, and there is nothing in between. One column predicts the answer completely and the other predicts nothing, which is a stronger statement than "size is not the whole story".

The index-time column explains why, and it does so without needing any of the timings. It reads "Nothing storable" on exactly two rows, and those two rows are exactly the two Nos. Every viable row has something it can compute once and write down: a posting list, one vector, one vector per token. The verdict column is not an extra judgement sitting beside the description; it is the description read out.

The 7-billion-parameter bi-encoder is three hundred times larger than the cross-encoder and is perfectly deployable as a first stage. Embedding 5.2 million passages with it costs somewhere around 30 GPU-hours as a one-time job, and that cost is amortised across every query the system will ever serve. Write the per-query share of it as a fraction: the numerator is fixed at 30 GPU-hours, while the denominator is the query count and only grows. The share therefore falls without limit as traffic arrives, and by a million queries it rounds to nothing against the 45 ms the query itself takes. The 22-million-parameter cross-encoder costs 58 minutes of GPU per query, which is the same fraction with the number of queries in the numerator as well as the denominator. The share does not fall. It is 58 minutes at one query and 58 minutes at a million.

That is the difference the word "big" cannot express. One of these is a fixed cost and the other is a marginal cost, and shrinking a marginal cost leaves it marginal. Halve the cross-encoder's per-pair time and an hour of GPU per search becomes roughly half an hour of GPU per search, which is not a search. What would have to change is which side of the index-time line the work sits on, and that side is chosen by one property of the score function: whether the document half of it mentions the query. Distillation does not change that property. Quantisation does not change it. It is settled by the architecture before any weights exist.

That diagram is the whole architectural argument. The cross-encoder box sits inside the query-time lane and its cost scales with whatever number feeds into it, so the only way to afford it is to make that number small. Retrieve many with something precomputable, rerank few with something that is not: the pipeline described in the Retrieve & Re-Rank page on sbert.net, and the reason it has the shape it does.

The objection is about the candidate universe, not the model#

Read the arithmetic again and it never mentions the cross-encoder's quality, its parameter count, or its architecture. It says $5{,}200{,}000 \times t$ is too large, where $t$ is around 0.7 ms. Change the multiplicand instead of the multiplier and the same model becomes a perfectly reasonable first and only stage.

Contract review is the clean case. A customer's agreement set is 340 clauses. Scoring all 340 against a query costs 227 ms on the same A10G, or about 60 ms if you batch them across a GPU that has the headroom, and there is no retrieval stage in front of it at all — no ANN index to build, no embedding model to keep in sync with the index, no reindex when you upgrade the scorer. You swap the model and the next query uses it. A per-user collection behaves the same way: if the universe a query can legally touch is one person's 900 documents, exhaustive scoring is a design choice rather than a compromise.

That 227 ms will not reproduce if you check it the obvious way, and the reason is worth knowing before you conclude the figure is wrong. Multiply 340 clauses by the 0.7 ms quoted above and the answer comes out noticeably larger. Divide 340 by 1,500 pairs a second instead and it is 227 ms exactly as printed. The two routes disagree because 1,500 pairs a second and 0.7 ms a pair are the same measurement written two ways — one second divided by 1,500 rounds up to 0.7 ms — and multiplying by a rounded-up reciprocal inflates the result. The figure is right, and it was computed from the throughput rather than from the rounding of it. Any figure in this chapter that comes out slightly high when you check it with 0.7 ms is worth re-checking against 1,500 before it is called an error.

The pattern generalises to a hard filter. When a query arrives already scoped by tenant_id = 'acme' AND effective_date >= '2026-01-01' and that predicate leaves 280 rows, those 280 are the collection for this request. What makes these cases work is that the universe is bounded by something other than search — ownership, a tenancy boundary, a date window, a document set someone uploaded — so the bound holds regardless of how good or bad any retriever is. If you bound the universe by running a retriever first, you are back to reranking, which is the rest of this topic.

The bound has to be checked at the tail rather than the median, because the shape of these distributions is rarely friendly. A per-customer contract set that averages 340 clauses will have a customer at the ninety-ninth percentile with 9,000, and 9,000 pairs at 0.7 ms is six seconds of scoring on a request that was budgeted for a quarter of one. The two ends of that sentence are worth lining up. A quarter of a second is very close to the 227 ms the average customer costs, so the budget was, in effect, set by measuring the average tenant; and six seconds is more than twenty times that budget. A latency figure taken on an average tenant cannot warn anybody about this, because it does not move when the dangerous property — how many documents one tenant happens to own — moves, and on a skewed distribution the average is itself well below the tail it is being used to reassure you about. The workable version of exhaustive cross-encoding therefore comes with a cap: score everything up to some ceiling — 500 documents, 1,000, whatever your budget divided by per-pair cost allows — and fall back to a cheap retrieval stage above it. That makes the small-universe case an optimisation for most tenants rather than an architecture, and knowing which one you have is a matter of querying the size distribution before writing any code.

The reader who wants to check this against their own data can measure both properties at once, on identical inputs, in one script.

import time, numpy as np

def compare(queries, candidates_by_query, qrels, bi, ce):
    """candidates_by_query: {qid: [(doc_id, text), ...]} — identical for both."""
    timings, ndcgs = {"bi": 0.0, "ce": 0.0}, {"bi": [], "ce": []}
    for qid, q in queries.items():
        cands = candidates_by_query[qid]
        texts = [t for _, t in cands]

        t0 = time.perf_counter()
        dv = bi.encode(texts, normalize_embeddings=True)
        qv = bi.encode([q], normalize_embeddings=True)
        bi_s = (qv @ dv.T)[0]
        timings["bi"] += time.perf_counter() - t0

        t0 = time.perf_counter()
        ce_s = ce.predict([(q, t) for t in texts], batch_size=64)
        timings["ce"] += time.perf_counter() - t0

        for name, s in (("bi", bi_s), ("ce", ce_s)):
            order = np.argsort(-s)
            ndcgs[name].append(ndcg_at_k([qrels[qid].get(cands[i][0], 0)
                                          for i in order[:10]], k=10))
    n_pairs = sum(len(c) for c in candidates_by_query.values())
    return {name: {"ndcg@10": float(np.mean(ndcgs[name])),
                   "pairs_per_sec": n_pairs / timings[name]} for name in ndcgs}

Running that over 200 queries with 50 candidates each gave 0.63 nDCG@10 at 4,100 pairs per second for the bi-encoder and 0.79 at 1,480 pairs per second for the cross-encoder — a 25% relevance gain for a 2.8× drop in throughput on the same hardware. Both of those ratios are worth reading carefully. The 25% is 0.79 divided by 0.63, so it is a gain relative to the bi-encoder measured inside this same harness, and not relative to the 0.61 that model scores end to end in production; a percentage is only meaningful against a stated denominator, and this one's denominator is 0.63. The 2.8× is 4,100 divided by 1,480. The bi-encoder number here is deliberately unflattering to itself, because it re-encodes documents it would normally have cached; that inflates its measured time and so deflates its throughput, which means 2.8× is a floor on the real throughput gap rather than an estimate of it. The comparison that matters is quality per pair, and throughput at whatever depth you intend to run.

Distillation, late interaction and the limits of reordering#

A cross-encoder is not a bi-encoder with more layers, and the practical consequence is that you cannot extract anything reusable from it. There is no ce.encode(document), and asking for one is asking for the thing the architecture gave up in exchange for joint attention. Teams sometimes try to distil a cross-encoder's scores into a bi-encoder's training objective, which works and is how several strong retrievers were trained. The artefact you deploy afterwards is still a bi-encoder, though, with a bi-encoder's ceiling: at serving time it is once again writing down everything about a passage before it knows the query, so the conditional it could not represent before it was distilled is one it still cannot represent.

The middle ground exists and has a name. Late interaction models such as ColBERT store one vector per token rather than one per document, then compute a MaxSim between query tokens and document tokens at search time. That recovers some token-level interaction while keeping the document side precomputable, at the cost of roughly forty times the index storage — the row in the table above is not hypothetical, and the storage figure is why it stays a minority choice. It sits where it does in the table for a reason that follows the same rule as everything else here: a per-token vector is still something computable without a query, so it is still hoistable, so ColBERT is still a Yes.

The exhaustive-scoring cases have a boundary that is easy to state and easy to cross by accident. A universe bounded by ownership or a date predicate stays bounded no matter what the search quality is. A universe bounded by "the first stage returned 500 of them" is not bounded by anything except the first stage's behaviour, and behaviour changes. The two look identical in a request trace and they fail differently: the first is an architecture, the second is a reranking pipeline whose depth nobody wrote down.

A cross-encoder also does not find anything. It takes a list and returns the same list in a different order, which is a property so obvious that it is easy to state and surprisingly easy to forget when a quality problem is on the table. Whatever the first stage failed to retrieve is not in the list, so no reordering can put it in front of the user, and the eighteen-point nDCG gap above was measured on candidate sets that already contained the right answers.

Everything in this chapter comes back to one property of a scoring function: whether the document half of it can be computed without the query. When it can, the expensive part of the work leaves the per-query loop and becomes a fixed cost that dilutes with traffic, and a seven-billion-parameter model is affordable. When it cannot, the expensive part stays inside the loop and becomes a marginal cost that traffic multiplies, and a twenty-two-million-parameter model is not. Parameter counts, benchmark scores and vendor comparisons all sit downstream of that one question, which is why the largest model in the table is viable and the smallest is not.

The accuracy the cross-encoder buys by reading the pair together is real, and the arrangement that makes it affordable is to run it over a short list. That short list is produced by a different model with different failure modes, and nothing in this chapter has said what happens when it hands over fifty documents that do not contain the answer. What that gap looks like when they do not is the next thing to work out.