An LLM reranker is a different bargain from a trained cross-encoder

An LLM reranker is a general-purpose language model asked, in a prompt, to put a list of candidate passages in order for a query. It sits in the same slot as a trained cross-encoder — after a cheap first stage has produced a shortlist, before whatever consumes the ranked results — and it differs in where its notion of relevance comes from. A cross-encoder's notion is in its weights, put there by thousands of labelled query-passage pairs. A language model's notion arrives in the prompt, as English, at request time.

That difference decides when each one is the right instrument, and the deciding case looks like a requirement arriving as a sentence from a domain expert: "rank the passage that states the rule above one that merely cites it, and prefer the more recent amendment when both state it." A cross-encoder needs labelled pairs to fine-tune and a ms-marco model has never seen a statutory citation, and neither of those problems is solved by a sprint. The requirement is also not a relevance label in any conventional sense — it is a preference over document roles that no ranking dataset encodes. A general-purpose language model handles that sentence directly, because the sentence is the input format, and it comes with a price list a trained cross-encoder does not have.

Scoring one candidate at a time, or a window of them together#

Pointwise scoring asks the model, once per candidate, how relevant a passage is on some scale. It is the simplest to implement and the most expensive, since fifty candidates mean fifty calls, and it produces scores that are poorly separated because the model has nothing to compare against — a great many passages come back as "7".

Listwise ranking hands the model a numbered block of candidates and asks for a permutation. One call covers a window of candidates, the model sees them together, and the comparisons that make ranking work happen inside the context rather than across independent calls. Practical windows run 10 to 25 passages depending on passage length, with a sliding window and overlap when the candidate list is deeper than one window fits.

import anthropic, json, re

client = anthropic.Anthropic()

INSTRUCTIONS = """Rank the passages by how well they answer the query.
Domain rules, in priority order:
1. A passage that states a rule outranks one that only cites or summarises it.
2. When two passages state the same rule, the later amendment outranks the earlier.
3. A passage that is merely on-topic outranks nothing; place it below all of the above."""

def rank_window(query, window, model="claude-haiku-4-5"):
    """window: [{'id': str, 'text': str}, ...] — 20 passages, ~220 tokens each."""
    listing = "\n\n".join(f"[{i}] {p['text']}" for i, p in enumerate(window))
    msg = client.messages.create(
        model=model, max_tokens=300, temperature=0,
        system=INSTRUCTIONS,
        messages=[{"role": "user", "content":
                   f"Query: {query}\n\nPassages:\n{listing}\n\n"
                   f"Return only a JSON array of indices, best first."}],
    )
    order = json.loads(re.search(r"\[.*\]", msg.content[0].text, re.S).group())
    seen, out = set(), []
    for i in order:                      # drop hallucinated / duplicate indices
        if isinstance(i, int) and 0 <= i < len(window) and i not in seen:
            seen.add(i); out.append(window[i])
    for p in window:                     # append anything the model omitted
        if p not in out:
            out.append(p)
    return out

The defensive block at the end is not decoration. Across 180 evaluation queries this parser recovered from a truncated array 3 times, a duplicated index 11 times, and an index outside the window twice, which is sixteen malformed responses in total. Spread across 180 queries that is a failure on roughly one query in twelve, and a reranker that raises on any of those is a reranker that returns a 500 on 8% of queries. The reason this happens at all is structural rather than a quality problem with any particular model: the output contract is a permutation, and nothing in a decoder guarantees that a sampled sequence of indices is a permutation of anything. Every LLM reranker in production has some version of that loop, and the fallback order — original first-stage order for anything the model did not place — is the safe default because it degrades to the arm you already had.

What it costs, stated as a bill#

The arithmetic is worth doing one step at a time, because each step is a knob somebody chose. Fifty candidates in windows of 20 with a stride of 10 means windows beginning at candidate zero, ten, twenty, thirty and forty, which is five calls. Each call carries 20 passages at 220 tokens each, so about 4,400 input tokens, plus the instruction block, which is sent again on every call. Five calls of that size come to roughly 23,000 input tokens per query, and the outputs are short JSON arrays adding to about 1,000 output tokens.

Most candidates appear in two of those windows, which is why the input figure is so much larger than the text being ranked. The overlap is not waste — it is what lets a sliding window carry a passage from the bottom of a deep list to the top, since a passage can only be promoted past candidates it has been shown alongside — but it is paid for in tokens, and the stride is the dial that sets the price.

At $0.80 per million input and $4.00 per million output for a small hosted model, 23,000 input tokens is a little over eighteen thousandths of a dollar and 1,000 output tokens is four thousandths. Those add to $0.0224 per query, or $22.40 per thousand queries. The split inside that sum is the part worth remembering: the input side is more than four fifths of the bill, so the only lever that moves the total is the quantity of passage text put in front of the model — shorter passages, a narrower window, a longer stride. Asking the model for a terser answer economises on the smaller fifth.

The cost column spans a factor of fifty and the latency column a factor of seven, which is why the LLM reranker is rarely the answer for a high-volume search box and often the answer for a low-volume, high-value query. The latency row is tidier than it looks: the three bands are scalar multiples of each other, with the hosted API at one and a half times the self-hosted cross-encoder at both ends of its range and the LLM at seven times it at both ends. Because the multiple is the same at each end, a latency budget can carry one ratio rather than a worst case, and wherever inside its band a particular call lands the comparison against the alternative is unchanged.

Non-determinism is the row that surprises people who set temperature=0, the sampling setting that makes the model take its highest-probability token at every step instead of drawing at random among the plausible ones. Repeating the same 180 queries three times through the same model and prompt changed at least one position in the top 10 on 8% of queries, mostly swaps between adjacent near-ties. That is small enough to be irrelevant to a user and large enough to break a golden-file test, to make an A/B comparison noisier than its effect size, and to make a bug report unreproducible. Caching the ranking by (query, candidate-set) hash removes it for repeated queries and is worth doing for that reason alone.

Two separate 8% figures now sit a few paragraphs apart and they are not the same population. One is the share of queries whose response could not be parsed as a clean permutation; the other is the share whose parsed permutation differed between runs. A query can land in either, both or neither, and adding them together to get a failure rate would be counting two unrelated defects as one. The first is fixed by the fallback loop; the second is fixed by a cache, and nothing fixes both.

The operations row hides a latency story that the tracing chapter predicts. Five sequential window calls at 200 ms each is a second before anything else happens, and running the windows concurrently only works when the sliding window has no overlap dependency, which the standard stride-10 arrangement does. Rate limits then supply a queue you do not control: a 429, the status code an API returns when you have exceeded the requests it will accept in a window, with a retry adds a full call's latency to the request, and at the busy hour the retry rate is exactly when it is highest, so the p99 of an LLM reranker is governed by someone else's admission control. The mitigations are the familiar ones — a concurrency cap below the account's limit, a short retry budget with a hard deadline, and a fallback that returns first-stage order when the deadline passes — and the last of those is the one that keeps the feature available rather than fast.

Two ways the notebook lies#

Position bias is the first, and it is strong enough to fake an entire result. Language models attend unevenly across a long list, and the passages at the start and end of the block are systematically favoured over those in the middle. The measurement is easy: rank a window, then rank the same window with the candidates shuffled, and compare. On this corpus, with a 20-passage window, the passage that ended up at output rank 1 was in the first five input positions 47% of the time, against the 25% you would expect if input position carried no information.

That comparison figure is not estimated from data. Five positions out of a window of 20 is a quarter of the window, so a model indifferent to input order would draw its winner from those five a quarter of the time by counting alone. The observed share is close to double it, which is a large effect measured against a baseline that cannot itself be wrong.

That interacts badly with a detail of how experiments get set up. Candidates are normally fed to the reranker in first-stage order, so the first-stage top 5 occupy the input positions the model favours, and the model's output correlates with the first stage for reasons that have nothing to do with relevance. If the first stage is decent, the reranker inherits its quality and appears to be doing well. The control is to shuffle the candidate order before every call, and either to accept the noise or to average two or three permutations per query, which multiplies the cost by two or three and is the honest way to run the experiment.

Grading is the second. An evaluation that asks a model to judge whether the reranked results are relevant, using the same model family that produced the ranking, measures agreement rather than correctness. Models prefer their own outputs and their own notions of what a good passage looks like, and the effect is large enough to reverse an ordering between two arms. Human judgments on a few hundred pairs are the sound answer; a judge from an unrelated model family is the practical compromise; the same family on both sides is a number that should not leave the notebook.

Same 180 queries, three evaluation protocols, same two arms.

                              cross-encoder    LLM listwise
same-family LLM judge             0.71             0.86
different-family LLM judge        0.72             0.79
human judgments (n=900 pairs)     0.74             0.76

Read that block as a column of gaps rather than six scores and it says something sharper than "both controls help". The gap between the arms is fifteen hundredths on the top row, seven on the middle one and two at the bottom, so it narrows by thirteen hundredths overall. Ten of those thirteen come from the LLM column falling, from 0.86 to 0.76, and three come from the cross-encoder column rising, from 0.71 to 0.74. Ten and three account for the thirteen with nothing left over, which means the distortion is almost entirely a preference for the arm that shares the judge's model family rather than a general miscalibration hurting both arms equally. It also prices the practical compromise: moving to a different-family judge recovers eight of those thirteen hundredths, about three fifths of the whole correction, without labelling a single pair by hand.

Both controls narrow the gap, and after both are applied the LLM reranker on this domain was ahead by 0.02 of nDCG@10 at fifty times the cost — for the general relevance criterion, and that 0.02 is simply the bottom row read across. On the subset of queries where the domain rules applied, states-versus-cites and amendment recency, it was ahead by 0.19, and no cross-encoder available off the shelf expressed those rules at all. The instrument earned its place on the part of the problem it was chosen for and not on the part where a cheaper model was already competent.

Distilling the orderings into a cross-encoder#

The interesting property of an LLM reranker is that it produces exactly the artefact a cross-encoder needs. Run it offline over a few thousand queries with their candidate lists, keep the resulting orderings as soft labels — training targets written by a model rather than by a human annotator — and train a cross-encoder to reproduce them using the sentence-transformers reranker training scripts, which is the procedure usually called distillation. The result is a model that has absorbed the domain rules into its weights, costs $0.42 per thousand queries, adds 100 ms rather than 1,100 ms, and is deterministic.

The economics of that flip on one observation. The distillation run in this system cost about $180 of inference over 8,000 queries and three hours of training, which is the same $22.40 per thousand that made serving unaffordable, multiplied by a query count that is a one-off rather than a run rate. The unit price does not fall at all; what collapses is the number of times it is paid. A price of $0.0224 per query is ruinous against a search box answering millions and immaterial against a labelling job that runs once, and the distilled model recovered roughly 80% of the LLM's advantage on the rule-sensitive queries.

Criteria the passage pair does not contain#

That pattern — LLM as label generator, cross-encoder as the thing you serve — is where most production systems that started with an LLM reranker end up, and knowing it exists changes how the first experiment should be framed. The question to ask when the LLM arm wins is not whether to serve it but whether its advantage is distillable, and the answer depends on whether the criterion is expressible from the passage pair alone. States-versus-cites is; "prefer the version our legal team has approved" is not, because the approval is a fact about the world that no amount of reading the passage recovers. A cross-encoder sees a query and a passage and nothing else, so a criterion that needs a third input can be described to a language model in a prompt and still cannot be trained into weights.

Distillation also freezes the rules at the moment it runs, and that is a real cost rather than a footnote. While the criterion lives in a prompt, changing it is an edit to a string. Once it lives in a cross-encoder's weights, changing it is another labelling run and another training run, and the served model goes on applying the old rule until that finishes. A domain whose rules are settled is a good candidate for distillation; one where the expert is still refining what they meant is better served by paying the token bill until the sentence stops changing.

The practical depth row in the table has the same origin and is worth reading as a consequence rather than a fact. A cross-encoder scores candidates independently, so a deeper list is more forward passes that batch together on one GPU, and depth past 500 costs throughput and nothing else. A listwise LLM reranker turns every candidate into prompt tokens inside a sequence of calls that mostly cannot run in parallel, so depth raises the bill and the latency together. Depth is where the two instruments diverge most sharply, and a team that has decided to rerank deeply has already narrowed the choice.

Everything across this topic collapses into a short sequence when a new system arrives. Measure candidate recall and run the oracle rerank, because the headroom number decides whether any of the rest is worth doing. Sweep depth until the recall curve bends and read precision alongside it. Calibrate the score, put a bar on it, and let queries return fewer passages than n. Run quality, latency and spend as one report against replayed traffic. Trace the request when the tail misbehaves, and expect the answer to be queueing rather than the model. Write down the case where the gain was too small to ship, with the conditions that would reopen it. The reranker itself is a component you can swap in an afternoon; the numbers that tell you whether it belongs there are the part that takes a week and the part that survives the next model release.