Learned sparse retrieval predicts weights for terms the document never used

The thermal interlock passage from chapter 1 is unreachable by the lexical arm for one mechanical reason: the postings lists for cold, arm, stop and moving do not contain its document id, because those words are not in the text. Everything downstream of that — scoring, fusion, weighting, reranking — is irrelevant, because a postings walk over four terms cannot visit a document that appears in none of the four lists.

There is an obvious repair, and it follows from stating the problem that way. The document is unreachable because its id is absent from the right postings lists. So put its id in them. Learned sparse retrieval is the family of methods that does this by having a model decide which terms a passage should be indexed under and how heavily, rather than reading those terms off the text. This chapter is about how the weights are produced, why an ordinary inverted index can still serve them, and the specific place the approach is worse than the twenty-year-old baseline it appears to supersede.

The crude version of the repair came first and is worth knowing because it makes the idea concrete. Nogueira and Lin's doc2query, and its stronger successor docT5query in 2019, did it literally: run a sequence-to-sequence model over each passage, generate a handful of questions the passage would answer, append the generated text to the passage, and index the result with an ordinary BM25 pipeline. The generated questions contain the vocabulary real users bring, so the postings lists pick up the document. On MS MARCO — the public retrieval benchmark built from a million real Bing queries with human-marked answer passages, which most work in this area reports against — the technique lifted MRR@10 by roughly 15% over BM25 with no change to the retrieval engine at all. MRR@10 is mean reciprocal rank at ten: for each query take one divided by the position of the first correct passage in the top ten, zero if there is none, and average over the queries.

Learned sparse retrieval is the same idea done without the detour through generated text. Rather than writing words into the document and letting term-frequency arithmetic assign them weights, a model emits the weights directly, over the whole vocabulary, in one pass.

Predicting terms the passage never used#

The mechanism in SPLADE, which is the family most widely deployed, reuses the masked-language-model head that a BERT-style encoder was pretrained with, which is the output layer trained to guess a word that has been blanked out of a sentence. That reuse is the whole trick, and it works because of what the head already computes. For each input position that head produces a logit — an unnormalised score, before anything turns it into a probability — for every one of the roughly 30,500 WordPiece tokens in the vocabulary, WordPiece being the subword scheme these encoders are built on, where common words are single entries and rarer strings arrive as fragments.

That is exactly the shape needed: a score per term per position. Three operations turn it into an index entry, and each has a reason.

Pass the logits through log(1 + ReLU(x)). The ReLU part sets every negative logit to zero, so a term the model does not want contributes nothing at all rather than a small negative — which matters because the vector has to be sparse and a sum of tiny negatives is not sparse. The logarithm compresses what is left, so one very confident position cannot dominate the vector.

Then max-pool over the sequence, so each vocabulary term takes its highest activation anywhere in the passage. Maximum rather than sum is the right choice because the question being asked is whether the passage is about a term at all, and one position wanting it strongly is enough; summing would reward a term weakly implied by many positions over a term strongly implied by one.

The result is a single vector over the vocabulary, one entry per term.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("naver/splade-v3")
mlm = AutoModelForMaskedLM.from_pretrained("naver/splade-v3").eval()

@torch.no_grad()
def encode_sparse(text: str) -> dict[str, float]:
    enc = tok(text, return_tensors="pt", truncation=True, max_length=256)
    logits = mlm(**enc).logits                          # (1, seq_len, |V|)
    activations = torch.log1p(torch.relu(logits))
    mask = enc["attention_mask"].unsqueeze(-1)
    weights = (activations * mask).max(dim=1).values.squeeze(0)   # (|V|,)
    nz = torch.nonzero(weights).squeeze(-1)
    return {tok.convert_ids_to_tokens(int(i)): float(weights[i]) for i in nz}

Left alone, that vector would have thousands of non-zero entries and the index would be useless, so training adds a regulariser — in SPLADE's case a FLOPS term that penalises the expected cost of a postings intersection — which pushes almost all of the weights to exactly zero. What survives on the thermal passage is 168 terms, of which these are the head:

The passage says below 4 °C and the model writes temperature at 1.32 and cold at 0.94. It says joint actuators and the model writes arm at 0.58. It says inhibit motion and the model writes stop at 0.66. Those weights are the answer to the quiz question and worth stating precisely: a model predicts term weights, including terms absent from the text, into a sparse vector. It is not counting anything.

The query goes through the same encoder, producing 28 non-zero terms for why does the arm stop moving when the shop floor gets cold, with cold at 1.19, arm at 1.02, stop at 0.88, temperature at 0.62 and motion at 0.51. Scoring is the dot product over the terms the two vectors share:

cold          1.19 × 0.94 = 1.119
temperature   0.62 × 1.32 = 0.818
motion        0.51 × 1.55 = 0.791
arm           1.02 × 0.58 = 0.592
stop          0.88 × 0.66 = 0.581
ambient       0.33 × 1.48 = 0.488
                    total = 4.389

That column closes exactly, and the closure is what licenses reading it. A dot product runs over every dimension, and a dimension contributes only when both vectors are non-zero there. Six products sum to the whole score with nothing left over, so those six are the entire overlap: every one of the passage's remaining terms met a zero in the query's vector, and every one of the query's remaining terms met a zero in the passage's. Nothing is being summarised.

Now look at which side of the In the text? column those six terms came from. cold, arm and stop are words the user typed and the passage does not contain. motion and ambient are words the passage contains and the user did not type. temperature appears in neither text. Not one of the six is a literal word shared by the query and the passage — which is precisely why BM25 scores this pair 0.000, since BM25 can only accumulate over terms present on both sides and there are none. The 0.000 is not a tuning failure. It is the empty intersection, computed correctly.

Add up where the score came from and the mechanism is unmistakable: cold, temperature, stop and arm are all absent from the passage, and between them they carry a little over seven tenths of the 4.389. Nearly three quarters of this document's relevance to this query is being asserted by the model rather than read off the text.

The scoring is still a postings walk#

What makes this a sparse method rather than a slow dense one is that the dot product over a sparse vector is exactly the operation an inverted index performs. An inverted index scores by walking the query's terms and accumulating each document's contribution, which is a sum of per-term products — the same shape as the column above. Quantise the float weights to small integers — SPLADE deployments typically scale by 100 and round, giving impacts in the range 1 to about 300 — and write them as the impact values in the postings lists. The engine then walks the query's terms, accumulates impacts, and applies the same WAND or block-max pruning it uses for BM25, those being the skipping strategies that let it step over stretches of a postings list whose best possible contribution cannot reach the current top k.

def to_impacts(vec: dict[str, float], scale: int = 100, floor: float = 0.05):
    """Quantise to integer impacts an inverted index can store and sum."""
    return {t: max(1, round(w * scale)) for t, w in vec.items() if w > floor}

Every operational property of the inverted index carries over. Postings are still incrementally updatable, still mergeable across shards, still filterable by metadata before scoring, and a result is still explainable by listing which terms contributed how much — the same debugging affordance that makes BM25 pleasant to operate, now with the expansion terms visible in the explanation. When the thermal query returns that passage you can see that cold contributed 1.119 of the 4.389, which is a far more actionable trace than a cosine of 0.847.

Where the weights come from#

The encoder is not learning to predict term weights from term weights; there is no supervision that says cold should be 0.94 on this passage. Training is contrastive and identical in shape to how dense retrievers are trained: take a query with a known relevant passage, take a set of negatives, and adjust the model so the dot product with the positive exceeds the dot products with the negatives.

The expansion terms are an emergent consequence of that objective rather than a designed feature, and the causal chain is short. Raising the dot product with a positive query means raising the product on some shared dimension. The cheapest shared dimension to raise is one the query already weights heavily. Query vocabulary is user vocabulary. So writing cold into the passage's vector is simply how the model wins on queries that say cold, and the regulariser stops it from writing everything.

Two details of that training determine how well the result transfers to your own documents. Negatives matter enormously: sampling them at random gives a model that learns to separate a robotics manual from a recipe, which is not the discrimination retrieval needs, so modern recipes mine hard negatives from the top of a first-stage retriever's own results. And the strongest variants distil from a cross-encoder, using its scores over query–passage pairs as soft targets rather than the binary relevant/not label, which transfers some of the reranker's judgement into a representation that can be served by an index.

Both of those happen on the training corpus, typically MS MARCO, and the transfer to a domain collection is where the practical risk sits. The vocabulary is fixed at pretraining, so domain jargon that was not in the WordPiece vocabulary is represented only by fragments, and the expansion terms the model chooses are the ones that were useful on web passages about general topics. On a robotics collection the expansions are good for ordinary physical vocabulary — cold, stop, temperature — and noticeably worse on the vocabulary that is specific to the product line, where the model expands harmonic drive toward music and audio about 4% of the time. That failure is visible in the term weights, which is one of the practical arguments for this family: you can read the vector and see it happen.

Where it sits between the two#

Learned sparse is a third retriever with its own profile, and the temptation to file it as "BM25 but better" or "dense but cheaper" loses what is distinctive about it.

On this collection it is the strongest single retriever, which is the common finding — SPLADE variants and ELSER, Elastic's packaging of the same idea as a managed model you do not train yourself, generally beat both BM25 and a mid-sized dense model on standard benchmarks — and it is also the one with the least favourable latency and storage profile.

The 19 GB index is not an accident, and the size follows from the expansion. Each passage now carries around 168 posting entries instead of the 110 or so distinct terms it actually contained, which is about half as many again. But the index is three times the size of the BM25 one, not one and a half times, and the gap between those two ratios is the second cost: the added terms are common words with long, dense postings lists, and every entry now carries an impact value as well as a document id. The traversal grows for the same reason the index does. Read the latency row against it — the sparse arm's 46 ms includes a 14 ms encoder call, leaving a postings walk that costs well over BM25's entire 19 ms query.

The current work in this area is mostly about that cost. Efficiency variants prune the query vector, use the BM25 index to guide traversal, or drop query expansion entirely and expand documents only, which removes the query-time encoder pass at some quality cost. Models like BGE-M3 emit a dense vector, a sparse vector and a multi-vector representation from a single forward pass, which makes the three-arm system cheaper to build because two of the arms share an encoder.

Neither a dense retriever nor a better BM25#

The name invites a specific confusion, so it is worth being blunt: this is not a dense retriever with sparse storage. There is no embedding space, no notion of distance, and no nearest-neighbour structure. The dimensions are vocabulary terms with names, the vector is served by a data structure that predates transformers by four decades, and the scoring is a sum of term impacts. The learned part is only where the weights come from.

It is also not a strict improvement on BM25, and the place it loses is exactly the place chapter 1 said the lexical arm earns its keep. The chain is worth following because it explains why the loss cannot be tuned away. HX-2140R is not in the WordPiece vocabulary, so it splits into hx, ##21, ##40, ##r. Those fragments are shared with every other string that happens to contain them, so the model's weights for them reflect what the fragments mean across the whole pretraining corpus, which is close to nothing. Worse, the FLOPS regulariser is a pressure toward dropping low-value terms, and a subword fragment appearing in one document in half a million looks low-value to a model that has never seen the part.

That is the stumble in this chapter, and it is a belief about a defence: that adding a stronger arm lets you retire a weaker one. The property doing the good work here — representing a string by what its parts usually mean — is the same property doing the damage. A model that generalises a fragment across contexts is a model that cannot treat one fragment as an identifier, so there is no setting of the regulariser or the vocabulary that keeps the bridging and removes the identifier weakness. On this collection, learned sparse gets recall@10 of 0.79 on the identifier query class against BM25's 0.94. It bridges vocabulary well and it handles rare exact tokens worse than the twenty-year-old baseline, which means adding it does not let you delete the lexical arm.

The serving cost is a real change in the shape of the system. The query-side encoder forward pass is on the critical path of every query, so a hybrid that includes a learned sparse arm now has a model inference in its p99 and a GPU in its dependency graph — with all that implies for cold starts, capacity planning and what happens when the inference service is degraded. That is a different operational commitment from BM25, whose failure modes are disk and memory.

Adding it to a fusion is mechanically trivial, because RRF takes another list and weighted fusion takes another normalised score, and neither cares that the third arm's index is inverted. What is not trivial is knowing whether it helped.

There are now five candidate configurations — three single retrievers and at least two ways to fuse them — and the numbers quoted in this chapter came from a comparison whose fairness has been asserted rather than shown. The next chapter builds that comparison properly, and the first thing it fixes is that a fused arm which looked at a hundred candidates is not comparable to a single arm which looked at ten.