Equal candidate budgets are what make a fusion ablation honest

An ablation is a comparison that runs a system repeatedly with one component swapped or removed, so that the difference between runs can be attributed to that component. The attribution is the whole value, and it holds only if everything else was held fixed. In retrieval the things that quietly fail to be held fixed are not features but budgets — how much of the index each arm was allowed to look at — and they are easy to miss because they live in client-library defaults rather than in the code under test.

This chapter is about making those budgets explicit, what happens to a headline number when you do, and which other constants have to be pinned alongside them.

What "the same k" hides#

Take a comparison of a production dense-only configuration against a hybrid arm written for an experiment, both asked for ten results and both scored at depth ten. That sounds controlled. Three separate budgets differ between the two rows, and only one of them is fusion.

The first is search effort inside the approximate index. Topic 8.10 established that an ANN structure trades recall against work, and a client library that ties its search-effort parameter to the requested result count will set ef_search=10 when asked for ten documents. That is the extreme end of the trade: on this graph it returns the true top-10 with about 91% accuracy against an exact scan. Nine percent missing is roughly one document in eleven, so about one document in every list that should have been there is not. Raising it to 128 costs 6 ms and takes ANN recall to 99.5%. That single parameter accounts for 3 of the 10 points below, and it has nothing to do with lexical retrieval, fusion or rank aggregation.

The second is how many distinct documents each arm considered. The hybrid arm retrieved 50 from each retriever, and the two lists overlapped on 61% of their members at this corpus size. Work the deduplication through: 61% of fifty is about thirty documents present in both lists, and each of those was counted once in each fifty, so the hundred retrieved rows resolve to about seventy distinct documents. The dense arm ranked 10. A system that inspects seven times as many candidates and then picks the best ten has an advantage that exists whether or not the fusion rule is any good, because the selection is being made over a larger pool.

The third is the evaluation depth relative to what the downstream consumes. On this system retrieval feeds a generation step with the top ten chunks, so recall@10 is the right metric, and both arms are measured at ten. That much was fair. The moment a reranker enters the pipeline, though, the budget that matters is the size of the pool handed to the reranker, and comparing a 10-document pool against a 100-document pool measures pool size.

Equalising the first two changes the picture:

Read the delta column downwards. Ten points, then seven, then two — and each step removes one confound rather than changing anything about the fusion rule, which did not move between rows. The dense arm's own recall climbs 0.68 to 0.71 to 0.79 across the three rows while the hybrid arm climbs 0.78 to 0.78 to 0.81, so almost all of the closing is the baseline being repaired rather than the fused system being degraded.

The last row is the one that should change what the team ships. If the roadmap already contains a reranker, then most of what fusion was going to buy is bought by giving the existing dense arm a deeper candidate pool and a cross-encoder, and the incremental value of building, storing and operating a second index is two points of recall rather than ten. That is still a real gain and it may still be worth having, but it is a different business case, argued against a different cost.

Note what equalising the ANN effort does not do, because this is where the attribution in row two is either earned or not. For a single dense retriever with no downstream stage, retrieving 100 candidates and truncating to 10 gives exactly the same ten documents as retrieving 10 with the same ef_search. Ranking by cosine is a total order, so the best ten of the best hundred are the best ten outright — the extra ninety are all worse than every one of the ten, by construction, and discarding them changes nothing. Since the depth change provably contributes zero, the entire 0.68 to 0.71 movement in row two is the search-effort parameter. It is worth being precise about this, because "just give the baseline a bigger k" is repeated as a fix and it only helps where k is coupled to search effort or where something downstream re-ranks the pool.

That coupling is also the stumble. The phrase that makes the original comparison sound controlled is "both arms asked for the same k", and k does not mean the same thing in the two arms: in one it was silently setting the search effort and in the other it was not. A parameter whose meaning differs between the systems being compared cannot equalise them, however identical its value looks in the config. The only way to know is to name the quantity you intend to hold fixed — documents examined — and check that each arm's configuration produces it.

A harness where the budget is a constant, not a per-arm setting#

The defect in the original comparison is structural: each arm was configured separately, so each arm's author made independent choices about depth and effort. The repair is to make the budget a property of the experiment that arms cannot set.

from dataclasses import dataclass

@dataclass(frozen=True)
class Budget:
    candidates: int = 100      # unique documents any arm may consider
    depth: int = 10            # evaluation depth, and what the consumer sees
    ef_search: int = 128       # ANN effort, identical for every dense call
    index_snapshot: str = "2026-03-04T02:00Z"

def run_arm(arm, queries, budget: Budget):
    for q in queries:
        pools = arm.retrieve(q, budget)            # arm splits the budget
        assert sum(len(p) for p in pools) <= budget.candidates * 2
        fused = arm.fuse(pools)
        unique = dedup_by_content_hash(fused)[: budget.candidates]
        yield q.id, unique[: budget.depth]

The frozen=True is doing more than tidiness: it makes the budget unmodifiable after construction, so an arm cannot quietly raise its own effort mid-run. The assertion is a loose guard rather than the real constraint — it permits twice the budget, on the assumption of two arms, and the multiplier is itself a constant that has to move when a third arm is added. What actually binds is the truncation two lines below, which cuts the deduplicated pool to budget.candidates for every arm alike.

The dedup_by_content_hash step is not decoration. Support corpora contain the same paragraph in a knowledge-base article, a release note and three tickets that quoted it, and a fusion that pulls 50 from each arm can easily spend 12 of its 100 slots on textually identical chunks. Deduplicating before truncation is both the honest thing to measure and the behaviour you want in production, since the generation step gains nothing from three copies of one paragraph and loses the two documents they displaced.

The fixed index snapshot in the budget is the constraint most often violated by accident. Ablations that take two weeks to run frequently span a reindex, an analyser change or an embedding model rollout, and chapter 3's drift table is a measurement of exactly how much a score distribution can move inside such a window. An arm measured after any of those is not comparable to one measured before. Pinning a snapshot identifier and asserting it at the start of every arm costs four lines and removes an entire class of unexplainable result.

The other constants, and the manifest that records them#

Candidate depth and ANN effort are the two that get discussed. Four others quietly differ between arms often enough to be worth naming.

Chunking has to be identical, because a judged set is a set of judgements about chunk ids, and an arm built over 512-token chunks is not scoring the same objects as an arm built over 256-token chunks. This is the most damaging violation available, and the reason is a category difference rather than a matter of degree: the other violations bias a comparison between two measurements of the same thing, while this one means the labels no longer refer to the objects being scored. It happens whenever a dense arm was built by a pipeline that re-chunked to fit the embedding model's context.

Tie-breaking has to be deterministic and the same everywhere. RRF produces exact ties routinely, and chapter 2's weights say why: at k=60 the difference between adjacent ranks is in the fourth decimal place, so two documents at rank 5 and rank 6 in one list and absent from the other differ by 0.0002. If one arm breaks ties by document id and another by insertion order, a measurable slice of the result differences is bookkeeping. Sort by fused score and then by a stable secondary key, in one place, in the harness.

The judged set has to be frozen, including the depth to which judgements were collected. If it was judged to depth 20 against the original dense arm's results, a new arm that surfaces documents nobody ever looked at will have those documents scored as irrelevant by default, which systematically penalises exactly the arm that is finding new material — the arm doing the thing the whole exercise is supposed to reward. Pooled judging — collecting judgements over the union of all arms' top-20 before scoring any of them — is the fix, and it has to happen before the arms are compared, not after one of them loses.

Finally, write the constants down with the results. A run manifest carrying the budget, the snapshot id, the chunker version, the analyser version, the embedding model version and the git sha of the harness turns a table of numbers into something a reviewer can interrogate three months later. Every ablation that cannot be reproduced becomes, eventually, an argument about what somebody remembers running.

The five arms, run properly#

Under one budget, one snapshot, and the same 412 queries scored with paired statistics:

Two independent checks say this table and the previous chapters are describing one system rather than several. The α sweep in chapter 4 was run over the same judged queries, and its endpoints are single arms by definition — α = 0 is pure lexical and α = 1 is pure dense. Those endpoints read 0.541 and 0.598, which are exactly the nDCG figures in the first two rows here. And the cross-encoder measurement in that chapter started from 0.652, which is exactly the RRF row. Neither number was carried over by hand; they agree because the sweep and the ablation are the same experiment observed from two directions, and that agreement is what licenses reading the weighted row as a point on the sweep rather than as a separate result.

Reading the table in order matters more than any single row.

Fusion helps, and it helps by more than the evaluation set's resolution. Chapter 4 worked out that with 412 paired queries and a per-query difference standard deviation around 0.15, the smallest detectable effect is roughly 0.021 of nDCG. RRF over dense is +0.054, which clears that comfortably. The question of whether hybrid is worth anything on this corpus has an answer, and it is yes.

Weighted fusion over RRF is +0.009 in nDCG and +0.01 in recall, which does not clear it. On this data the two fusion rules are indistinguishable, and chapter 3's argument then decides between them: RRF has no state to go stale and no distributions to monitor, so it wins on operational grounds when the quality evidence is a tie. This is what "tune the weighting or justify leaving it at the default" means in practice — the justification is a measured non-difference plus a maintenance cost, written down, and it is a stronger position than a tuned α with no error bar. The order of operations is fixed: establish that fusion helps at all, then ask whether the fusion rule matters, and only then consider tuning a weight inside the rule you chose.

The third arm is the interesting row. Learned sparse alone beats both classical arms, and adding it as a third list gains +0.019 nDCG over two-arm RRF at about 20 ms of additional p95 latency and a GPU dependency on the query path. That sits right at the resolution limit, so the correct statement is that the three-arm configuration is at best marginally better on aggregate quality and is definitely worse on latency and operational surface. Whether it earns its place is a per-class question rather than an aggregate one.

A mean over a mixed workload#

Every number above is a mean over 412 queries, and a mean is a summary that discards the thing you most need to know before a rollout.

Two systems can post identical recall@10 while disagreeing on a third of the individual queries, one winning where the other loses. Two systems can differ by +0.07 on aggregate while one of them has become materially worse on a class that represents 23% of production traffic. The paired statistics above tell you the mean difference is real; they say nothing about whether that difference is uniform or is a large gain on one class net of a loss on another.

There is also a structural reason to expect exactly that on a hybrid system, and it is not a suspicion but a consequence of the rule. Chapter 2 derived that at k=60 a document found by both arms outranks any document found by only one, for every rank either arm can supply. Fusion therefore systematically promotes documents both arms found and demotes documents only one arm found, however emphatically. Any query class where one retriever is reliably right and the other is reliably, confidently wrong is a class where fusion should be expected to underperform the retriever that was right. Identifier queries are precisely that class, and they are the class the lexical arm was built for.

So the ablation is necessary and not sufficient. It establishes that the comparison is fair and that the aggregate difference is larger than the noise, which are the two things nothing else can establish. It cannot say who paid for the average.

Before any of these six rows becomes a default, the same run has to be broken out by query class, and the next chapter does that — with a result that makes the 0.78 in this table look considerably less like a reason to ship.