In production a fused query costs the slower retriever plus the outage risk of both

Every decision in this topic so far has been made offline, on a fixed snapshot, against a judged set. Production changes four things about the answer: what a fused query costs in the tail, what happens when one arm does not answer, what a second index costs to keep, and whether the query mix the evaluation was run on resembles the one users send. None of the four is visible in a recall number, and the first is the one that arrives soonest.

The tail is the maximum, and maxima compound#

A fused query issues both retrievals concurrently and cannot return until the slower one finishes. Its latency is therefore the maximum of two random variables, and maxima behave badly in the tail even when both inputs are well behaved.

The arithmetic behind that is worth doing in full, because it predicts the table rather than merely describing it. The maximum of two values is under some threshold t exactly when both values are under t. If the two arms were independent, the probability of that is the product of the two individual probabilities: P(max ≤ t) = P(a ≤ t) · P(b ≤ t). The product of the two per-arm probabilities therefore has to reach 0.99, and if both arms are similar that means for the fused latency to be under t 99% of the time each arm needs to be under t about 99.5% of the time — because √0.99 ≈ 0.995, and the square root is what undoes a product of two equal factors.

Now read that prediction off the table. The same quantile of the maximum is a further out quantile of each component, so the fused p99 should land on the slower arm's p99.5 rather than its p99. The fused p99 is 96 ms. The dense arm's p99.5 is 96 ms. Two independent instruments — a probability identity and a measured distribution — arrive at the same number, which is the strongest available evidence that the mechanism is the one described and not something else producing a similar-looking table.

The BM25 arm barely participates in that, and the same model says why: its p99.5 is 44 ms, so at the threshold where the dense arm is still deciding, the lexical arm is under it essentially always and contributes a factor of nearly one to the product. The effect is also small at the median, where the fused p50 of 14 ms is barely above the slower arm's 12 ms, because at typical values both arms clear the threshold comfortably and the maximum of two comfortable numbers is close to the larger of them. It grows as you move right, which is exactly the region where service level objectives live — the latency an endpoint promises its callers, stated at a percentile precisely because a mean can sit inside budget while one request in fifty times out.

The arms are not independent in practice, which cuts both ways and is the stumble in this section. Having derived a clean identity, it is tempting to treat the model as a calculator for the fused SLO. It is not one. The arms share a host, a network, a garbage collector and a request queue, so their slow moments correlate, and a correlated stall makes the fused tail closer to the slower arm's own tail than the independence model predicts. Coordinated omission — the measurement artefact where a load generator blocked behind one slow response stops issuing requests, so the slow period produces fewer samples rather than more — in whatever measures your latency will also flatter both numbers. The model is worth carrying because it tells you the sign of the effect and roughly how large; the number to put in the SLO comes from measuring the fused path directly, never from adding or averaging the arms.

The three-arm row is the one that decides whether the marginal 0.019 of nDCG from chapter 6 is purchasable, and the same reasoning scales to it: with three arms each needs to clear the threshold further out still, since a product of three factors reaching 0.99 requires each to exceed what two would. A p99 of 128 ms and a median of 37 ms, up from 14 ms, is not a rounding error in a 250 ms budget, and the median matters more than the tail for a search box people type into repeatedly. On this system the third arm was not shipped, and the reason recorded in the decision log was latency rather than quality.

What happens when one arm stops answering#

Concurrency introduces a question that a single retriever never poses: the dense arm times out, the lexical arm has answered, and something has to decide what the user gets. Leaving that decision to whatever the HTTP client does by default means it gets made by a stack trace.

import asyncio

DEADLINE_MS = 60

async def fused_retrieve(query: str, budget: Budget) -> Result:
    tasks = {
        "lexical": asyncio.create_task(bm25_async(query, budget)),
        "dense": asyncio.create_task(dense_async(query, budget)),
    }
    done, pending = await asyncio.wait(
        tasks.values(), timeout=DEADLINE_MS / 1000
    )
    for task in pending:
        task.cancel()

    lists, missing = {}, []
    for name, task in tasks.items():
        if task in done and not task.exception():
            lists[name] = task.result()
        else:
            missing.append(name)
            RETRIEVER_MISSING.labels(arm=name).inc()

    if not lists:
        raise RetrievalUnavailable(query)
    if missing and classify(query, IDF) == "identifier" and "lexical" in missing:
        raise RetrievalUnavailable(query)      # dense alone scores 0.52 here
    return Result(rrf(lists), degraded=bool(missing), arms=list(lists))

The DEADLINE_MS = 60 sits between the fused p95 of 31 ms and the fused p99 of 96 ms, which is a deliberate placement: a deadline below the p99 means the policy below it runs on a real fraction of traffic rather than being dead code nobody has exercised.

The interesting branch is the class-aware one, and it is the second belief about a defence in this chapter. A blanket "degrade to whatever survived" policy sounds obviously right, and it is a single rule applied identically to every query — which is precisely the problem, because the harm it causes varies by class while the rule does not. Chapter 7 measured what each arm does alone: with the lexical arm missing, identifier queries fall to recall@10 of 0.52, which means roughly half of them return a page of confidently ranked, plausible, wrong documents. A user who gets an error retries or calls support. A user who gets the HX-2140 datasheet instead of the HX-2140R datasheet fills the wrong lubricant into a gearbox. Serving a known-bad result is not automatically safer than serving none, and the per-class table is what lets you make that call with numbers instead of instinct.

Note that the diagram is asymmetric between the two single-arm states, and the asymmetry is the per-class figure showing up in the control flow. Losing the dense arm always degrades to serving, because the lexical arm alone is adequate on every class. Losing the lexical arm degrades to serving only for non-identifier queries.

Every degraded response should carry the flag into logs and into the response envelope, because the alternative is a quality investigation three weeks later that cannot distinguish "fusion is worse than we thought" from "the dense arm was shedding 4% of requests all month". The counter incremented per arm is what makes the second hypothesis checkable in thirty seconds.

The half of the budget nobody prices#

Quality reviews compare recall numbers. The other column has the second index in it.

Two of those cells are computable rather than measured, which is worth checking because it tells you which numbers move when the shape of the system changes. The dense storage figure is 1.4 million chunks at one four-byte float per embedding dimension, and that product is the 4.3 GB in the table — so it scales with chunk count and with dimension and with nothing else. The dense rebuild time is 1.4 million chunks divided by about 1,100 chunks per second, which is a little over twelve hundred seconds, or the 21 minutes printed. Both fall out exactly, which means a plan to double the corpus can be priced without running anything.

The rebuild times look modest until you attach them to the events that trigger them. An embedding model upgrade means re-encoding every chunk and rebuilding the graph, and doing it without downtime means running both versions concurrently for the duration of the backfill and the validation, which doubles the RAM footprint for a day or two. An analyser change means rebuilding the lexical index. Neither index can be rebuilt in isolation without thinking about the other, because a fusion whose two halves reflect different corpus snapshots produces results that are hard to reason about and impossible to reproduce.

Skew between the arms is the routine version of the same problem, and it follows from the two refresh cadences rather than from any defect. The lexical index on this system refreshes in about 40 seconds; the embedding pipeline runs as an hourly batch. An edit that lands just after a batch waits nearly the full hour for the dense side while the lexical side has it in well under a minute, so for up to 58 minutes fusion combines a current lexical view with a stale dense view. That mostly does not matter and occasionally matters a great deal — a corrected safety procedure is live in one arm and superseded content is live in the other, and the fused list can contain both.

Deletions are worse, because a chunk removed from the inverted index immediately can persist in the ANN graph until the next compaction, the periodic rebuild that is what actually reclaims entries marked as deleted, so the fused candidate list will surface documents that no longer exist. The fix is a batched existence check against the source of truth on the fused candidates before returning them, which costs about 2 ms and removes a class of incident that is very unpleasant to explain.

Candidate overlap is the metric that drifts without anyone touching a config. When the ablation ran, the two arms' top-50 lists shared 61% of their members, so a 50 + 50 fusion ranked about 70 unique documents. Seven months and 1.7 million chunks later — the growth chapter 3 recorded, which took the collection to 3.1 million — the overlap is 43%. Run the same deduplication arithmetic on the new figure: 43% of fifty is a little over twenty documents in both lists, so the hundred retrieved rows now resolve to about 79 distinct ones rather than 70.

That drift changes the fusion's behaviour without changing its code. Chapter 2 derived that under RRF a document found by both arms outranks any document found by only one; fewer documents now qualify for that RRF agreement bonus, so more single-list candidates reach the top ten, and the balance described there as "agreement beats confidence" has quietly shifted toward confidence. Overlap belongs on the dashboard next to latency, with an alarm on a 10-point move, because it is the earliest signal that the fusion is behaving differently from the version that was evaluated.

The offline winner is not always the deployed one#

The last correction comes from the traffic itself. The 412-query judged set was assembled by hand, and hand-assembled sets over-sample the queries that are interesting to think about. Classifying a month of production queries with the same rules gives a different mix, and re-weighting the per-class results to it changes the headline:

The two bolded totals are the ones the decision turns on, and both come from multiplying each class's figure by its live share and adding. The dense column weights 0.52, 0.66, 0.79 and 0.80 by fourteen, nine, fifty-seven and twenty hundredths, which comes to 0.74; the hybrid column does the same with 0.83, 0.80, 0.75 and 0.78 and comes to 0.77. Nothing here is a new measurement — the per-class figures are chapter 7's, unchanged — and only the weights moved.

Against live traffic the gap between dense and hybrid is 0.03 rather than 0.07, because the judged set contained nearly twice the proportion of identifier and rare-term queries that users actually send, and those are the classes where fusion's advantage over dense is largest.

The correction cuts both ways, which is the mark of a real measurement rather than a convenient one. The identifier regression that grounded the decision to route is also smaller in absolute terms than the last chapter's weekly estimate implied, since it lands on 14% of searches rather than 23% — scaling the 1,800 by the ratio of the two shares puts it closer to 1,100 affected searches a week. It remains the largest resolvable per-class regression in the ablation and the routing rule remains the right response, but the argument for routing is now made on a number that came from traffic instead of from a judged set that was never sampled to match it.

Three points of recall is still worth having; it is not the same argument as seven, and it has to be weighed against a second index, a 60 ms deadline, a degraded-response policy and an overlap metric to watch. Re-weighting an offline evaluation by the measured production mix costs an afternoon and is the single cheapest thing on this list.

Rolling this out, then, has a shape. Ship behind a flag at 1% with the CI canary set green and the per-class dashboard split before any traffic moves. Watch three things that are not aggregate recall: per-class quality at the depth the product consumes, the degraded-response counters per arm, and the score distributions from chapter 3's replay job. Keep a kill switch that reverts to the single best arm rather than to nothing, since a fusion layer with a bug is more likely than both indexes failing. Re-run the ablation on the current snapshot every quarter, because the analyser, the corpus and the embedding model will all have moved and the weight or the routing rule that was correct in March has no mechanism to notice.

The judgement underneath all of it is the one the topic opened with. Hybrid retrieval is a strong default because the two retrievers fail on different queries, and it is not a universal win because fusion buys its aggregate by giving up ground on the classes where one arm was already right. Every chapter here has been a different instrument for telling those two apart, and each is blind to what the one before it caught: an aggregate cannot see a class, a class figure cannot see a rank demotion inside the top ten, an offline number cannot see the live query mix, and none of them can see a tail latency.

What tells you which of those you have is a pair of queries, a judged set with pre-declared classes, an ablation with a fixed budget, and a production trace — in that order, and none of them optional. Weaviate's documentation on fusion algorithms and Dave Ebbelaar's end-to-end walkthrough of a sparse-plus-dense-plus-reranker stack are both good places to see the machinery assembled against a working index once you have decided what you are measuring.