The second sub-query cannot run until the first has answered

Decomposition splits a question that is really two questions into parts that can be retrieved separately, and joins the results afterwards. It is the repair for the compound defect of chapter 1: one query produces one vector, one vector cannot represent two intents, and the intent occupying more of the string wins. Splitting removes that competition by giving each intent a query of its own.

What splitting does not remove is the relationship between the parts. A sub-query is one of the produced parts; two sub-queries are independent when neither needs anything the other produces, and dependent when one of them contains, or should contain, a value the other has to supply first. The distinction is not about execution strategy. It is a property of the question, decided before any code runs, and getting it wrong produces an answer that is confident, well-cited and about the wrong set of things.

Dependency is a property of the split, not of the executor#

The question is which of our EU enterprise accounts churned after the March price change, and what did their exit interviews say? and the decomposer handles it exactly as intended.

{"sub_queries": [
  "EU enterprise accounts that churned after the March price change",
  "exit interview notes from EU enterprise accounts"
]}

Both fire concurrently, both return top-10, the twenty chunks go to the generator. The first leg is right: seven accounts, the churn tracker rows for Northwind Logistics, Halden Rail, Berger Pharma and four others, all with termination dates between April and June. The second leg returns ten exit interviews, of which one is from an account on that list. The other nine are from accounts that renewed, downgraded, or churned in 2024 for unrelated reasons, and the answer the generator wrote attributes their complaints to the price change.

The second retrieval did nothing wrong. It was asked for exit interview notes from EU enterprise accounts and a corpus of 340 of them was available; it returned the ten that most resemble exit interview notes from EU enterprise accounts. The constraint that would have selected the right ten — from these seven accounts — did not exist yet when the query ran.

Nothing errors, and the reason is worth stating as a chain rather than as an observation. A dependent sub-query written correctly is a template with a slot in it, and a template with an unfilled slot is visibly broken — a query reading exit interview notes for announces itself. The decomposer above did not produce a template. It produced a second question that is complete, grammatical and answerable, and which a human reader would accept as a reasonable rephrasing of the second half. So there is no empty slot for anything to notice, the retrieval call is well-formed, the results are genuinely relevant to what was asked, and the only place the dependency existed was in the original question, which nothing downstream still has. The failure is invisible at every layer because every layer is behaving correctly on its own input.

Some decompositions genuinely are independent. what is our EU data residency policy and what is our UK data residency policy splits into two queries that share no information; either can run first, and running them together is correct. The churn question splits into two queries where the second one's retrievable form contains a value the first one produces, which makes the parallel execution not a performance choice but a semantic error.

The planner has to say which it is, which means the output format needs more than a list.

{
  "steps": [
    {
      "id": "q1",
      "query": "EU enterprise accounts that churned after the March price change",
      "depends_on": [],
      "extract": "account_names"
    },
    {
      "id": "q2",
      "query": "exit interview notes for {{q1.account_names}}",
      "depends_on": ["q1"],
      "fan_out": "per_value",
      "max_fan_out": 8
    }
  ],
  "merge": "join_on:account_names"
}

Three fields carry the weight. depends_on makes the ordering explicit rather than implied by list position. extract names what the first step must yield for the template slot to be fillable, so the executor can fail loudly when the churn query returns prose with no extractable account names instead of quietly rendering exit interview notes for . And fan_out decides whether the dependent step is one query naming seven accounts or seven queries naming one each.

The requirement running through those three fields is a single one: a dependent query must not execute until the value it needs exists, and must fail visibly when that value cannot be produced. Every field is a design decision serving it, and each could have been made differently. Ordering could have been inferred from list position rather than declared, which is cheaper and silently wrong the first time a planner emits the steps in a different order. The extraction target could have been left to the executor to guess from the query text. The fan-out cap could have been a token budget rather than a count. Separating the one requirement from the four choices is what makes it possible to argue about the choices without reopening the requirement.

The fan-out choice is not cosmetic. One query containing seven account names produces a vector that is a blur of seven entities plus the phrase "exit interview", and it retrieves the interviews for whichever two or three accounts have the strongest lexical presence in the corpus. Seven separate queries produce seven tight retrievals, one per account, at a cost of seven ANN calls that run concurrently in about 45 ms total. The cap exists because the first leg sometimes returns forty accounts rather than seven, and forty dependent retrievals plus forty times the context is a query that costs more than the answer is worth. When the cap binds, the honest behaviour is to answer for the top eight and say so, rather than to silently answer for eight of forty as though that were the whole set.

def execute(plan: Plan, index) -> dict[str, list[Chunk]]:
    results: dict[str, list[Chunk]] = {}
    values: dict[str, list[str]] = {}

    for step in toposort(plan.steps):
        if step.depends_on and any(d not in values for d in step.depends_on):
            raise PlanError(f"{step.id} needs {step.depends_on}")

        if step.fan_out == "per_value":
            src = values[step.depends_on[0]][: step.max_fan_out]
            queries = [step.query.replace(SLOT, v) for v in src]
        else:
            queries = [render(step.query, values)]

        hits = concurrent_search(queries, index, k=5)
        results[step.id] = hits
        if step.extract:
            values[step.id] = extract_values(step.extract, hits)
    return results

The extract_values call is the step that decomposition implementations most often skip, and it is where the design stops being a prompt trick. Something has to read the churn tracker chunks and produce the list ["Northwind Logistics", "Halden Rail", ...], which is a small generation call over the first leg's results with a structured output schema. That call is the reason the two legs cannot overlap in time: the dependent query is not merely ordered after the first, it is constructed from the first leg's answer, so the sequential cost is a retrieval plus an extraction plus a second retrieval.

Read that table as two gaps rather than three rows and it says something the rows alone do not. The first column improves by about the same amount twice: supplying the constraint at all lifts the count by three, and un-blurring the constraint into one query per account lifts it by three again. The two errors are therefore comparable in size, which means fixing only the ordering leaves roughly half the damage in place — and that is the version most implementations ship, because ordering is the part that feels like the bug.

The latency column tells the opposite story about where the cost sits. Almost all of the added time appears between the first row and the second, where the number of retrieval calls did not change at all. Going from two calls to eight adds a small fraction of what serialising two calls added. So the price is not the fan-out width, which runs concurrently; it is the round trip through extraction that has to complete before the second query can be written. The call-count column confirms it arithmetically — the eight calls are one root query plus one per account, with nothing else in them — which is why widening the fan-out is the cheap axis and shortening the chain is the expensive one.

Going from 120 ms to 1.3 s is the price of correctness here, and there is no version of this that is fast, because the dependency is in the question. Latency budgets that assume every retrieval is parallel are budgets that only hold for questions that decompose into independent parts.

Merging is not concatenating#

Twenty chunks in one context with no structure lets the generator do the join, and the join is where accounts get attached to the wrong complaints. Three merge strategies, in increasing order of how much they protect:

Concatenating everything is the default and it loses the association between chunk and sub-query. The model sees churn rows and interview notes interleaved by relevance score and infers the pairing from names it can match, which works when names are unambiguous and fails when two accounts share a parent company or one interview mentions a peer.

Answering each sub-query separately and synthesising over the sub-answers keeps the association and costs one extra generation call per step. The churn leg produces a list, each fan-out leg produces a two-sentence summary of one interview, and the final call assembles them. This is where most implementations should land, and its weakness is that a sub-answer is lossy — if the synthesis needs a detail the summariser dropped, it is gone.

A structured join keyed on the entity keeps everything: the executor builds a mapping from account name to that account's retrieved chunks, and the final prompt is laid out per account rather than per relevance rank. It is the most code and the most robust, and it only applies when the dependency really is an entity list rather than a free-form fact.

The merge also has to carry provenance, and this is where decomposition changes what a citation means. In a single-query pipeline every retrieved chunk is evidence for the one question that was asked, so a citation points at a passage and that is the whole story. After a dependent decomposition, a chunk is evidence for one step, and the answer to the original question is an inference over several steps. If the churn tracker is stale and Berger Pharma did not actually churn, the exit interview cited under Berger Pharma is a real document quoted accurately in support of a claim that is false, and nothing in the citation reveals that the error is upstream. Recording which step retrieved each chunk, and rendering the answer so the account list and the interview excerpts are separately attributable, is what lets a reader check the join rather than only the quotes.

Detecting that a question is compound at all can be folded into the planner rather than kept as a separate classifier. A planner that returns a single step has told you the query has one intent, which makes the design self-diagnosing and costs a generation call on every query including the 95% that are single-intent. The cheaper arrangement is the answer-type count from chapter 1 as a gate, with the planner running only on what it flags, and the number that decides between them is the gate's recall: a gate that misses a fifth of compound questions is sending them into a pipeline that cannot serve them, and those failures are invisible because a compound query always returns ten confident-looking chunks about its dominant intent.

Five paraphrases of a compound question#

Chapter 1 argued structurally that paraphrasing cannot separate intents, since a paraphrase preserves meaning and therefore preserves both of them. Running it on query D from that chapter gives the shape of the failure in numbers.

The five variants generated for which regions are still on the old rotation policy and what does migrating each one cost were: regions remaining on the legacy rotation policy and their migration costs, cost of migrating regions off the old rotation policy, which regions have not migrated from the previous rotation policy, and migration expense, legacy rotation policy regional footprint and migration pricing, and old rotation policy regions plus cost to move each. Every one is a faithful paraphrase and every one contains both intents.

The six retrieval legs, including the original, returned 60 chunks that deduplicated to 13, which is a 78% overlap rate — the pool barely grew. Of those 13, all 13 were rotation-policy documents and none was from the finance sheet holding per-region migration costs. The best-scoring cost chunk across all six legs sat at rank 31. Fusion over six legs that agree on the wrong answer produces a very confident wrong answer, and the reported recall for this query, measured against the churn-side gold chunk alone, would have looked fine.

That overlap rate is the same measurement chapter 5 made on a single-intent question, computed the same way from the same number of legs, and it comes out more than twice as high. The comparison is the diagnostic. On a question with one intent, paraphrases diverge enough to explore genuinely different regions of the index, which is the entire mechanism by which multi-query earns its points. On a question with two intents, they converge, because the dominant intent supplies most of the tokens in every variant and every variant therefore lands in the same place. An overlap rate is thus a cheap runtime signal that a query is compound, available before any judgement about the results and without a labelled set.

The decomposed version retrieves the finance sheet at rank 1 on its second leg, because per-region cost of migrating off the legacy rotation policy is a query about cost.

Where decomposition goes wrong on its own#

Over-decomposition is the common failure and it comes from a planner that has been told to split and therefore splits. how do I rotate the signing key becomes how do I rotate and signing key, two fragments that retrieve unrelated material and whose union is worse than the original query. The guard is an instruction to return a single step when the question has one intent, plus a check that every sub-query retains at least one entity or answer-type word from the original — a fragment that has neither is not a query.

Error compounding is the failure that surprises people. The second leg is conditioned on the first leg's output, so its accuracy is the product rather than the average. A churn query that gets the account list right 85% of the time feeding an interview retrieval that is right 85% of the time given a correct list gives 72% end to end, and adding a third dependent step takes it to 61%. Chains of dependent retrievals degrade multiplicatively, which puts a practical ceiling of two or three steps on this design before the correct architecture is an agent, meaning a loop in which the model inspects the results of one retrieval, decides whether they are good enough, and chooses the next query rather than following a plan fixed in advance.

Multiplying rather than averaging is the whole of that argument, and it is worth seeing why multiplication is the right operation. The second step is correct only in the cases where the first step was correct and the second step then succeeds, because a second step handed a wrong account list cannot produce a right answer by accident about the right accounts. That is a conjunction of two conditions, and the share of cases satisfying both is the share satisfying the first times the share satisfying the second given the first. Averaging would be the right operation for two steps each independently contributing part of an answer; it is the wrong one here precisely because the second step's input is the first step's output.

Decomposition also does not fix vocabulary. Each sub-query is a fresh query with all four defects available to it, and exit interview notes for Northwind Logistics still needs to be phrased the way the corpus phrases exit interviews — which in this corpus is offboarding call summary. The sub-queries are inputs to the rest of this topic's machinery rather than replacements for it.

There are now five distinct interventions, each targeting one of the four defects: filler-stripping rewrite with a preservation contract, contextual rewrite for referents, multi-query fan-out and HyDE for vocabulary, decomposition for compound questions. Each has been measured on the slice it addresses and each costs between 200 ms and 1.3 s.

The obvious next build is to run them all, in a sensible order, on every request. That version is cheap to write, easy to justify in a design doc, and it is how a pipeline arrives at a tripled p95 and a one-point eval movement.