An answer built from four of five workers reads exactly as whole

Coverage, in a fan-out design, is the question of whether the set of worker results being synthesised is the set that was dispatched. It is not a property of any single component. The dispatcher knows what it sent, the synthesis step knows what it received, and until something compares those two lists nobody holds the difference. That makes a coverage gap a peculiar kind of defect: it is not a crash, not an exception, not a malformed response, and not visible anywhere in the output it corrupts.

The system this chapter works from is Scout, an internal competitive-research assistant whose orchestrator — the model call that reads a goal and decides what smaller pieces of work it implies — dispatches those pieces as subtasks, one per worker, each worker being a separate model call with its own tools and its own fresh context. A synthesis step then folds the returned findings into the answer a user reads. Every dispatched subtask carries a deadline, the time the parent is willing to wait before giving up on it, because a parent that waits forever on a hung worker never answers at all. One of Scout's workers exceeded its deadline often enough to matter, and the system kept answering.

An answer built from four subtasks is the same kind of object as one built from five#

The CRM worker had a 25-second deadline. The CRM — the customer-relationship management system where sales representatives log what happened on a call — had a search endpoint with a p95 of 22 seconds, which is the time all but its slowest requests came in under. Its tail ran well past that whenever an account had more than about 400 notes attached, which described most of the enterprise accounts anyone cared about. Across the two months after launch, that worker exceeded its deadline on 6.1% of Scout's runs — 143 runs out of 2,340.

On every one of those runs Scout returned an answer. The chain that produces that outcome has no defect anywhere along it, which is what makes it worth walking. The dispatcher gathers whatever has arrived by the deadline, because that is what a deadline means. The synthesis step receives four worker outputs instead of five, and nothing in a list of four documents announces that it is short. The model writes the answer the four documents support, at the length and in the shape its prompt asks for. Every step is behaving correctly, and the composition of correct steps is a wrong answer that reads as a right one.

The measurements bear that out. Those runs produced four well-organised paragraphs with headings and a source list, at a median 620 words against 640 for the complete runs. Nobody filed a bug. The thumbs-down rate on the affected runs was 4.2% against 3.9% overall, a difference well inside the noise for 143 samples.

The missing worker was the one that read CRM notes. The 143 answers built without it discussed public pricing, packaging and review-site sentiment with complete confidence and said nothing about what was happening in live deals, which for half of those questions was the part the reader needed. There was no marker in the output, no hedge in the prose, no field in the API response. An answer built from four subtasks is byte-for-byte the same kind of object as one built from five.

Every metric was working and none of them could see it#

The dashboards were not neglected. Worker timeouts were instrumented from day one and the 6.1% figure comes straight from them; it was on a panel, and it was read as a latency problem in the CRM integration, which is what it was, at the layer where it was measured.

Read that table down the right-hand column rather than across the rows. Six metrics, six different explanations, and every explanation is the same statement in different vocabulary: each metric is a function of one component's behaviour, and each was true about that component. The defect lives in the relationship between two components — what was dispatched and what came back — and there was no metric whose value depended on both. The 6.1% panel and the answer-quality panel were both green because each was true about its own layer, and the join between them existed nowhere except in a request-scoped variable that was garbage-collected when the response was written.

That is the stumble worth naming, and it is the same shape as the previous two chapters'. A quantity was held constant and read as a defence — the timeout panel at a steady 6.1%, the synthesis success rate at 99.8%, both watched by people who were not being careless — while the property that decided whether a reader got a usable answer varied freely underneath, that property being whether the run's output represented what it was missing. Instrumentation of components is not instrumentation of a run.

The human-review sampling deserves a second look because it is the control people trust most, and its failure is arithmetic rather than judgement. Twenty runs a week, graded by someone competent, over eight weeks, is 160 runs. At the 6.1% rate that is about 10 degraded runs in the whole eight weeks — call it one a week, arriving alone, among nineteen normal ones. A reviewer grading a degraded answer sees four coherent paragraphs answering the question asked, with no internal evidence that a fifth line of investigation was attempted. They mark it a seven out of ten, note that it could have said more about live deals, and move on. Detecting this by review would require the reviewer to hold the decomposition next to the answer, which is not what a grading rubric asks for unless someone has already suspected the failure.

Coverage information lives in dispatch state#

The first proposal in the incident review was to add a line to the synthesis prompt: if evidence for part of the question is missing, say so explicitly rather than answering as though it were complete. This is the reflex that a prompt problem has a prompt solution, and it fails for a reason worth being precise about.

Follow what the synthesis call actually holds. Its context contains four worker outputs and the original goal. Nothing in that context refers to a fifth subtask. The subtask existed in the orchestrator's dispatch table, which lives in the parent process and is not serialised into the prompt. So the model is being asked to notice the absence of something it has never been told about. It can, at best, notice that the goal mentions live deals and that none of its four documents discuss them, which is a weak inference the model will make inconsistently. That fallback fails entirely when the missing worker's territory is not named in the goal, which is the common case, since the whole purpose of decomposition is to derive sub-questions the goal does not spell out. Asking a model to perceive an absence is asking it to compare what it received against an expectation it does not hold.

Coverage information is dispatch state. It exists in exactly one place, and the fix is to carry it out of that place and into everything downstream, which means the synthesis input, the answer object, and the response the caller receives.

Making the gap representable#

Start with the subtask lifecycle, because it is the thing the manifest records — the manifest being the parent's own list of what it dispatched and how each dispatch ended, the dispatch state that the synthesis call never sees. A dispatched subtask ends in one of four terminal states, and only one of them is what the plan assumed.

The branch structure is the content of that diagram. The first split is about time — did anything arrive before the deadline — and it is decided by the parent's clock. The second split is about what arrived, and it is decided by reading the payload: schema-valid and non-empty is ok, schema-invalid is invalid, schema-valid with nothing in it is empty.

empty matters as much as timed_out and is easier to miss: a worker that searched correctly and found nothing is a legitimate result about the world, while a worker whose source was misconfigured also returns zero findings, and the two are distinguishable only if the worker reports how many sources it actually reached.

@dataclass(frozen=True)
class SubtaskOutcome:
    subtask_id: str
    question_ids: list[str]
    required: bool
    state: Literal["ok", "timed_out", "invalid", "empty"]
    sources_reached: int
    findings: list[Finding]

async def dispatch_all(plan: Plan) -> list[SubtaskOutcome]:
    async def run(st: Subtask) -> SubtaskOutcome:
        try:
            raw = await asyncio.wait_for(worker.run(st), timeout=st.deadline_s)
        except asyncio.TimeoutError:
            return SubtaskOutcome(st.id, st.question_ids, st.required,
                                  "timed_out", 0, [])
        return classify(st, raw)

    return await asyncio.gather(*(run(st) for st in plan.subtasks))

The important property of that function is that a timed-out subtask still produces a SubtaskOutcome. Nothing is dropped from the list; a failure becomes a record with state="timed_out" and an empty findings list, so the outcome list is always the same length as the plan's subtask list. That is what makes the comparison downstream possible at all — you cannot compare what was dispatched against what returned if the failures simply do not appear.

asyncio.gather over the plan's subtasks gives correlation for free — the outcome at index i belongs to the subtask at index i, and the subtask_id carries it explicitly for anything that reorders. The deadline is per subtask rather than per run, so a slow worker cannot consume the budget of the ones behind it. That per-subtask placement is a decision, not a requirement: a single run-level deadline is a legitimate design, and it makes one worker's tail everyone's problem.

The synthesis boundary is where the representation earns its keep:

def synthesise(goal: str, outcomes: list[SubtaskOutcome],
               policy: CoveragePolicy) -> Answer:
    missing = [o for o in outcomes if o.state != "ok"]
    unmet_required = [o for o in missing if o.required]

    if unmet_required and policy.on_required_gap == "fail":
        raise IncompleteCoverage(unmet_required)

    answer = model_synthesise(goal, [o for o in outcomes if o.state == "ok"])
    return answer.with_coverage(
        Coverage(
            complete=not missing,
            missing_questions=[q for o in missing for q in o.question_ids],
            degraded_reason=[f"{o.subtask_id}:{o.state}" for o in missing],
        )
    )

Answer now has a coverage field that is not optional, so no code path can produce an answer object without stating what it covers, and the serialiser renders it. A degraded response comes back looking like this, and the client renders the banner from the structured field rather than hoping the prose mentions it:

{
  "text": "Vantage moved its enterprise floor from $48,000 to $60,000...",
  "coverage": {
    "complete": false,
    "missing_questions": ["q3"],
    "degraded_reason": ["st_2:timed_out"]
  }
}

The required flag comes from the decomposition schema of chapter 2, set by the orchestrator when it assigns questions to subtasks, and it is what turns coverage from a binary into a policy. Not every gap should stop the run. A missing review-site worker on a pricing question is a footnote; a missing CRM worker on "which live deals are exposed" removes the answer's subject. Marking subtasks required or optional at plan time lets the parent fail loudly on the second and degrade visibly on the first, and it puts that judgement where the goal is still in view.

What makes this impossible rather than merely unlikely is that the gap has nowhere to hide, and the argument has three separate legs. Coverage is a required field on Answer, so a code path that forgets it does not compile in a typed language and fails at construction in Python. The value is computed from the outcome list rather than from anything a model said, so no wording, temperature or prompt revision can change it. And a test that stubs one worker to raise TimeoutError asserts that the returned answer has complete == false and names q3, which fails if someone later adds a shortcut that synthesises from a raw list of findings — the realistic way this regresses. Compare that with the prompt instruction, whose correctness cannot be asserted by any test that does not itself call a model and grade prose.

What the coverage field licenses is exactly one claim: the answer states which of the dispatched sub-questions produced usable material. Its residual is everything on either side of that. It says nothing about whether the dispatched set was the right set, which is chapter 2's problem and which no downstream component can see. And it says nothing about material that arrived and was then dropped inside the synthesis call, which is a later chapter's problem, because a claim that reached synthesis and did not reach the answer leaves complete reading true.

One retry lowers the rate and leaves the class alone#

Retrying the timed-out worker is a reasonable thing to do and it is not the fix. Scout added one retry with a 40-second deadline, and the rate at which a run finished with a missing CRM worker fell from 6.1% to 1.4%. That is a real improvement in how often the bad thing happens, and it changes nothing about what happens when it does: on those 1.4% of runs the synthesiser still receives four documents, still writes a whole-looking answer, and the reader still cannot tell. The class of failure — a hole that the output does not represent — survives at a lower frequency, which is the definition of making something unlikely rather than impossible.

Retry also has costs that argue for doing it deliberately rather than reflexively. The second attempt is serial with the first, so p95 end-to-end latency rose from 26 to 51 seconds on the runs that took it. Some states should not be retried at all: an invalid outcome from a worker that violated its output schema will usually violate it again, and an empty outcome from a correctly configured source is a finding, not a failure. The sensible arrangement is retry as one branch of the coverage policy, chosen per state and per subtask criticality, sitting behind the representation rather than in place of it.

policy = CoveragePolicy(
    retry_states={"timed_out": 1, "invalid": 1, "empty": 0},
    on_required_gap="fail",       # raise rather than answer
    on_optional_gap="degrade",    # answer, with coverage populated
)

Every value in that object is a decision rather than a requirement, and writing them in one place is what makes them arguable. Retrying a timeout once and an empty result never is a claim about which states are worth a second attempt. Failing on a required gap and degrading on an optional one is a claim about what a caller would rather receive. Both are the sort of claim that used to live in nobody's head and now lives in a file somebody can change.

Refusing to answer is the other failure mode#

Representing gaps is not an argument that partial answers are unacceptable. The opposite policy — treat any non-ok outcome as a run failure and return nothing — would have converted 143 degraded answers into 143 non-answers, and for the questions where the CRM worker was incidental that trades a slightly thin answer for no answer at all. Availability is a real property and a research assistant that refuses to speak whenever a review-site scrape times out will be abandoned.

The distinction the required flag draws is what keeps both failure modes off the table. It is worth being sceptical of an orchestrator that marks everything required, which is the lazy setting that produces the refusing system, and equally of one that marks nothing required, which produces the original defect with better logging. Scout's decomposition prompt was changed to require a one-line justification per required subtask, and the fraction of subtasks marked required settled around 40%, concentrated in the workers whose questions were named directly in the goal.

That settling point is a fact about Scout's goals rather than a number to copy. An assistant whose questions are mostly answerable from one source would settle higher, and one that fans out over many optional colour sources would settle lower. What transfers is the mechanism: requiring a justification makes marking a subtask required cost something, and a flag that costs nothing to set gets set on everything.

With the policy in place the parent knows exactly which of its subtasks produced usable material and which did not, and the answer it returns says so. What it still does not know is what to do when two of the outcomes it did receive assert incompatible things, or what to do when there are eleven of them and they do not fit in one context window.

Both of those are properties of the aggregation step rather than of the dispatch, and neither is solved by knowing that the set is complete. A complete set of contradictory findings is still a set somebody has to resolve, and the resolution is where the next chapter starts.