Coordination loses through failure surface, duplicated context and hops

The support-triage workload went the other way. The task is to take an inbound customer ticket, decide what it is about, look up the account's entitlements and recent incidents, check the request against refund and escalation policy, and produce a reply plus a resolution action. The team built it as a five-stage pipeline — classify, retrieve, policy-check, draft, review — because the stages map cleanly onto the runbook a human agent follows. A single loop with the same four tools was written as the comparison arm in an afternoon.

On a held-out set of 200 tickets at five trials each, both arms audited once under the previous chapter's rules, the loop scored 91.0% and the pipeline 84.2%. The paired gap was −6.8 points with bounds at −9.4 and −4.2. Reading the pipeline's 158 failures produced no seventeen-of-twenty-six cluster: the failures were spread across all four boundaries and the five stages, with different immediate causes, and the largest single group was eleven. Two engineers spent a day fixing the three most common ones, which recovered 1.4 points and left a gap of 5.4.

That distribution is the signature of a topology losing on its merits, and saying so credibly requires measuring the three mechanisms rather than asserting them.

Every boundary is a place the task can fail#

A handoff can drop a field, as the contract pipeline's did, and it can also degrade a task in ways no schema catches: a classification that is defensible but slightly wrong narrows what the retrieval stage looks for, a policy check that receives a summarised ticket misses that the customer already escalated once, a draft that receives four policy citations foregrounds the third when the second was decisive. None of those is a bug against a stated contract. Each is a small loss of fidelity, and there are four places for one to occur.

Attributing a failure to a boundary means checking whether the stage downstream of it would have succeeded given the correct input. That is answerable by re-running one stage with a hand-corrected input, which is expensive per case and cheap enough on a sample.

def boundary_attributable(trace: Trace, boundary: str, gold: Payload) -> bool:
    """Re-run everything downstream of `boundary` with the corrected payload."""
    downstream = trace.steps_after(boundary)
    replayed = replay(downstream, initial_input=gold, seed_pool=5)
    return score(trace.task, replayed) > score(trace.task, trace.result)

Applied to a stratified sample of 60 of the 158 failures — drawn so that each stage and each failure category contributes in proportion to its share of the whole, rather than taking the first 60 and inheriting whatever the ordering favoured — and scaled back up, the distribution comes out flat.

Two per cent per boundary sounds negligible and compounds badly, because a task has to survive all four. If each boundary independently degrades the task with probability 0.02, the chance of clean passage is 0.98⁴ = 0.922, so about 7.8 points of loss arrive from boundaries alone before any stage has reasoned incorrectly. The loop has no boundaries and pays none of it. The measured total, 8.0 points of boundary-attributable failure across the four, sits almost exactly on that prediction, which is a useful sanity check on the attribution method.

The failure surface argument is often stated as an inevitability of decomposition, and it is not quite. Boundaries between stages whose outputs are independently verifiable cost far less, because a wrong output can be detected and retried at the boundary rather than propagating. The classify stage's output is verifiable in that sense — a ticket either mentions billing or it does not — and its boundary rate is nonetheless the highest of the four, because "defensible but slightly wrong" is exactly the case a verifier passes.

Context re-sent is context paid for#

The pipeline spends 47,800 input tokens per ticket against the loop's 12,300. The stages are not doing four times the reasoning; they are reading the same material repeatedly. Measuring how much is arithmetic on the trace, once you define duplication as content in a stage's input that was already sent to an earlier stage in the same run.

def duplicated_input_tokens(trace: Trace) -> tuple[int, int]:
    seen: set[str] = set()
    total = dup = 0
    for step in trace.steps:
        for block in step.input:
            for chunk in shingles(block.content, size=256):
                n = token_count(chunk)
                total += n
                if chunk in seen:
                    dup += n
                else:
                    seen.add(chunk)
    return dup, total

Across the 1,000 pipeline runs, 29,200 of the 47,800 input tokens per ticket were duplicates, or 61%. The ticket body appears in all five stages, the account record in four, and the policy excerpt in three. At the arms' measured rates the cost per ticket is $0.031 for the loop and $0.142 for the pipeline, and roughly $0.087 of the pipeline's cost buys nothing except the same bytes arriving at a second, third and fourth model call.

Prompt caching changes that number substantially and does not remove the mechanism. Providers will store the processed form of a prompt prefix for a few minutes and charge a fraction of the usual input rate when a later request begins with the identical bytes, which is exactly the situation five stages re-sending the same ticket create. Marking the ticket body and account record as cache-eligible prefixes brought the pipeline's cost per ticket to $0.061 on a 78% cache-hit rate, which closes more than half the cost gap. Latency improved less than the cost did, at 9.8 seconds p50 to 8.9, because cached prefixes still have to be transferred and the stages remain serialized. Caching is a discount on one of the three mechanisms, and a comparison run with caching on one arm and off the other is measuring the discount rather than the topology.

Hops add to the tail, and not only through generation#

Chapter 1 established why a sum of stage latencies has a worse tail than any individual stage. Each hop in a pipeline is one stage's output becoming the next stage's input, with a model call waiting on the far side of it, and the stages run one after another because each needs what the last produced. What the support pipeline adds to that picture is how much of the added latency is not generation at all. Instrumenting each stage boundary separately splits the total into model time and coordination time, where coordination covers serialising the payload, schema validation, the retry the validator triggers on malformed output, and the queue wait before the next stage's request is dispatched.

The 2.2 seconds of coordination at p50 is 22% of the pipeline's latency spent on plumbing, and the retry rate is where most of it lives: 0.31 validation-triggered retries per task means roughly a third of tickets pay one full extra model call somewhere in the chain. At p95 the picture is worse than the sum of the stage p95s would suggest, because a validation retry is correlated with a long generation — the outputs that fail schema validation are disproportionately the long, rambling ones.

The test that separates coordination from everything else#

The three measurements above are consistent with coordination being the cause, and consistency is weaker evidence than most write-ups admit, since a pipeline also differs from a loop in prompt length, in tool availability per stage, and in how the team happened to write it. The controlled version of the question is whether the gap scales with the number of boundaries, which can be tested by merging adjacent stages while holding the workload, the tools, the fixtures, the rubric and the manifest fixed.

Four configurations were built by merging: the five-stage original, a three-stage version (classify+retrieve, policy-check, draft+review), a two-stage version (everything up to policy-check, then draft+review), and the loop. Each was audited once, run at five trials over the same 200 held-out tickets.

The gap grows at roughly 1.3 points per boundary, the token count at about 8,900 per boundary, and p95 at about 5.6 seconds per boundary, each close enough to linear that the boundary count explains the trend better than any other property that varies across the four builds. That is the evidence which makes "the coordination is the problem" a measured claim rather than a preference, and it is what the previous chapter's discipline demands: a loop win is discountable as somebody's broken payload until the loss is shown to be distributed and to scale with the thing being blamed.

The ladder is not a perfect control. Merging two stages produces a longer prompt for the merged stage, so the five configurations differ in prompt length as well as in boundary count, and prompt length has its own effect on quality. The honest version of the claim is that boundary count and merged-prompt length are confounded in this design, that the token and latency trends are attributable to boundaries with near certainty because the mechanism is arithmetic, and that the quality trend is attributable to boundaries with moderate confidence. Attribution of individual failures to a boundary also involves judgement; two reviewers labelling the same 60 failures agreed on 48 of them, which is enough to trust the shape of the distribution and not enough to trust any single row to the tenth of a point.

Why the same measurement pointed the other way on contracts#

The contract pipeline won its confirmation set by 4.2 points at 2.07× cost, and this support pipeline loses by 5.4 at 4.6× cost, under the same harness and the same rules. The difference is legible in the duplication figure. Contract review decomposes into stages that need genuinely different context: the extraction stage reads the document, the normalisation stage reads clauses and the policy list, and the drafting stage reads normalised clauses and needs almost none of the original text. Measured duplication there was 18%. Support triage decomposes into stages that all need the ticket, the account and the policy, so its duplication is 61% and every boundary is an opportunity to lose fidelity on shared context that never needed to be split in the first place.

Topic 11.17 gave the qualitative version of that condition. The contribution here is that duplicated input tokens is a number you can compute from traces before committing to a design, and it predicts which side of the comparison a workload will land on well enough to be worth computing early. A proposed decomposition whose stages share most of their context is one where the measurements in this chapter are likely to reproduce.

Where this is heading is handoff by reference rather than by value — stages that receive a pointer into a shared context store and read what they need, so duplication falls without merging the stages. Early implementations cut the token cost substantially and leave the failure-surface and hop-latency mechanisms untouched, which the ladder above suggests are together most of the quality gap. That is the state of the art rather than settled practice, and it is worth re-measuring against rather than adopting on description.

Two workloads have now produced opposite answers, each with a named and measured cause. What has to happen next is that one of them gets written down in a form that a reader in six months can check, including the thresholds that made the choice and the conditions that would undo it.