The split's cost lives in the tail, on the runs that re-plan

A pipeline at one supplier-facing finance team takes documents off an intake queue — contract amendments, invoices, tax forms, insurance certificates — and classifies each one, extracts the fields the ERP needs, validates them against the purchase order, files the document, and notifies the owning buyer. The ERP here is the enterprise resource planning system, which is the finance team's system of record. The pipeline has run as a fused agent loop for eight months, and the proposal on the table is to replace it with the planner and executor of the previous four chapters.

The evidence brought to the review is that both designs complete 91% of a 300-document evaluation set, that the split adds one planner call up front, and that plans are much easier to debug than transcripts. Two of those three statements are measurements. The third is the recommendation wearing a measurement's clothes, and the first one, being equal, does no work at all until somebody asks what the incomplete 9% consists of in each design. What decides this is the shape of the cost distribution and the composition of the failures, and neither was in the room, because the instrumentation reported a mean.

Where the extra calls actually land#

The split fires a re-planning call on 12% of runs. That number came from the same 300-document set: 36 documents hit an executor failure that was final, most of them because the purchase order referenced on the document had been closed or amended between the plan being written and the validation step running.

Each of those runs pays a second planner call, and the planner's input is not small, because it carries the tool manifest, the plan schema, the goal, the completed steps with their bindings, the remaining steps and the failure record. On this pipeline that call measures 11,400 input tokens against the first planner call's 9,800 — the recovery context is longer than the original context, which is the opposite of what people expect from a call described as "handling an error". And then the steps the new plan re-issues have to run, so the total cost of a recovery is the second planner call plus the re-execution, which comes to 16,600 input tokens on this workload.

Averaged over 300 documents that is a modest addition. Write the mean as base cost plus the probability of recovery times the cost of recovery: 22,600 input tokens plus 0.12 × 16,600, which lands around 24,600. Set that against 41,000 for the fused loop, whose transcript grows with every tool result it has ever seen and therefore pays for its own history on every turn. Mean latency comes out at 15.5 seconds for the split against 19.8 for the loop. Presented as means, the split wins on both axes and the review would take four minutes.

The 12% is not spread across the distribution, and that is the sentence the means conceal. Every run that re-plans is, by construction, a run that did more work than a run that did not — it executed some steps, failed, produced a second plan, and executed more steps. So those 36 documents occupy the top of the latency and token distributions almost exclusively.

Now locate the percentiles inside that. Sorting 300 documents by latency, the 95th percentile is document 285, which leaves only a handful of documents above it, and every one of those handful is drawn from the 36. The 99th percentile is document 297, which on this set is a document that re-planned twice before exhausting its budget. The median sits far below anything that re-planned. So the median measures a population the recovery path never touches, and the two percentiles measure a population that is almost entirely recovery — which means the mean, sitting somewhere between them, is a blend of two populations that behave completely differently and describes neither.

The sign of the difference column flips as you read down the latency rows and again as you read down the token rows, and the flip is the whole finding. At the median the split is 4.2 seconds faster and uses 17,200 fewer input tokens. At p95 it is 12.1 seconds slower and still slightly cheaper in tokens. At p99 it is slower and dearer on both. Two designs, one measurement set, and which one is better depends entirely on which percentile the question is about.

The pipeline's operational commitment is that 95% of documents are filed within 45 seconds of arriving on the queue, which is a number in a service description that the finance team reads. The split's median is four seconds better and its p95 breaches that commitment by 1.2 seconds. Nothing about the mean, the cost or the completion rate surfaces that, and the mean is what the existing dashboard shows — a statistic computed over the whole population, compared against a commitment written about one point in it.

The two tails also have different causes, which matters when someone proposes to fix one. The fused loop's p99 is made of documents where the model iterated close to its cap, usually multi-page amendments where extraction and validation ping-ponged. The lever there is the iteration cap, and it is a bad lever: raising it makes that tail worse and lowering it converts slow successes into failures. The split's p99 is made of documents that re-planned, and the levers there are the re-plan budget and the size of the recovery context — the first bounds how many recoveries a document may attempt and the second reduces what each one costs. Those are unrelated pieces of engineering with unrelated risks, and calling them both "the tail" hides that a fix for one is not a fix for the other.

Measuring it without flattering either design#

The measurement has two failure modes that are easy to walk into and both make the split look better than it is.

@dataclass
class TaskRecord:
    task_id: str
    design: str                  # "fused" | "split"
    outcome: str                 # "completed" | "failed" | "halted"
    wall_seconds: float          # arrival on queue -> terminal state
    model_calls: int             # planner + re-planner + any llm.* steps
    input_tokens: int            # every call, including recovery turns
    output_tokens: int
    replans: int

def report(rows: list[TaskRecord], design: str) -> dict:
    r = [x for x in rows if x.design == design]          # every task, whatever its outcome
    lat = sorted(x.wall_seconds for x in r)
    tok = sorted(x.input_tokens for x in r)
    return {
        "n": len(r),
        "completion": sum(x.outcome == "completed" for x in r) / len(r),
        "lat_p50": pct(lat, 50), "lat_p95": pct(lat, 95), "lat_p99": pct(lat, 99),
        "tok_p50": pct(tok, 50), "tok_p95": pct(tok, 95), "tok_p99": pct(tok, 99),
        "replan_rate": sum(x.replans > 0 for x in r) / len(r),
    }

The first failure mode is aggregating over the wrong unit, and it is arithmetic rather than taste. A re-planning document contributes one slow task and about a dozen ordinary steps, since most of what a recovery re-issues are cheap dispatches. So moving the unit from tasks to steps multiplies the denominator by roughly the step count while leaving the number of slow observations where it was, and a tail that was a few per cent of tasks becomes a fraction of a per cent of steps. The step-level p95 barely moves while the task-level p95 moves by twelve seconds, and neither number is wrong — they answer different questions. The unit has to be the thing the commitment is made about, and the commitment here is about documents.

The second is filtering. Dropping failed and halted runs before computing latency removes exactly the population that spent the most time, and the bias has a direction: on the split that population is disproportionately the re-planning runs that exhausted their budget, so the filter takes more off the split's tail than off the loop's. A filter that improves one design more than the other is not a cleaning step. The line r = [x for x in rows if x.design == design] has no outcome filter for that reason, and resisting the urge to add one is most of the discipline.

Beyond that, the comparison has to be paired. Both designs run the same 300 documents, and each document produces two records, so the variance from document difficulty cancels when you look at per-document differences rather than at two aggregate numbers — a hard document is hard for both designs, and pairing removes that shared difficulty from the comparison instead of leaving it in both averages.

Pairing is also what makes the 91% figures interpretable at all, and the arithmetic here is short enough to do. The standard error of a proportion is the square root of p(1 − p)/n, where p is the proportion and n the sample size. At a proportion near the 91% these designs report, with n = 300, that comes to about 1.7 percentage points. So two unpaired estimates each carrying 1.7 points of noise cannot resolve a difference of a point or two in either direction, and "91 versus 91" means the difference is smaller than the noise rather than that the designs are equivalent. Reporting it as "none measurable" in the table above is the honest phrasing; reporting it as "no difference" would be a claim the sample cannot support.

Where the split can still be right when the numbers are equal#

Equal completion is not equal failure, and this is where the evidence on this pipeline actually turned up. Pair the outcomes on the 27 failures each design produced — 300 documents minus 273 completions, the same count on both sides — and classify them.

Read the rows for domination rather than the totals for a winner. The split is better on every row except the first, and the first row is the benign one — halted with a stated reason and nothing written is the failure you want, because it costs a person a look and costs the ERP nothing. On the four rows that describe something wrong reaching a real system, the split is better or equal every time, and on two of them it is at zero.

Add up the rows where something wrong was written without the run saying so: filed against the wrong purchase order, buyer notified from a superseded amendment, and extraction wrong but filed anyway. That is 4 plus 3 plus 6 for the fused loop, which is thirteen, against 0 plus 1 plus 0 for the split, which is one. The last row is excluded because a run that gives up at the iteration cap does say so, even though it leaves writes behind.

The mechanism producing those thirteen is the one chapter two described. The fused loop had no notion of a remainder that a failure invalidates, so it kept going and produced a plausible terminal answer. Every one of the thirteen is a run that carried on after something did not hold.

If the cost of a silently misfiled document is a correction that takes a person twenty minutes and occasionally a supplier phone call, then thirteen of them per 300 documents is 4.3 hours of rework per 300, and that is a number to put beside the 1.2-second breach of the service-level agreement — the SLA the finance team reads — rather than a vibe. The evidence you need for this argument is a severity classification of every failure, agreed with the person who cleans them up, applied to both designs on the same documents. Without that agreement the classification is the same recommendation-wearing-a-measurement that opened the chapter, with a table around it.

Two other conditions make the split worth its tail. If the pipeline moves to overnight batch, the p95 commitment stops existing and the median and the cost per thousand are what remain, at which point the split wins outright — the entire case against it was a commitment about a percentile, and a batch has no such commitment. And if the plan can be shown to a buyer for approval before anything is written, which the fused loop cannot offer since it has no artefact until it has already acted, then the four wrong filings become four rejected plans. The evidence for that second one is the plan-rejection rate in a pilot rather than an argument from first principles, because a plan a buyer approves without reading rejects nothing.

Chapter one's observation about executor steps has a cost consequence here that is easy to miss, and it explains two rows of the first table at once. Because most steps in the split are dispatch rather than generation, the split's median model call count is three against the loop's six, and two of those three are the planner and a single llm.extract step. That is what produces the $49 per thousand saving. It is also what makes the recovery path expensive by comparison: when a document re-plans it adds another long-context model call to a run that only had two, so a re-plan roughly doubles the model work on that document, where the same event on a design that already made six calls would barely register.

What "two smaller prompts" is worth#

The maintainability argument is usually stated as a fact about prompt length, and taken literally it is true. The fused loop's system prompt on this pipeline is 2,100 tokens and holds the task description, the tool descriptions, the ordering rules, the validation policy and the error-handling guidance. The split's planner prompt is 1,400 tokens and the executor needs no prompt at all for four of its six step types.

That accounting stops too early, because the split replaced prompt surface with system surface, and the second is larger than the first. The artefacts the team now maintains are the planner prompt, the plan schema, the validator, the run-state format, the binding language, the re-plan policy with its budget, the tool registry's idempotency and recovery declarations, and the two extra step types that do call a model. The fused loop maintains one prompt and a tool registry. Fewer tokens of prompt is a real improvement in one dimension and a larger number of moving parts is a real cost in another, and the two are not denominated in the same unit, which is why stating only the first makes the argument feel decisive when it is not.

There is a version of the claim that can be measured. Topic 11.1 made the point that few moving parts give a failure one address, and the mirror of it applies once the parts exist: an explicit plan artefact turns some triage from reading a transcript into diffing a plan against what executed. On this pipeline the team timed it over twenty failing documents, ten per design, and reached a median of 6 minutes to diagnose a split failure against 14 minutes for a fused one. The gap was concentrated in cases where the question was "which decision was wrong" rather than "which field was extracted badly" — the second question is answered equally well by either record, because the extracted field is in both. That is evidence, it is comparable to the SLA number and the rework number, and it is the form the maintainability argument has to take to belong in the decision at all.

None of this generalises off this pipeline, and the reason is a single ratio. The numbers above are one team's measurement of one document-processing workload with one model and one set of tools, and the thing that drives all of them is the 12% re-plan rate, which is a property of how often supplier purchase orders change between planning and validation rather than a property of the architecture. Run the same arithmetic at a different rate and the conclusions invert. A pipeline whose world is stable re-plans on 1% of runs, which puts the recovery population below the 99th percentile entirely, so there is no tail to argue about and the median and cost figures decide it. One whose world changes constantly re-plans on 40% of runs, at which point recovery is not a tail at all but the ordinary case, and a design that commits to a sequence in advance is committing to something that will be wrong twice a week — which is an argument for not planning ahead rather than for planning better. The re-plan rate is the first thing to measure and the last thing anybody thinks to.

Which leaves the case where none of this arithmetic gets a chance to run, because the task is three steps long, nothing about it fails in an interesting way, and the argument for splitting it was never about recovery.