Back-port the fixes before comparing the tree with the loop

The service being compared here writes a weekly competitive brief on eighteen companies, and it exists in three shapes. The flat loop is one tool-using agent that works through all eighteen companies in sequence and writes the brief itself. The two-level tree is an orchestrator that delegates to six segment supervisors, each of which runs three research workers and returns a segment summary. The three-level tree puts two sector supervisors between the orchestrator and those six segments, so a finding crosses three delegation boundaries on its way up. A delegation boundary, throughout this topic, is one place where a task is handed from one agent to another and a result is handed back.

Choosing between those shapes is an empirical question, and the experiment that answers it is unglamorous. A comparison supports a causal claim about topology when the arms differ in topology and in nothing else, which sounds obvious and is violated by nearly every such comparison in practice — because the arms are not built at the same time. The tree is built second, every repair discovered during that work lands in the tree's prompts, and the baseline sits untouched in the repository where it was left. The gap that comes out is real, and what it measures is the calendar.

Everything that has to be equal before a difference means topology#

Writing down "everything else" is most of the work, because a variable nobody has named cannot be held fixed. On this workload it comes to five things.

The model must be the same, including the effort setting — the dial that tells the model how much reasoning to spend before it answers. A tree whose supervisors run at high effort against a baseline at medium is an experiment about effort with a topology label on it.

The tool budget must be the same, and the budget rather than the tool list. A single loop making 96 tool calls against a tree whose eighteen workers make 91 between them is a fair fight, since both arms did about the same amount of work in the world. A tree with no cap against a loop capped at 40 is not a fight at all. The distinction matters because the tool list is a capability and the budget is a resource, and only the second one is what a topology is usually accused of spending more of.

The task set must be identical, drawn from the same distribution as production, and run in the same order against the same fixtures. A fixture here is a recorded copy of the pages and filings a run reads, replayed from disk so that both arms see the same web on the same day. Without fixtures, a filings endpoint that goes slow for an hour hits whichever arm happened to be running, and that hour becomes a topology result.

The prompt and tool repairs must be back-ported. This is the one that gets skipped, and it is skipped for a structural reason rather than a careless one: the repairs were discovered while building the tree, so they live in the tree's files, and nobody has a reason to open the baseline's files at all. Nothing in a test suite fails when a baseline goes stale. The staleness is invisible until someone reads a modification date.

The failure injection must be identical across arms. Recovery — whether a bad intermediate result is stopped before it reaches the output — is the property hierarchy has the most plausible claim on, and it is the one nobody measures, because a regression suite scores runs that succeed and a run that recovered from an injected defect is simply a run that succeeded.

The harness that refuses a confounded run#

A good intention about back-porting survives until the first deadline. Making it a mechanical precondition costs a dataclass and a loop, and turns the omission into a crash rather than a footnote in a deck.

from dataclasses import dataclass

@dataclass(frozen=True)
class Arm:
    name: str
    topology: str                    # "flat" | "two_level" | "three_level"
    model: str = "claude-opus-5"
    effort: str = "high"
    tool_budget: int = 100           # total tool calls per run, all agents
    prompt_pack: str = "2026-03-r4"  # the shared prompt + tool revision
    constraints: tuple[str, ...] = tuple(BRIEF_CONSTRAINTS)
    return_contract: str = "sector_return_v3"
    injection_rate: float = 0.05

ARMS = [
    Arm("flat", topology="flat"),
    Arm("two_level", topology="two_level"),
    Arm("three_level", topology="three_level"),
]

def assert_only_topology_differs(arms: list[Arm]) -> None:
    fields = [f for f in Arm.__dataclass_fields__ if f not in {"name", "topology"}]
    for f in fields:
        values = {getattr(a, f) for a in arms}
        if len(values) > 1:
            raise ConfoundedComparison(f"{f} differs across arms: {values}")

The assertion is a set operation and it is worth reading slowly, because its strength comes from what it does not enumerate. It collects the names of every field on Arm except name and topology, gathers the values each arm gives that field, and raises if any field has more than one value across the arms. Nothing in it knows what effort or prompt_pack mean. The consequence is that adding a sixth thing to hold fixed is a matter of adding a field, and from that moment the check covers it — whereas a hand-written list of comparisons covers exactly the confounds somebody thought of on the day.

prompt_pack is the field that carries the chapter's title, and it is the reason the flat arm cannot silently keep its 2025 prompt. The pack is a single versioned directory holding the system-prompt fragments, the constraint list and the tool wrappers, and all three topologies load from it. That arrangement is what makes the back-port mechanical: there is no per-arm copy to forget, because there is only one copy. Whether the flat researcher uses the sector-summary fragment is a topology question and is allowed to differ between arms; whether it gets the extracted-text tool wrapper is not, and the pack takes that choice away from whoever is in a hurry.

Two things in Arm are worth separating, because they look alike and are not. tool_budget = 100 and effort = "high" are requirements of a fair comparison: any value would do, provided every arm gets the same one. injection_rate = 0.05 is a decision — a rate somebody chose, which sets how many of the 200 tasks carry a corrupted claim and therefore how precisely the recovery column can be estimated. Raising it would tighten that column and change nothing else. Knowing which knobs are which is what lets a reviewer ask the right question about each.

A defect the system could plausibly have produced#

Injecting a failure is easy to do badly. A corruption that is obviously wrong — a null field, a company that does not exist, a revenue figure of zero — is caught by a schema check in every arm, and the recovery column comes back at 100% everywhere, having measured nothing. The injection has to look like the system's own mistakes.

def inject_worker_defect(payload: dict, rng) -> dict:
    """Corrupt one claim the way a real worker fails: plausible, well-formed, wrong."""
    company = rng.choice(payload["companies"])
    claim = rng.choice(company["claims"])
    claim["statement"] = perturb_magnitude(claim["statement"])   # $310M -> $410M
    claim["as_of"] = "2025-08-14"                                # outside the window
    return payload

def recovered(final_brief: dict, injected: Claim) -> bool:
    """Did the corrupted claim fail to reach the customer-visible output?"""
    return injected.statement not in flatten_text(final_brief)

Both corruptions are the shapes a real worker produces. perturb_magnitude moves $310M to $410M, which is a figure of the right kind in the right units attached to the right company, and no check that does not know the true number can reject it. The as_of is moved to a date outside the fiscal window, which is checkable, and the pair is deliberate: one half of the defect tests whether any arm holds a rule that catches it, and the other half tests whether an arm can catch something no rule covers. The function mutates the claim in place and hands the payload back, so the run continues with the corruption in it rather than around it.

recovered is the strict half. It asks one question — is the corrupted statement absent from the customer-visible brief — and it is deliberately blind to how that happened. It does not care whether a segment supervisor rejected the return, whether the contract check caught the stale as_of, or whether the orchestrator noticed the magnitude was implausible. A definition that credited the mechanism would let an arm score well for rejecting a claim it then reinstated, and the customer would still have read the wrong number.

What the honest comparison returned#

All three arms ran the same 200 tasks on the same afternoon against pinned fixtures. The stale baseline is kept as a fourth column because it is the number the original deck was built on, and the gap between the two flat columns is the whole lesson.

Read the first row as two gaps rather than as four levels. The distance from the stale flat loop to the two-level tree is 17 points, and that is the number the deck reported. Inserting the back-ported column splits it. From 71% to 86% is 15 points, and every one of those points came from prompt and tool work that has nothing to do with having a supervisor: a return contract, an enumerated scope, a constraint list, an extracted-text tool wrapper. From 86% to 88% is the 2 points that remain once the arms are matched, and that is the largest quantity the experiment can offer as a topology effect. Seventeen points became two, and the split is additive only because the back-ported column exists to stand between them.

That decomposition licenses one conclusion and refuses another. It licenses attributing the bulk of the original claim to craft the flat design was always entitled to. It does not license calling the remaining 2 points a smaller win of the same kind, because 2 points is inside the noise of a 200-task set — the next section works out how far inside. And the additivity itself is an assumption: it holds only if the craft improvements and the topology change do not interact, which is plausible here and is not something the experiment tested.

One row in that table is not a result at all, and reading it as one is a mistake worth heading off. Tool calls per run come out at 96, 88, 91 and 94 across the four columns — a spread of a few calls across designs that differ in shape entirely. That flatness is the evidence that tool_budget did its job: all four arms did about the same amount of work in the world, so no column can be explained by one arm having simply looked at more pages than another. A row that varies tells you about the arms; this row tells you about the experiment, and had one arm come back at half again the calls of another, the rest of the table would have been uninterpretable.

Two rows separate the arms in ways the first row does not, and they point in opposite directions. Recovery is where the tree wins decisively, 78% against 41%, and the mechanism is mechanical rather than mysterious. The two-level tree runs the return-contract check at six segment boundaries, so a stale as_of on a worker return meets a check with three companies in scope, where an out-of-window date or an out-of-scope name is a comparison against a list of three. The flat loop has exactly one place to check, at the end, where the corrupted claim is one of about two hundred and the check is competing with the writing task for the model's attention. The gap of 37 points is the difference between six small checks and one large one, and it is available to any design that can arrange six small checks — a fact the last chapter of this topic turns into a result.

The recovery row also carries the chapter's own thesis a second time, in a place nobody looks for it. Recovery on the stale flat loop is 22%, and on the back-ported flat loop it is 41% — a large move produced by prompt and tool work with no supervisor anywhere in it. The back-port is what gave the flat loop a constraint list to check against, and a check it did not previously hold will catch defects it did not previously catch. So part of what the first comparison read as containment-by-hierarchy was containment-by-contract, and the tree's genuine share of the recovery gap is 78% against 41% rather than 78% against 22%. The same correction that shrank the quality claim shrinks the recovery claim, and it shrinks it less, which is why recovery survives as the tree's real case.

The tree loses on spend, running 33% more per successful run, and the third level loses on everything at once. Set the third column against the fourth and the three-level tree is worse on every row that moves: it costs 32% more than the two-level tree, it is 22 seconds slower at p95, and it recovers slightly worse, while completion quality is identical. A column that is beaten on every dimension by another column is not a trade-off with a case to argue; the extra summarisation step re-states claims often enough to occasionally launder a corrupted one past the contract check that would have caught it verbatim, so the depth is buying negative containment on top of the money.

The p95 row is the one place the tree's advantage is structural rather than contract-driven. The flat loop is a single agent working through eighteen companies one after another, so its wall-clock is the sum of its tool waits. Both trees fan out, so their wall-clock is bounded by the slowest branch plus coordination. That is a parallelism win, and parallelism is available to a flat orchestrator with workers as well — the design chapter 1 called Design A, where one orchestrator dispatches workers directly with no supervisor between. If latency is the reason for building a tree, the comparison that answers it is against the flat orchestrator rather than against the single loop, and this table does not contain that arm.

Two hundred tasks is fewer than it sounds#

How the arms were scheduled matters as much as what was in them. Tasks were run interleaved: task 1 through the flat arm, the two-level arm and the three-level arm back to back, then task 2. Running one arm to completion and then the next is the natural thing to do and it is the thing to avoid, because a provider-side latency excursion or a rate-limit window lands on whichever arm is running at the time and is indistinguishable from a topology effect afterwards. Interleaving spreads that noise across all three arms, which converts it from bias into variance — the numbers get noisier, and they stop being wrong in a direction. Each arm also ran with the same three seeds for the injection sampler, so the same tasks received the same corrupted claims in every topology.

The sample size is the limit worth internalising before anyone quotes a row of that table. At n = 200 with a base completion rate near 86%, the 95% interval on a single arm is about ±4.8 points, which is the range a repeat of the whole experiment would be expected to land in. The 95% interval around a difference of two points spans roughly ±4.7 points. An interval that wide swallows the difference whole, so on completion quality the flat loop and the two-level tree are indistinguishable on this workload, and the honest report of the two-point gap is that no difference was detected. Detecting a genuine 3-point difference at 80% power — power being the chance that an experiment of a given size notices a difference that is really there — would take roughly 2,700 tasks per arm. Nobody has 2,700 labelled tasks for this workload. Running the suite again until a favourable number appears is how a two-point gap becomes a slide.

There is a way to get more out of 200 tasks without collecting more, and it comes from the fact that both arms ran the same 200. Comparing two means throws that pairing away: it treats the arms as two independent samples and pays for the task-to-task variation twice, when most of that variation is a property of the task and cancels. Counting the tasks where the arms disagreed keeps it. Of the 200, 171 succeeded in both arms and 12 failed in both, leaving 17 discordant pairs — 9 where only the tree succeeded and 8 where only the loop did. The tasks both arms agreed on carry no information about which arm is better, and the whole question sits in the 17.

A near-even split of 9 against 8 is the same conclusion the interval gave, reached with the information the pairing preserves rather than in spite of throwing it away. What makes the paired form worth computing is the other outcome it could have produced: a split of 15 to 2 out of the same 17 pairs would have been a real signal at this sample size, because a lopsided disagreement is unlikely when two arms are equally good. The unpaired interval would have called that inconclusive as well. Pairing is therefore not a presentation choice; it is the difference between an experiment that can detect something and one that cannot.

One denominator choice in the table is also doing work and should not pass unnoticed. Tokens and cost are reported per successful run rather than per run, because an arm that fails cheaply and often looks efficient on the per-run measure — the failures divide the same spend across a larger count. On these numbers the two framings differ by about 9% for the three-level arm and 14% for the stale baseline. The two figures differ from each other because the failure rates differ and because failed runs do not cost what successful ones cost, and the per-successful-run form is the one that answers what a delivered brief costs.

The regression suite, the pinned fixtures, and one shape of defect#

A three-arm offline comparison is not a production A/B test and should not be described as one. The task set is the regression suite, which over-represents tasks that once broke and under-represents the mundane ones. A 2-point difference on that distribution can be a 5-point difference on real traffic in either direction, and nothing in the experiment says which.

The fixtures cut both ways. Pinning them is what makes the arms comparable, and it also means both arms see a filings endpoint that never rate-limits. Rate limiting is precisely the condition under which a tree's fan-out behaves worse than a sequential loop, since a fan-out issues its requests together and a sequence spreads them out. The experiment has removed the failure mode that most favours the arm it declared cheaper.

The injected defect is one shape: a plausible magnitude error carried with a stale date, chosen because it is the most common real one. The recovery column therefore measures resilience to that, and not resilience. A defect of a different shape — a claim that is correctly sourced and answers a neighbouring question — has no date to check and no magnitude to perturb, and there is no reason to expect the 78% to survive it.

The result also does not license "quality is a tie, so pick either". Parity on quality with a 33% cost gap is a loss for the more expensive arm, and the burden of proof sits with the design that spends more. The tree's case rests entirely on the recovery row, and that row is worth interrogating before it is banked: the containment happened at a contract check that ran at the segment boundary, and whether the check needed a supervisor standing there is a separate question from whether the check helped.

There is one confound the harness cannot eliminate, and it is worth naming rather than pretending away. A confound is a second difference between the arms that travels with the one being tested, so that any gap in the results could be attributed to either. The three topologies have genuinely different context profiles: the flat loop holds all eighteen companies' raw findings at once, and the trees hold three at a time. Equalising the tool budget equalises the work done in the world, and it cannot equalise how much material each arm's model is reasoning over — and that difference is the topology. Removing it would remove the thing being compared.

Every comparison of this kind is therefore a comparison of packages rather than of one isolated variable, which is a real limit on what the word "topology" can carry in a conclusion. The honest reading of the table is that on this workload, at this size, the package called "two-level tree" buys 37 points of recovery for 33% more tokens and roughly the same answer quality. That is a defensible trade, stated in the form a trade has to be stated in: a gain, a price, and the population both were measured over.

It says nothing at all about how deep the tree is allowed to get. The three-level column is already showing what happens past the point where the trade stops paying — more money, more latency, no quality, slightly worse containment — and a design that keeps adding levels will keep finding columns like it. Depth needs a cap, and the cap turns out not to be set by any of the numbers in this table.