One trial per task cannot separate four points from noise

The comparison landed in the design doc as a four-row table with a recommendation underneath it. Two architectures for the same contract-review task — a single tool-using loop, and a three-agent pipeline that extracts clauses, normalises them against the customer's policy list, and drafts the redline — had each been run across a fixture set of 150 real customer documents. The loop scored 84%, the pipeline 88%, and a line beneath the table noted that the pipeline's fastest observed run finished in 4.1 seconds where the loop's fastest took 5.2. The recommendation was to adopt the pipeline before the next release cut, and nobody in the thread argued with the arithmetic. The arithmetic is fine: 126 successes against 132 is four percentage points, and 150 fixtures is a set that took two engineers a fortnight to label and score.

That table is an architecture evaluation, which is what this topic is about: taking one task, running it through two or more candidate topologies — a topology being the shape of the system, meaning how many model-driven components it has, which of them calls which, and what passes between them — and comparing them on measured success, cost and latency rather than on how each one reads in a design document. The two shapes in the doc are the pair you will meet most often. A single loop is one model with a set of tools that keeps calling them until it judges the job done. A multi-agent pipeline splits that work into several such loops, each with a narrower brief and its own prompt, arranged so that one hands its output to the next. Three pieces of vocabulary come with the practice and recur through every chapter here. Each candidate under test is an arm, a word taken from clinical trials, where two groups receive different treatments and everything else about them is held the same. The tasks both arms are run against are the fixture set, documents stored with known-good outcomes so that a run can be scored without a person reading it. The scoring is done by the rubric from topic 10.9, the fixed list of criteria that turns an output into a pass or a fail.

The reason to run a comparison of this kind is that the alternative is choosing an architecture by how convincing it sounds. Decomposition reads well on a whiteboard, because separation of concerns is a genuine virtue in ordinary software and the argument transfers without anyone checking whether it survives each component being a language model that can misread its input. Topic 11.17 set out the ways a multi-agent design can cost more than it returns, and a matched comparison is how a team finds out whether its own does, on its own workload, while the design is still cheap to change. The occasions are usually one of three: someone proposes splitting a working loop into stages, someone questions a pipeline that is already serving traffic, or the choice comes up for re-checking after models and prompts have moved. The output is a table close to the one above, carrying success per arm, spend per task, and latency reported as a p95 — the value 95% of runs come in under — with a recommendation attached.

What is worth arguing with is what a single cell of that table contains. Each of those 300 numbers is one execution of one document against a system that samples tokens, calls tools whose responses arrive in whatever order the network delivers them, and takes a different branch when a retrieval returns four snippets instead of three. Run the same document through the same loop twice and you get two draws from a distribution, not one measurement repeated. The table treats each draw as if it were the task's true score, which is the assumption everything downstream rests on.

Four points, and what one draw of a task can hold#

Treat each task as a coin whose bias is the system's chance of succeeding on that document, and the numbers become easier to reason about. At an overall success rate near 86%, the standard error of a single arm's rate — the typical distance between the rate you measured and the rate the system really has, arising from which particular tasks and which particular draws landed in the run — over 150 tasks is sqrt(0.86 × 0.14 / 150), or 2.83 percentage points. The standard error of the difference between two independently run arms is larger by a factor of √2, so about 4.0 points. The observed gap is 4 points. It is one standard error, which is the region where a fair coin spends most of its time, and a two-sided test on those figures returns roughly p = 0.32.

The more useful way to feel the size of that noise is to ask how much two identical systems would disagree. If both arms have exactly the same per-task success probability of 0.86, then on any given task the chance that one succeeds and the other fails is 2 × 0.86 × 0.14, about 24%. Over 150 tasks, two copies of the same architecture will produce roughly 36 tasks where their results differ, and the four-point gap in the doc is what happens when 21 of those disagreements fall one way and 15 the other. Nothing about that outcome requires a difference in design.

The obvious repair is more fixtures, and it does work, at a price. To resolve a four-point gap with a 95% interval that excludes zero — an interval being the range the true gap sits in on 19 runs out of 20, so one that excludes zero is one where the arms are unlikely to be equal — you need the difference's standard error under about 2.0 points, which means roughly 580 tasks per arm rather than 150. That is four times the labelling effort for a set that already cost a fortnight, and it buys precision on exactly one question while leaving a second one untouched: with one trial per task you still cannot tell a task that fails every time from a task that fails one time in five. Those two failures have different causes and different fixes, and the fixture set cannot distinguish them no matter how large it grows.

What repeated trials buy, and what they cost#

Running each task k times against each arm changes the unit of analysis from a coin flip to a per-task success rate, and it lets the comparison be paired. Both arms see the same documents, so the variance that comes from some contracts simply being harder than others cancels out of the difference instead of being estimated through it. On this fixture set, per-task difficulty is by far the largest source of spread — a handful of five-party amendments fail under both architectures, and a long tail of two-page NDAs succeed under both — so removing it from the comparison is where most of the statistical power comes from.

The two token figures in each row are the loop and the pipeline, priced at the arms' measured per-task cost of $0.087 and $0.180. Five trials over the existing 150 fixtures costs $200 of model spend across both arms and about 40 minutes of wall-clock with 20-way concurrency, which is less than a single engineer-day and considerably less than labelling 430 more documents. The interval it produces is tighter than the one 580 fresh fixtures would give, because the pairing is doing work that extra unpaired tasks cannot do.

Re-running the original comparison at five trials per task produced a different picture, and the difference is mostly in what it refuses to claim.

The gap survived, smaller, with an interval that still touches zero. What the original table could not have shown at all is the mixed row: about a sixth of the fixture set succeeds sometimes and fails sometimes under either architecture, which means a single run of the full set has roughly 25 coin flips in it before either arm has demonstrated anything. That figure is the honest measure of how much resolution one run has, and it is the number to quote when someone proposes deciding on a single pass.

The fastest run is the number least connected to anyone's experience#

The 4.1-second minimum is a real observation and it describes a document that was short, whose three stages each hit a warm cache, and whose model calls all returned first-token quickly. Users do not experience minima. They experience whatever their own document does, and for a pipeline they experience the sum of three stages, each with its own tail. The pipeline's p95 is 23.8 seconds against the loop's 14.2, and the mechanism is arithmetic rather than mysterious: the total is a sum of three stage latencies, so a run only needs one stage to land in its own tail for the whole run to land in the total's tail. Three chances at a bad draw per request compounds into a tail that is worse than any single stage suggests, while the minimum improves because the pipeline's stages are individually shorter than the loop's full trajectory.

Topic 10.9 established percentile reporting for cost. Latency wants the same treatment for the same reason, and it wants one extra column when the arms have different shapes, because the pipeline's p95 decomposes into per-stage contributions that the loop has no equivalent of. Reporting min alongside a success rate invites the reader to combine the best number from one distribution with the average of another, which is how a design gets adopted on the strength of a run nobody will ever see again.

Running it so the trials are actually matched#

The runner is not complicated, and the parts that matter are the ones that keep the two arms comparable: the same task list, the same trial indices, the same fixture revision, and per-run capture of tokens and wall-clock rather than a single aggregate at the end.

@dataclass
class Run:
    arm: str            # "loop" | "pipeline"
    task_id: str
    trial: int
    success: bool
    input_tokens: int
    output_tokens: int
    tool_calls: int
    latency_ms: int
    trace_id: str

async def compare(arms: dict[str, Agent], tasks: list[Task], k: int = 5) -> list[Run]:
    jobs = [(name, task, trial)
            for name, _ in arms.items()
            for task in tasks
            for trial in range(k)]
    random.shuffle(jobs)                 # interleave arms across the run window
    sem = asyncio.Semaphore(20)

    async def one(name: str, task: Task, trial: int) -> Run:
        async with sem:
            t0 = time.monotonic()
            result = await arms[name].run(task.input)
            return Run(
                arm=name, task_id=task.id, trial=trial,
                success=score(task, result),          # the rubric from topic 10.9
                input_tokens=result.usage.input_tokens,
                output_tokens=result.usage.output_tokens,
                tool_calls=len(result.tool_calls),
                latency_ms=int((time.monotonic() - t0) * 1000),
                trace_id=result.trace_id,
            )

    return await asyncio.gather(*(one(*j) for j in jobs))

Shuffling the job list matters more than it looks. If you run all 750 loop trials first and all 750 pipeline trials afterwards, any provider-side slowdown, rate-limit backoff or regional failover during the second half is charged entirely to one arm, and it will show up as a latency difference that has nothing to do with topology. Interleaving spreads that noise across both arms, which is the same reason clinical trials randomise the order of treatment rather than dosing one group on Monday and the other on Friday.

The paired gap then comes from per-task rates rather than pooled counts, so that a task both arms fail contributes nothing to the comparison instead of dragging both means down and shrinking the apparent difference:

def paired_gap(runs: list[Run]) -> tuple[float, float, float]:
    rate = defaultdict(dict)
    for r in runs:
        rate[r.task_id].setdefault(r.arm, []).append(r.success)
    diffs = [mean(v["pipeline"]) - mean(v["loop"]) for v in rate.values()]
    m = mean(diffs)
    half = 1.96 * stdev(diffs) / math.sqrt(len(diffs))
    return m, m - half, m + half

What repetition does not fix#

Repeated trials narrow the interval around a difference and say nothing about whether the difference is worth having. A pipeline that wins by a statistically clean 2.2 points while costing 2.1× the tokens and 1.7× the p95 latency is still a trade the product may not want, and the arithmetic that settles that question is a budget rather than a confidence interval. Nor does repetition repair a fixture set drawn only from tidy two-party NDAs, or a rubric that scores formatting more strictly than substance — errors in what you are measuring are unaffected by measuring it more often, and they will be reproduced precisely.

Setting temperature to zero does not remove the need for trials either, which surprises people who expect determinism. Tool responses vary, retrieval returns different neighbours as an index is rebuilt, timeouts fire on one run and not the next, and providers batch requests in ways that make token-level determinism a courtesy rather than a guarantee. The pipeline in this comparison ran at temperature 0 throughout and still produced 24 tasks with mixed outcomes across five trials, so the variance being measured here is a property of the system rather than of a sampling parameter.

The result now has an interval, per-task resolution, a cost column and a tail statistic, which is enough to argue about. It is enough to argue about on the day it was run, and that turns out to be a stronger caveat than it sounds, because nothing in the runner above records which prompts, which model build or which rubric produced those 1,500 runs. Three months later that omission is what makes the same comparison come back with the opposite answer.