Runs that produce nothing belong inside the quality number

Scout is an internal competitive-research assistant built on the orchestrator–worker arrangement, in which a parent decides what pieces of research a goal implies, dispatches them to workers running in isolation, and folds what comes back into one answer. Over the preceding chapters it acquired a validated decomposition, a coverage field, merge rules attached to predicates, grounded claims and a recorded selection step. Each of those cost tokens and latency. The question that has been deferred since the first design review is whether the assembled thing beats one agent given a large budget, and the evaluation that was supposed to settle it went to a production-approval meeting.

Two terms decide how that evaluation reads. An arm is one of the two systems under comparison, each run over the same task set. Delivered quality is the expected score of asking a system a question, taken over every question you ask it rather than over the subset it happened to complete. The gap between those two populations is where this chapter lives.

Restating the comparison so the two arms are the same object#

The slide asking for production approval had four numbers on it. On a 200-task evaluation set, Scout scored 71.4 against the single agent it would replace at 65.2, both graded blind — the reviewers not being told which system had produced the answer in front of them — on the same 100-point rubric by the same three reviewers. It used 3.4 times the tokens. Its p95 latency was 2.8 times higher, 73 seconds against 26. And 4% of runs failed outright with coordination errors: a worker returning output that would not parse after its retry, a result that arrived after the run had already been assembled and was discarded, a synthesis call that never completed. The recommendation on the slide was to approve the six-point gain and open a ticket for the 4%.

Three of those four numbers describe only the runs where the system worked, and the arithmetic that shows it is one line. Four percent of 200 tasks is 8 runs, so the runs that produced an answer are 200 minus 8, which is 192. The 71.4 is the mean score over those 192. The token figure and the latency figure were computed over the same 192. The fourth number is the population that was excluded from the other three.

It was excluded by the ordinary mechanics of an evaluation harness, which scores what it is given and cannot score a run that returned nothing. Nobody chose to hide it. The harness dropped the failures because there was no score to record, and the summary was assembled from what the harness emitted. That is worth dwelling on, because it is the mechanism rather than the motive: a failed run produces no row, a missing row contributes nothing to a mean, and a mean over the rows that exist is a mean over the successes. No step in that chain requires anybody to want a favourable number.

A run that fails produces no answer, and from the position of whoever asked the question, no answer is a worse outcome than a mediocre one. Making the comparison about delivered quality requires assigning a score to a failed run, and the defensible assignment is zero, on the grounds that the user received nothing. Doing this to both arms matters, because a single agent has its own failure rate and it is not zero. Scout's baseline failed on 0.7% of the 200 tasks, mostly on tool errors that exhausted its retry budget.

The restatement is then a weighted mean over two outcomes, one of which contributes nothing. If a fraction of runs succeed and score some mean, and the rest score zero, the overall mean is the success fraction times the mean of the successes. Scout's completing share is 0.96, so its delivered quality is 0.96 × 71.4, which is 68.5. The single agent's completing share is 0.993, so its delivered quality is 0.993 × 65.2, which is 64.7. Subtracting gives a gap of 3.8 points where the slide showed 6.2.

Two of those restatements are worth spelling out, because they move in a direction people do not expect. The token multiplier rose rather than fell. A failed run still spends: a synthesis that never completes has already paid for five workers, so excluding failures from the token average removes the most expensive runs from it and understates the bill. Work the delivered-answer row from the mean-over-all-runs row and the mechanism is visible in the arithmetic. Total tokens are the mean over all runs times the number of runs, 34,600 across 200, and dividing that total by the 192 answers actually delivered gives 36,000 per answer. The same division for the single agent, 9,900 across 200 spread over 199 answers, gives 9,970. Dividing each cost per delivered answer by its token count gives the same blended rate for both arms, which is the check that the two were priced the same way.

The latency figure moved further, from 2.8x to 4.4x, and 118 divided by 27 is where that multiplier comes from. Failures do not resolve quickly. Most of them hit the run-level deadline of 180 seconds, so they sit at the extreme of the distribution where a p95 is computed. Dropping them is precisely the exclusion that a tail metric cannot survive: a percentile is a statement about the ordering of the whole population, and removing the slowest members changes which run sits at the ninety-fifth position.

The mean-of-means is also the wrong summary for a 200-task set where the two arms were run on the same tasks. Paired scoring is available and more informative: the orchestrator scored higher on 88 tasks, lower on 61, and within a point on 51, which accounts for all 200. Those first two counts as shares of the set are 44% of tasks won and 31% lost. A design that wins on 44% of tasks and loses on 31% is a different proposition from one that is uniformly 6 points better, and the aggregate hides that entirely.

def delivered_quality(runs: list[EvalRun], failure_score: float = 0.0) -> Summary:
    scores = [failure_score if r.outcome != "answered" else r.score for r in runs]
    tokens = [r.tokens for r in runs]                  # including failed runs
    latencies = [r.latency_s if r.outcome != "timeout" else RUN_DEADLINE_S
                 for r in runs]
    delivered = sum(1 for r in runs if r.outcome == "answered")
    return Summary(
        mean_score=mean(scores),
        tokens_per_delivered=sum(tokens) / max(delivered, 1),
        p95_latency=quantile(latencies, 0.95),
        failure_rate=1 - delivered / len(runs),
    )

def paired_gap(a: list[EvalRun], b: list[EvalRun]) -> tuple[float, float]:
    by_task_a = {r.task_id: (0.0 if r.outcome != "answered" else r.score) for r in a}
    by_task_b = {r.task_id: (0.0 if r.outcome != "answered" else r.score) for r in b}
    diffs = [by_task_a[t] - by_task_b[t] for t in by_task_a.keys() & by_task_b.keys()]
    return mean(diffs), stdev(diffs) / sqrt(len(diffs))   # gap, standard error

Every list comprehension in delivered_quality runs over runs rather than over a filtered subset, and that is the whole content of the function. scores substitutes a zero instead of dropping the row. tokens takes every run's spend. latencies substitutes the deadline for a run that timed out, so the run keeps its place at the top of the ordering rather than vanishing from it. Only delivered counts answers, and it appears as a denominator rather than as a filter.

The standard error is the part that decides how much of an argument the remaining 3.8 points can carry. It is how far the measured gap would wander if you re-ran the whole evaluation on a fresh set of 200 tasks, and paired_gap computes it in the last line as the spread of the per-task differences divided by the square root of how many there are. Dividing by a square root is what makes averaging work: individual differences vary a great deal and their average varies much less, by the square root of the count. The per-task difference had a standard deviation of about 15 points on the 100-point rubric, and 15 divided by the square root of 200 gives a standard error near 1.1. A 3.8-point gap is roughly three and a half standard errors, so it is real; a gap of one point would not have been, and teams routinely present one-point gaps as findings because the harness reports a mean without one.

The experiment that decides it#

Everything above is arithmetic on numbers already collected. The measurement that would most change the decision has not been taken, and it is the obvious one once the token column is in view: the orchestrator was given 3.4 times the budget of the thing it is being compared against, and nobody has checked what the single agent does with the same money.

This is a comparison against the wrong reference, and it is worth stating as a general shape because it recurs. Two systems are being compared on quality while differing on two variables at once — topology and spend. Any gap can then be attributed to either, and the one that gets the credit is the one the team set out to test.

There is published reason to expect spend to matter a great deal. Anthropic's How we built our multi-agent research system reports that token usage on its own explained about 80% of the variance in performance — the spread between the better results and the worse ones — on their browsing evaluation, and that their multi-agent system used roughly fifteen times the tokens of a chat interaction. Simon Willison's notes on that piece pull the multiplier out as the number to watch, for the reason it is being used here: if spend is the dominant explanatory variable, then any architecture that spends more will look better than one that spends less, and the architecture is not what is being measured.

Budget parity for a single agent can be bought in several ways, and the honest version tries the one most likely to win rather than a strawman. Best-of-n sampling with a selection pass is the usual strongest baseline: run the single agent three times with sampling temperature raised above zero, so that the three runs take different paths instead of repeating one another, then have a fourth call pick the best answer against the goal. Alternatives are a longer sequential loop with a raised tool-call cap, or one pass followed by a self-critique and revision round. Scout's team ran best-of-3 with selection, which came to 3.1 times the original single-agent budget — 30,700 tokens against 9,900 — slightly under the orchestrator's 3.4, and close enough for the comparison.

Read that table by column rather than by row. The best-of-3 column ties the orchestrator on the row that matters — delivered quality, both at 68.5, the single agent's 69.1 discounted by its 0.9% failure rate — and beats it on every other row. Fewer failures, fewer tokens, less than half the tail latency. A column that ties on the metric under test and wins on all the rest is a dominated comparison, and the design being dominated is the more expensive one.

At equal budget the gap that justified the project disappeared, and the orchestrator was still nearly three times slower at the tail, because best-of-3 runs its samples concurrently while the orchestrator pays a planning round, a worker round and a synthesis round in series. That result is specific to this workload and this set, and running the experiment is what makes it a result rather than an assumption.

The segmented view is where the design earned something back. Chapter 3's writability test splits the task set into goals whose sub-questions can be enumerated before any research happens and goals whose later questions depend on earlier findings, and the two halves behave differently enough that the aggregate was hiding both.

The two segment sizes add to the whole set, 121 and 79 making 200, so nothing has been filtered out to produce this view. The orchestrator is worth roughly five points on the breadth-shaped goals and costs roughly three on the rest, which turns the approval question from a yes-or-no into a routing question: classify the goal, send the enumerable ones through the fan-out, send the rest through the sequential path. That is more machinery than a single approval, and it is what the data supports, whereas approving the orchestrator for everything would have shipped a regression on 40% of the traffic while the headline number said it was better.

The 4% is arithmetic, not a bug someone forgot to fix#

The team's response to the failure rate was that it is an implementation problem, and there is a real sense in which every one of those failures is a specific defect with a specific fix. What makes the argument insufficient is the composition, and the composition is a two-line derivation worth doing rather than asserting.

A run succeeds only if all five of its workers do. If each worker fails independently with some probability, the chance one of them works is one minus that probability, and the chance all five work is that quantity raised to the fifth power, because independent events multiply. Five workers that each fail 0.8% of the time give a run-level failure of 1 − 0.992⁵ = 3.9%, which is the observed number to within measurement noise. The 4% is not one broken component; it is the arithmetic of requiring five things to work.

Run the same identity backwards to price the fix. Reaching 1% at the run level means driving each worker to 0.2% — a fourfold reliability improvement on five separate components, plus the synthesis stage and the correlation logic, none of which has anything obviously wrong with it. That is what the exponent does to a reliability target: improvements at the component level arrive at the run level divided among however many components have to hold.

The residual on that identity is the independence assumption, and it is worth naming because the next topic breaks it. The fifth power is only the right composition if one worker failing tells you nothing about whether another does. A shared cause — one bad premise handed to all five, a rate limit that hits the whole fan-out at once — puts the failures on top of each other instead of spreading them, and the arithmetic above then understates the clustering while possibly overstating the total.

The productive path is the one chapter 4 already built. Coverage representation converts a class of outright failure into a scored degraded answer, because a run that lost one worker and says so in its coverage field has still returned something a reader can use. Applied to Scout's 4%, 2.6 percentage points were representable this way — timeouts and invalid outputs on individual workers — and 1.4 points were genuinely no-answer, being lost results and synthesis calls that never completed. Those two shares add back to the 4%, which is the check that the split is exhaustive.

The restated quality figure is then an expectation over three disjoint populations rather than two, and each term is a share of runs times the score that share earns. Complete runs are the 0.96 share and score 71.4. Degraded runs are the 0.026 share and score 44, which is the mean rubric score reviewers gave degraded answers that declared what they were missing. Genuine no-answers are the 0.014 share and score zero. Adding the three products, 0.96 × 71.4 + 0.026 × 44 + 0.014 × 0 = 69.7.

What that identity licenses is narrow and real: scoring degraded answers rather than zeroing them is worth the difference between 68.5 and 69.7, and it was bought by engineering rather than by an argument about what the numbers should be adjusted to mean. What it leaves unsettled is the comparison itself. That puts the orchestrator at 69.7 against the budget-matched single agent's 68.5, a gap of about one standard error of the paired difference, which is to say a gap that has not been established. The general form is worth carrying: when a failure can be made partial and legible it stops being a zero, and when it cannot, it stays a zero and belongs in the mean.

An evaluation run once is a fact about one quarter#

A budget-matched baseline is a strong test and it is not a permanent verdict. Both arms sit on top of a model, and each model release moves the single agent's curve as much as the orchestrator's — sometimes more, since a longer context window and better long-horizon tool use are exactly the capabilities that a fan-out was compensating for. The practical arrangement is to re-run the paired comparison whenever the underlying model changes, which is cheap once the harness exists and is the first thing that gets skipped.

The rubric also does not price operations. Five concurrent workers per request multiply rate-limit pressure, retry storms, queue depth and the number of things that page someone at three in the morning, and none of that appears in a quality score. Neither does the cost of the machinery this topic has accumulated: a decomposition validator, a coverage policy, merge rules per predicate, a document cache with span checking, a selection trace and a labelled reference set. Every one of those exists to hold a specific failure closed, and together they are a system with more moving parts than the thing it replaced, maintained by people who will not all be the people who built it.

There is a further limit on the zero, and it is worth being honest about. Scoring a failed run as zero is defensible from the reader's position and it is still a choice. A reader who gets nothing and retries has lost time rather than everything, and a reader who gets nothing on a question they could have answered another way has lost less again. The zero is the conservative end of a range, and the reason to use it is that the alternative — leaving the run out — is the extreme other end, where a failure is scored as though it never happened.

For a task that has not been built yet, the order that wastes the least is the reverse of the order this topic presented. Start with the writability test from chapter 3 on thirty real goals, because an afternoon with a whiteboard tells you whether a fan-out can serve your traffic at all. Then build the single agent and measure it at the budget you were prepared to give the orchestrator, since that is the number the orchestrator has to beat and it takes days rather than weeks. Build the fan-out only for the segment where the parity baseline leaves a gap, and build it with coverage represented and selection recorded from the first commit — both are cheap to add early and, as chapters 4 and 7 showed, invisible failures that ran for two months and six weeks respectively when they were left until later.

Where the gap survives budget parity, the pattern is doing something real, and the systems that report the largest wins are the ones whose goals fan out widest: many independent sources, each cheap to consult, with an answer that depends on breadth rather than on a chain of inferences. That is the shape to look for in your own workload before writing any of this, and it is the same shape the writability test was measuring, arrived at from the cost side instead.