One aggregate number hides everything after turn six

A multi-turn evaluation suite reports one headline number, and that number is end-to-end completion: the fraction of scripted scenarios in which the agent reached the goal its scenario file declared. Orbit — an internal travel-booking assistant whose multi-turn suite runs 40 scenarios of up to 12 turns each, driven against a scripted simulated user — prints it at the top of every run. Everything else in the report is measured per turn: a judge model reads each assistant turn against a written scale running from 0 to 4 and returns a rubric score, and the harness averages those scores across whichever scenarios are still running at that turn index. The two kinds of number are computed from exactly the same sessions, and which of them you look at decides which regressions you are able to see at all.

The protected-block change goes to review with one line of evidence: end-to-end completion 70 percent to 85 percent. The protected block is the never-summarised region of the prompt that carries every constraint the user stated and every commitment the agent made through the compaction step, so that a spend limit named at turn 2 is still in front of the model at turn 9. The first question is whether that is real, and the honest answer, computed from the same Wilson interval used in chapter 3 — an interval for a proportion, built so that it never runs off the ends of the scale when the counts are small — is uncomfortable. Twenty-eight of 40 gives an interval of 55 to 82 percent; 34 of 40 gives 71 to 93 percent. Those overlap across eleven points, so someone reading only the two rates and their intervals is entitled to say the change might have done nothing.

Two overlapping intervals on the same scenarios#

That reading is wrong, and the reason it is wrong is instructive. Independent intervals are the correct tool for two independent samples, and these are not independent samples — they are the same 40 scenarios run twice, which means most of the variance in each rate is scenario difficulty that cancels when you subtract.

The cancellation is worth writing out, because it is the entire argument. Every scenario carries its own difficulty: a one-leg booking with no policy question in it succeeds under almost any prompt, and one that holds a hotel, adds a colleague, changes a date and then confirms fails under many. Write a scenario's outcome in one arm as that difficulty, plus whatever the change under test contributes, plus the run's own noise. An arm's reported rate is the average of those outcomes across the suite, so it carries the average difficulty inside it, and the spread of difficulty across 40 scenarios is most of what makes each interval as wide as it is. Now subtract the two arms scenario by scenario rather than rate by rate. The difficulty term appears once in each outcome with the same value, because it is the same scenario both times, so subtraction removes it exactly and leaves the change's contribution plus the noise. Independent intervals are wide because they are still carrying a term the paired comparison has already cancelled.

The right analysis pairs the arms scenario by scenario, and doing that sorts the suite into four groups: of the 40 scenarios, twenty-five succeed in both arms, 9 succeed only with the protected block, 3 succeed only without it, and the remaining three fail in both. The twelve whose outcome differs between the arms are the discordant pairs, and they are where all the evidence about the change lives; a scenario that succeeds in both arms, or fails in both, is telling you about the scenario rather than about the protected block. That is the cancellation again, at the level of a single pair: a concordant scenario has a paired difference of zero whichever arm you subtract from which, and a zero cannot point towards either arm.

Those four outcomes are exhaustive by construction. Each arm returns a verdict for every scenario, and two verdicts can only agree on success, agree on failure, or disagree in one of two directions, so every scenario in the suite lands in exactly one cell and the four counts add back to 40. What that identity licenses is the thing that looks strangest about the test when you first meet it: the denominator is twelve rather than 40. Two of the four cells contribute nothing that can favour either arm, so the only evidence in the experiment is which way the other twelve fell, and the suite size has already done its job by determining how many discordant pairs you got.

What the identity does not license is reading the discordant count as a measure of how large the change is. A scenario lands in a concordant cell whenever it finishes on the same side of its own success threshold in both arms, and that happens both when the change did nothing to it and when the change helped it while it was already succeeding comfortably. A change that helps most of the suite a little, harms nothing, and pushes only a few scenarios across their thresholds produces few discordant pairs and a weak result. The test is sensitive to how close each scenario sits to its own threshold, and only through that to the size of the effect.

# Paired outcomes across arms, per scenario id.
improved, regressed = 9, 3           # discordant pairs
# Two-sided exact binomial on the discordant pairs:
#   p = 0.15   -> does not clear the usual bar at n = 40

That p-value is worth reproducing rather than taking on trust, because reproducing it is what shows how little twelve pairs can carry. Suppose the protected block does nothing. Then each discordant pair was equally likely to fall the improved way or the regressed way, and twelve such pairs have two-to-the-twelfth equally likely arrangements between them — four thousand and ninety-six of them. Count the arrangements at least as lopsided as the one observed. Nine improvements out of twelve can be arranged two hundred and twenty ways, ten improvements sixty-six ways, eleven improvements twelve ways, and twelve improvements exactly one way, which together come to two hundred and ninety-nine. A two-sided test also counts the mirror image, three improvements or fewer, and the symmetry of a fair coin makes that the same two hundred and ninety-nine. Five hundred and ninety-eight arrangements out of four thousand and ninety-six is 0.15 to two decimal places, which is the number the code above prints.

Even paired, the end-to-end rate cannot carry the claim at this suite size. Twelve scenarios changed their outcome and nine of them went the right way, which is the sort of split you get from a coin often enough that it proves little.

The per-turn rubric on the same runs#

Meanwhile the per-turn measurement on the same runs is unambiguous: the mean rubric at turn 8 moves from 2.94 to 3.49. To judge a move of that size you need to know how far the average itself would wander if the whole exercise were repeated, and that quantity is the standard error of the mean: the spread of the individual per-scenario differences, divided by the square root of how many scenarios each contributed one of. The square root is where extra scenarios earn their keep — averaging a set of numbers whose individual wobbles are unrelated leaves the average wobbling less than any one of them, by a factor of the square root of how many there are.

Put the chapter's own figures through that. The per-scenario paired differences at turn 8 have a standard deviation of 0.42 across the 36 scenarios still active at that turn, the rest having already ended. The square root of 36 is six, and 0.42 divided by six is 0.07, which is the paired standard error the report prints. The movement to be judged is 0.55, and 0.55 divided by 0.07 is nearly eight, so the effect is nearly eight times the amount the average would move on its own.

The same experiment, measured two ways, gives one answer that is a shrug and one answer that is decisive — and it is not because one of the analyses is cheating. Count what each has to work with. Forty sessions produce exactly 40 end-to-end outcomes, one per session, and no arithmetic can extract a forty-first. The same 40 sessions produce roughly 390 scored turns, because each session runs for several turns and every turn is scored. That is roughly ten times as many observations, and the square root above turns ten times the observations into roughly three times the precision. The turn-level estimator is also asking a narrower question — was this one turn a good turn — and a narrower question with far more data behind it is answerable at a resolution the broad question is not.

Most of your scenarios are not there for the ending#

The deeper problem with a single aggregate is compositional. Session length varies: Orbit's scenarios have a median length of 9 turns, and only 26 of 40 reach turn 12 at all. A change that damages behaviour only after turn 9 can touch at most the 30 sessions that get that far, and of those, most have already done the goal-critical work — by turn 10 a typical Lisbon session, the suite's script for booking a flight and a hotel for a workshop trip, is confirming and tidying up rather than deciding anything. Seven of the 40 scenarios have goal-critical work remaining after turn 9.

That is a chain of four steps and each one is checkable on its own. A session that has already ended cannot be damaged by anything, so the population exposed to a late regression is only the 30 sessions still running at that point. Damaging a turn changes the end-to-end outcome only if that turn was going to decide something, and in most of the exposed sessions the deciding is already done — a confirmation restated badly is a worse turn and still a completed booking. So the sessions a late regression can flip are the seven with goal-critical work left after turn 9, and no regression, however severe, can flip more than seven. End-to-end completion is then computed on a denominator of 40 whatever happens inside the sessions, so the movement it can show is however many of those seven flipped, divided by 40.

Put a severe regression through that chain. A regression that costs 1.2 rubric points on every turn from 10 onward — which is severe, the difference between a turn that advances the task and one that damages it — flips perhaps three of those seven to failure. Three fewer completions against the same denominator is a drop of three fortieths, which is 7.5 percentage points, so end-to-end completion moves from 70 to 62.5 percent, an interval of 47 to 76 percent that overlaps the baseline's almost entirely, on a change that is obvious to anyone who reads a late transcript. The same regression shows up in the per-turn series at turns 10 through 12 as a drop of 1.2 points against a run-to-run spread of 0.15, which is eight standard deviations and impossible to miss.

None of the rows dominates. The finding is in the first two rows read against each other rather than in the level of any row on its own. End-to-end completion is the only number that corresponds to something a user experiences, which is why it stays at the top of the report even though it is the least sensitive thing in it. The per-turn series is the sensitive instrument and it measures something no user directly feels; nobody has ever noticed that a turn scored 3.31 rather than 3.49. The gap between those two rows is the design problem in one line: the number that means something is the one that barely moves, and the number that moves is the one that means nothing on its own.

Believing that watching end-to-end completion is therefore a defence is the mistake this chapter is about, and it is tempting for a good reason rather than a lazy one. That number is the honest one: not a proxy and not an instrument reading, but the thing the suite exists to find out, with every other figure in the report an argument about whether to trust it. What stops it being a defence is the arithmetic two paragraphs up. The quantity moved 7.5 points, and stayed inside its own interval, across the whole distance between no late-session damage at all and damage severe enough to spoil every turn after the ninth. A quantity that holds nearly still while the dangerous property runs across its full range cannot tell you where in that range you are, and meaningfulness and sensitivity are separate properties of a measurement.

Reporting either row alone produces a failure, and the two failures are different. Completion alone gives a dashboard that never moves: it stays inside its interval through the regression above, the change ships, and the complaint arrives from production rather than from the suite. The per-turn series alone gives a dashboard that moves for reasons nobody cares about, because a rubric mean is not an outcome anyone asked for and a movement in it means nothing until it has been traced to a user-visible consequence. The lower two rows are in the table on a different argument again: both are free, and each catches a failure shape the top two rows are blind to.

What the report actually contains#

{
  "run_id": "orbit-2026-03-14-e-protected",
  "suite": {"scenarios": 40, "seeds": 3, "max_turns": 12,
            "scenario_set_hash": "a41f9c", "cassettes": "2026-03-02"},
  "instrument": {"rubric": "orbit-turn-quality-v3",
                 "judge_model": "vendor-mid-2026-02",
                 "user_model": "othervendor-mid-2026-01",
                 "human_calibration_kappa": 0.79,
                 "calibration_n": 60},
  "noise_floor": {"end_to_end_pp": 2.5, "per_turn_rubric": 0.12,
                  "from_identical_runs": 5},
  "end_to_end": {"completed": 34, "n": 40, "rate": 0.85,
                 "wilson95": [0.709, 0.929],
                 "vs_baseline": {"improved": 9, "regressed": 3,
                                 "exact_p_two_sided": 0.15}},
  "per_turn": [
    {"turn": 7, "n_active": 38, "rubric_mean": 3.55, "ci95": [3.41, 3.69],
     "adherence_pass": 0.96},
    {"turn": 8, "n_active": 36, "rubric_mean": 3.49, "ci95": [3.34, 3.64],
     "adherence_pass": 0.94, "paired_delta_vs_baseline": 0.55,
     "paired_se": 0.07},
    {"turn": 12, "n_active": 26, "rubric_mean": 3.31, "ci95": [3.09, 3.53],
     "adherence_pass": 0.92}
  ],
  "session_end_reason": {"goal_satisfied": 34, "max_turns": 4, "stall": 2},
  "efficiency": {"median_turns_to_completion": 9,
                 "median_tool_calls_per_session": 11,
                 "user_corrections_per_session": 0.8}
}

Every field in there is load-bearing. The n_active per turn is what stops someone reading the turn-12 mean as a statement about the suite; it is a statement about 26 scenarios, and its interval is correspondingly wide. The noise_floor block, computed from the five identical runs of chapter 5 — the same commit and the same cassettes run five times over, so that whatever the numbers do is the suite's own wobble — is what any claimed movement has to clear before it is discussed. The scenario_set_hash and cassettes date are there because a comparison against last month's number is invalid if either changed, and having them in the artifact makes that checkable rather than remembered. And human_calibration_kappa travels with the numbers so that a reader knows how much of this is a model agreeing with itself.

Comparisons across time need one more discipline, which is that the y-axis has a version. Rubric v3 and a hypothetical v4 with re-worded anchors produce numbers on different scales even though both run from 0 to 4, so a chart of per-turn quality across six months of releases is only valid within a rubric version. Drawing the two series on one axis assumes they came from the same instrument, and re-wording an anchor is precisely the act of changing the instrument: the same archived transcript can score differently under the two versions with the agent untouched. A step in the chart at the release where the rubric changed is then unattributable — it could be the agent, it could be the wording — and an unattributable step is worse than no chart at all, because a chart gets read.

When the rubric does change — and it will, because anchors that never change are usually anchors nobody is using — the way to preserve continuity is to re-score a stored sample of sessions under both versions and publish the offset, rather than to pretend the series is continuous. Re-scoring the same sessions holds the agent fixed and varies only the instrument, so whatever difference comes back belongs to the instrument, and that difference is what lets a number on one side of the seam be read against a number on the other. Orbit keeps 200 archived sessions for exactly this, and re-scoring them under a new rubric costs about $2.20.

The session-end reasons deserve a place at this level rather than in a debug log. A run where max_turns fires 4 times and stall fires twice is a different run from one where stall fires 12 times, even at identical completion rates, because stalls are a failure the completion number classifies as a plain miss.

How few scenarios makes the answer unknowable#

The practical resolution of a 40-scenario suite, at Orbit's observed discordance rates, is roughly this: end-to-end completion resolves changes of about 15 percentage points and larger; per-turn rubric at a given turn resolves about 0.3 points; per-turn adherence resolves about 8 percentage points. Below those, the honest answer to "did it help" is that the suite cannot tell.

Detecting a 5-point end-to-end change with reasonable power needs somewhere around 150 to 200 paired scenarios, given how few pairs are discordant. That is a real number with a real price attached, and it is the point at which the question stops being statistical and becomes a budget question. Two things are worth knowing before spending it. Adding scenarios that are near-duplicates of existing ones adds cost and very little resolution, because discordance is concentrated in the scenarios that stress something; 40 diverse scenarios resolve more than 120 variations on a theme. And a 5-point end-to-end change is often better detected by the per-turn instrument anyway, since a change that moves completion by 5 points usually moves some turn's rubric by considerably more than 0.3.

There is a way this reporting goes wrong that is worth naming, because it is the natural next step after building all of it. Once you have per-turn scores, adherence rates, efficiency metrics and completion, someone will want a single headline number and will propose weighting them into a Multi-Turn Quality Score. That number will move for uninterpretable reasons, it will be gamed within a quarter by whatever is cheapest to improve, and it will make the per-turn diagnosis that justified the whole harness inaccessible. Report the four numbers.

Two of those three predictions come out of one property of any weighted blend. A weighted blend is a many-to-one map: many different combinations of the four inputs land on the same output, which is what averaging is for rather than a defect of the weights. So the map cannot be run backwards. A movement in the blend has several possible causes behind it and carries no record of which one it was, and recovering the component that moved is exactly the diagnosis the harness exists to provide.

The third prediction, the gaming, follows from the same map one step further on. If the blend is what gets reported, raising it is the work, and the four components do not cost the same to raise. A per-turn adherence rate rises when somebody writes a firmer constraint into the system prompt; end-to-end completion rises only when the agent gets better at booking travel. The cheapest input moves, the headline moves with it, and nothing a user experiences has changed.

A second failure of reporting is subtler and happens at the boundary between the suite and the people who use its output. Completion rates get quoted in planning documents without their scenario count, and by the third document the sentence has become "the agent completes 85 percent of conversations", which is a claim about production made from 40 scripted sessions. The mechanism is that a percentage is a ratio with the denominator already divided out, so the form of the number carries no record of what it was computed over, and a sentence in prose carries no record either, because a sentence gets copied on its own while the paragraph around it stays behind. Those two facts together are what turns a suite measurement into a production claim in three hops with nobody lying at any of them.

The defence is mechanical: never publish a rate without its denominator in the same sentence, and prefer "34 of 40 scripted scenarios" to "85 percent" in any prose that will be read by someone who did not build the suite. The percentage is the right form inside the artifact, where the count is one field away, and the wrong form in a paragraph that will be copied. The two forms differ because the unit that travels differs: the artifact keeps completed and n in the same object as rate, so the denominator moves with the number whatever anyone does with the file, while the unit that travels out of a document is one sentence, which is why the denominator has to be inside it.

The intervals in the report also describe the suite and not production. Forty scenarios written from sampled transcripts are a better sample than 40 invented ones, and they remain a fixed set, so a 95 percent interval on end-to-end completion is a statement about how the suite would behave if re-run rather than a forecast of next month's incident count. The p-value on the discordant pairs carries the same boundary: both numbers are honest about the scenarios somebody sat down and wrote, and silent about the sessions nobody thought of.

Which returns to the size question with a price on it. Going from 40 scenarios to 160 to resolve five-point changes is a four-fold increase in every cost this suite has, and there are at least four of them, only one of which appears on the model bill. The alternative is not a cheaper suite but a different reading of the same one: keep end-to-end completion at the top of the report as the number that means something, keep its denominator and its interval welded to it, and put the questions it cannot resolve to the per-turn series, which is measuring the same sessions ten times over.