Compaction fires at turn 8 and the score drops: prove causation

Compaction is the step where an agent's accumulated turns are replaced by a shorter summary once the context crosses a token threshold, so that the prompt stops growing with the conversation. In Orbit, an internal travel-booking assistant, compaction fires at turn 8 and the mean rubric score a judge gives a turn falls at turn 8 — the cliff a per-turn plot shows (chapter 5, where the score is drawn against turn index instead of averaged across a session). The proposal in the channel was one line: compaction fires at turn 8, the score drops at turn 8, turn compaction off. It is the reasonable first reaction and it is wrong on both halves. The evidence is a single coincident boundary in observational data, where a session at the compaction threshold is also the longest, most tool-heavy, most-corrected session anyone has had — every one of which is an alternative explanation. And the remedy is a policy change whose consequences nobody has measured, applied on the strength of that evidence.

What settles it is an experiment with the same structure as any other controlled experiment: hold the scenarios, seeds, cassettes, agent prompt, model version, judge and rubric identical, and vary exactly one thing. A seed fixes the random choices the simulated user makes, a cassette is a recorded tool response replayed in place of a live call, and the judge is a model scoring each turn against a frozen rubric. Each configuration being compared is an arm, the term trials use for one of the variants a subject can be assigned to, except that here the subjects are scenarios and every scenario is run through every arm rather than allocated to one of them.

Five policies differing in one field#

ARMS = {
    "A_baseline":   CompactionPolicy(threshold_tokens=12_000, protect=[]),
    "B_off":        CompactionPolicy(enabled=False),
    "C_early":      CompactionPolicy(threshold_tokens=8_000,  protect=[]),
    "D_late":       CompactionPolicy(threshold_tokens=20_000, protect=[]),
    "E_protected":  CompactionPolicy(threshold_tokens=12_000,
                                     protect=["constraints", "commitments"]),
}

for arm, policy in ARMS.items():
    for seed in (11, 12, 13):
        run_suite(scenarios=SCENARIOS_40, policy=policy, seed=seed,
                  judge="orbit-turn-quality-v3", cassettes="2026-03-02")

Every entry in that dictionary differs from A_baseline in exactly one field. B_off disables compaction, C_early and D_late keep the empty protect list and move only the token count at which compaction fires, and E_protected keeps A's threshold of 12,000 tokens and changes only what compaction is allowed to touch. A comparison between any of them and A therefore contains one candidate explanation, which is the property the observational evidence lacked.

Five arms times three seeds times forty scenarios is 600 sessions, about $540 of model spend and close to eight hours of wall clock, which is an overnight job rather than something you wait for. That is the price of the answer, and it is worth stating next to the answer because the alternative — shipping a compaction change on a hunch and watching production for a fortnight — costs more and resolves less.

Spread over 600 sessions, that $540 is ninety cents a session, the same per-session figure the suite's ordinary nightly run works out to (chapter 8, which prices a suite as scenarios times turns times model calls). No session gets dearer for being part of an ablation. The ablation is the nightly run performed fifteen times over, once per arm per seed, and that fifteen is the whole surcharge.

The diamond in that diagram has a real second branch: if the drop stays at turn 8 while the threshold moves, compaction is not the mechanism and the search starts again. Arm E is absent because it is the change considered after the test answers.

One scheduling detail matters more than it sounds. Running arm A on Monday and arm B on Thursday confounds the policy with anything the provider changed in between, and hosted models do change under a stable model id — routing, batching, occasionally a silent revision. Orbit's runner interleaves: for each scenario and seed it executes all five arms back to back before moving on, so a provider-side shift lands on every arm roughly equally instead of loading onto one. The same argument applies to the judge, which is why every arm is scored by the same rubric version in the same run rather than compared against numbers recorded a fortnight earlier.

Interleaving is a decision; the requirement it answers is that no difference between arms may be explicable by anything except the policy. A single batch inside one hour would meet that requirement equally well. What the requirement rules out is the schedule giving each arm a day of its own, and scoring every arm with one rubric version in one run answers it on the judge's side.

A and B: what summarisation is buying and costing#

Read that table as differences rather than levels. That arm A completes 70 percent of the suite is a property of the suite as much as of the agent; the gap between the columns is what the experiment produced, because both columns were computed on the same scenarios. The latency rows measure two different things: p50 latency per turn is the median, the figure half of all turns come in under, and every turn pays it, while the 11.2 s is the one turn on which compaction runs and appears in a single column because arm B never performs the step.

The completion gap of 18 points on 40 paired scenarios is the headline, and it is worth checking properly rather than eyeballing: 9 scenarios succeed in B and fail in A, 2 fail in B and succeed in A, which under a one-sided exact binomial test on those 11 discordant pairs gives p = 0.033.

Most of the suite is absent from that calculation. Every scenario ran through both arms, so each falls into one of four buckets: succeeds under both, fails under both, succeeds only under B, succeeds only under A. The first two are concordant, and a concordant scenario carries no information about which policy is better, because it would have produced the same pair of outcomes with the arm labels swapped. Forty scenarios less the 11 that changed outcome leaves twenty-nine that did not, so all the evidence sits in the 11.

Now suppose the policy makes no difference. Then a discordant scenario changed outcome for some reason unconnected to the policy, so it was as likely to fall on A's side as on B's: one fair coin flip each, eleven flips. Eleven flips have two possibilities for the first, doubled for the second and again for the third and so on through all eleven — a little over two thousand arrangements, every one exactly as probable as every other, which is what lets the question be settled by counting.

So count the arrangements at least as lopsided as the observed one. Nine or more of the 11 falling B's way covers three cases: exactly nine leaves two on A's side, in fifty-five ways of choosing which two; exactly ten leaves one, in eleven ways; exactly eleven leaves none, in a single way. Fifty-five plus eleven plus one is sixty-seven arrangements, and sixty-seven divided by that little-over-two-thousand total is 0.033. So p = 0.033 is that frequency, about one run in thirty, which is the sense in which the gap is unlikely to be an accident of which scenarios happened to be in the suite.

Two things the frequency does not license are worth stating in the same breath. It says nothing about the size of the effect: an identical p-value would come from a policy that flips one scenario in a hundred, provided the flips it does cause nearly all go one way. And it is computed over a fixed set of authored scenarios, so it is an inference about this suite rather than about real conversations. What it does do is clear the bar at this suite size, which is not something the 30-scenario comparison in chapter 3 (87 percent against 80 percent, with the confidence intervals overlapping across nearly their whole width) could have said about a gap half as large.

The paired test exists at all only because every scenario goes through every arm. The two columns are not two samples of scenarios; they are the same forty scenarios twice, each acting as its own control. That licenses subtracting per scenario instead of comparing two rates, and it takes scenario difficulty out of every comparison in this chapter: a scenario that is hard because of how it was written is hard in both columns, so the difference between them cannot contain it.

The cost row is arithmetic over the two rows above it, and doing that arithmetic rather than trusting it shows where the money goes. Arm A's 196,000 prompt tokens are a shade under a fifth of a million, so at the row label's $3 per million they cost a little under sixty cents; its 9,200 output tokens are just under a hundredth of a million, so at $15 per million they cost about fourteen cents. The two make the $0.73. Arm B is the same pair of multiplications: 470,000 prompt tokens is a dollar and forty-one cents, its 9,600 output tokens about fourteen cents again, and the two make the $1.55.

Output barely moves between the arms, so almost the whole of the eighty-two-cent difference is the prompt-token line — the line compaction exists to hold down. That makes the claim that arm B costs more than twice as much per session checkable rather than something to accept (chapter 8, which reaches the same figure by tracking mean context size per model call). The cost of summarisation is the input tokens that were never sent.

Checking whether a declared fact survived#

The constraint-survival row is the one that turns a correlation into a mechanism. It is measured directly rather than inferred: each scenario declares a small set of facts the agent is meant to still be honouring at the end — five in this suite, which is what that row's median is a median of — and after every compaction the harness checks whether each is still recoverable from the context the agent is about to receive.

def constraint_survival(context_msgs, scenario) -> dict[str, bool]:
    haystack = "\n".join(m["content"] for m in context_msgs)
    out = {}
    for c in scenario.constraints + scenario.declared_facts:
        out[c.id] = c.survives(haystack)   # exact for numbers/dates/ids,
                                           # embedding match for prose
    return out

The function reads as the definition of the measurement. context_msgs is the list of messages that will be sent on the next call, and the first line flattens them into one string, because the question is whether a fact is present anywhere in what the model will see. c.survives(haystack) is exact for numbers, dates and identifiers, since a budget ceiling either appears or it does not and a near miss on a number is a wrong number; it is an embedding match for prose, since a preference stated in one wording can survive in another and a string comparison would score that as a loss. The return value is one boolean per declared fact, which is why the table row is a count rather than a score.

Losing a median of 2 of 5 declared facts per session at turn 9 is a fact about the summariser, independent of any rubric or judge, and that independence carries most of its weight. A rubric score is a model's opinion about a turn, so a judge miscalibrated at longer contexts could in principle produce the drop on its own. A count of which facts are absent from a string needs no model to adjudicate it, and it moves in the same direction as the score. Two measurements with different failure modes agreeing beats one measurement moving a long way.

The threshold sweep is the causal argument#

Arms A and B establish that removing compaction removes the drop, which is consistent with compaction being the cause and also consistent with "sessions with more context in them do better", which is a different claim. The sweep discriminates between the two, because it changes when compaction happens without changing whether it happens.

The general problem the sweep solves is that a cause and a companion are indistinguishable while both are present at the same instant. If two things always happen together at turn 8, no amount of staring at turn 8 says which produced the effect, because every observation contains both. They come apart the moment one of them can be moved on its own. That is a dose-response relationship, the pattern where changing how much of the suspected cause is applied changes the effect by a corresponding amount and in the corresponding direction, and dose-response is what separates a cause from a companion.

Here the threshold column is the dose and the drop-turn column is the response. The drop moves with the threshold, in the direction the threshold moves it, by roughly the number of turns the token budget buys: lowering it to 8,000 pulls the drop back to turn 6, and raising it to 20,000 pushes it out to turn 11. Nothing else that happens around turn 8 — history length, accumulated tool results, user fatigue in the simulated persona — moves to turn 6 when you set a configuration value to 8,000, because none of those things reads the configuration value. That last clause is the whole argument, and it is worth stating as the step it is: the rival explanations are all properties of the conversation, a configuration value is not part of the conversation, and a variable nothing in the conversation reads cannot drag a conversational property along with it.

The shrinking size of the drop as the threshold rises is a second piece of information. At an 8,000-token threshold the summariser is compressing 5 turns and drops less; at 20,000 it is compressing 10 turns into the same 220-token summary and drops more per session, yet the measured step is smaller — because by turn 11 the sessions still running are the ones with fewer live constraints, and several have already confirmed their booking. The drop is not a fixed penalty; it is the product of how much got dropped and how much of what got dropped still mattered.

As a rule, the measured step is the material lost multiplied by the share of that material still load-bearing at the turn it was lost. The first factor rises with the threshold, because more turns are squeezed into a summary of the same length; the second falls with it, because a constraint honoured in a completed booking cannot be violated afterwards. Two factors moving in opposite directions is why a drop measured at one threshold cannot be extrapolated to another.

Turning compaction off relocates the failure to the context limit#

The last two rows of the A/B table are the reason. At 12 turns, arm B never hits the context limit, so within the suite it looks free. Extending the same 40 scenarios to 24 turns puts 23 of them into a hard context-limit error, where the agent does not degrade gracefully — the request fails, or the provider truncates from the front and silently removes the system prompt along with the early turns. That failure mode is worse than a lossy summary because it is discontinuous and because it removes the instructions rather than the details.

The mistake available here is not a slip in the arithmetic; it is reading the context-limit row as a defence. That row counts sessions that hit the limit, and across the whole suite it reads zero for every arm, in the compaction-off column as much as the compaction-on one. What does move, enormously, is the exposure it is supposed to be watching: how close a session comes to the context window depends on how long it runs, and every session here stops at 12 turns. A quantity that stays flat while the dangerous property underneath it varies is not evidence of safety — it is a quantity pinned by something other than safety, and the pin is the turn cap.

The belief is tempting for real reasons, which is why it is worth naming rather than waving away. The measurement behind it is sound in every particular: the drop at turn 8 genuinely disappears when compaction is off, completion genuinely rises from 70 percent to 88 percent, and the count of missing constraints genuinely falls to zero. What is unsound is the step from those numbers to "and nothing else breaks". Turning compaction off does not delete the failure; it relocates it from turn 8 to whichever turn crosses the window, and it takes the agent cost per session from $0.73 to $1.55 and p50 turn latency from 3.4 s to 4.9 s on the way.

A protected block instead of a policy reversal#

The change that follows from the constraint-survival measurement is narrower. Arm E keeps compaction at 12,000 tokens and alters only what compaction is permitted to see: the facts a scenario declares and the commitments the agent makes are held in a protected block, outside the region the summariser rewrites.

That table is read along its rows, and the finding is which neighbour each row puts E beside. On every quality row E sits next to B: a turn 7→8 step of −0.11 against B's −0.05 and A's −0.66, and a missing-constraint count of 0 of 5, which is B's exactly. On every cost row it sits next to A: $0.76 against $0.73, 3.5 s against 3.4 s, no context-limit failures at 24 turns where B has 23 of 40. Neither adjacency means much alone; the finding is that both hold in one column.

The size of that bargain is arithmetic on the cost row. Moving from A to B costs an extra eighty-two cents a session and moving from A to E costs an extra three cents, and three against eighty-two is about four percent. Arm E recovers most of arm B's quality at four percent of its cost increase, and it keeps the property that made compaction worth having.

The three-point gap between E and B is real and is not free to close: it is the seat preferences, the tone of an earlier exchange, and the small contextual details that a 220-token summary flattens and that occasionally matter. Naming what the gap is made of shows why it is expensive. A protected block works on facts that can be recognised as they are stated, and that residue cannot be enumerated in advance; protecting it means protecting nearly everything, which is arm B with extra machinery.

The forty scenarios are an authored set, not a sample#

The experiment establishes that compaction as configured, on this suite, with this summariser prompt — the instruction given to the model that writes the summary, which is a component someone wrote rather than a fixed property of summarisation — caused the drop at turn 8. It does not establish that summarisation is harmful in general, and running the same five arms against a different agent with a different summariser would be a different experiment with possibly the opposite answer. It says nothing about sessions longer than 24 turns, where hierarchical summarisation of summaries — a summary itself compacted later together with the turns that followed it — introduces failure modes this suite never reaches. And it cannot rule out that some other threshold-driven behaviour rides along with compaction in the same code path, which is why the constraint-survival probe matters: it observes the mechanism directly rather than inferring it from the outcome.

The pairing that made the analysis strong also draws its boundary. It rules out arm B having looked better for drawing easier scenarios, and it does nothing whatever about the scenarios themselves, because both arms drew the same ones. Every number in this chapter is conditional on the 40 scenarios, and the 40 scenarios are a fixed set someone chose.

That decides what can honestly be claimed for arm E. A protected block that preserves budget ceilings and dates preserves them because those are what the scenarios declare; whether it preserves the thing a real user cares about in a scenario nobody wrote is unknown, and the honest summary of arm E is "removed the measured failure and did not measurably introduce another". Narrowing that gap means widening the authored set, with scenarios written by people who did not write the agent's prompt; running the existing forty under more seeds narrows the noise and leaves the scope where it was.

A coincidence at a single boundary became a mechanism in three moves. Arm A against arm B established that removing compaction removes the drop, which stayed compatible with a rival explanation about how much context a session has. Sweeping the threshold moved the drop with it, which no rival can imitate, since none of the alternatives reads a configuration value. And the constraint-survival probe named the facts that went missing, putting a narrow repair on the table in place of a policy reversal.

That leaves a reporting problem. Arm E moves end-to-end completion from 70 percent to 85 percent, and someone is going to ask whether the change helped — a question that on 40 scenarios has a wider answer than the difference itself, and which a single aggregate number is particularly bad at answering.