Evaluating a compaction change

A pull request that replaces a compaction summariser arrives with numbers attached, and the numbers are usually these three: mean prompt size after compaction, task success over a replay of recorded conversations, and a handful of sample outputs that read visibly better. All three are easy to produce, all three sound like evidence, and none of them can detect the failure the change is most likely to introduce.

Two separate objections apply and they need to be told apart. The first is ordinary statistics. With 200 conversations and a success rate near 0.83, the standard error of that rate is the square root of 0.83 × 0.17 / 200, which is 0.027, so the 95% interval around 0.83 spans roughly 0.78 to 0.88, and a move to 0.84 is well under half of one standard error. Running the identical summariser twice with the same fixtures and a non-zero temperature produces differences of that size routinely, and the direction is a coin flip.

That formula is worth reading rather than applying, because what it encodes is the reason the number moves at all. Each conversation either succeeds or fails, so the reported rate is the share of successes in 200 independent trials, and the spread of such a share has a known size: the rate multiplied by one minus the rate, divided by the number of trials, all under a square root. The 0.17 in it is simply one less 0.83. Two features of that expression decide everything here. The number of trials is in the denominator under a root, so halving the uncertainty needs four times the conversations rather than twice as many. And the numerator is largest when the rate is near the middle, which is where a task-success rate usually sits. Working it through gives an interval about a tenth of a point wide around a figure that moved by one hundredth, and a difference that small inside an interval that wide is not evidence of anything, in either direction. That objection has ordinary answers, which topic 4.17, on measuring context changes, works through: run more conversations, or pair the runs so each conversation is compared against itself. The second objection has no such answer, because it is about what kind of thing is being averaged.

Rare and catastrophic does not show up in a mean#

The failures a compactor causes are not a slight degradation spread evenly across conversations. Dropping a constraint about a forbidden directory does not make 200 conversations 0.5% worse; it makes 197 conversations identical and 3 conversations produce a schema change against a production database. Inverting a hedge into an approval does not shift a mean; it produces one conversation in which the agent tells the user they approved something they did not, and 199 in which nothing happens.

Work the arithmetic on the optimistic assumption that every affected conversation fails outright and the grader catches it. Three catastrophic failures in 200 is a 1.5-point drop in success, which is smaller than the noise band computed above, so the aggregate cannot distinguish "we introduced a constraint-dropping bug" from "we ran the eval on a Tuesday". The pessimistic assumption is worse: a compaction failure often produces work that looks correct to a grader, because the agent completes the task competently while violating a rule the grader was never told about. A reported 0.84 may include the schema change as a success.

Put the two figures next to each other and the aggregate is ruled out before any run happens. Three conversations in 200 is one and a half points, and one standard error on this rate is two and seven tenths of a point. The largest effect a constraint-dropping bug can have on the aggregate, under the most generous assumption available, is smaller than the ordinary run-to-run wobble of the measurement. No amount of care in running the eval changes that, because it is not a question about execution. The failure is concentrated in a handful of conversations and the metric divides by all of them, so the metric's resolution is coarser than the thing being looked for. That is worth establishing on paper first: it tells you the aggregate cannot be fixed by running it better, only replaced by a measurement that counts items instead of conversations.

The measurement has to be per item rather than per conversation, and it has to ask about specific things that were stated, because those are the things whose loss is invisible in the output.

A fixture that names what must survive#

Annotating a recorded conversation for this purpose turns it into a fixture, a stored replayable input paired with the answers any correct run still has to be able to give. Building one means listing the items that bind later turns — the same classes as the pinned ledger — with the turn where each was stated and its literal text.

{
  "conversation_id": "sess_2026_03_11_a4f1",
  "turns": 63,
  "compactions_expected": 2,
  "must_survive": [
    { "id": "c1", "class": "constraint", "turn": 3,
      "literal": "never modify anything under /migrations",
      "probe": "Is the agent forbidden from modifying files under /migrations?",
      "expect": "yes" },
    { "id": "c2", "class": "commitment", "turn": 19,
      "literal": "I'll open the PR against develop, not main",
      "probe": "Which branch should the PR target?",
      "expect": "develop" },
    { "id": "c3", "class": "open_question", "turn": 22,
      "literal": "Would an enum be better than a varchar for orders.status?",
      "probe": "Has the user decided whether orders.status becomes an enum?",
      "expect": "no — still open" },
    { "id": "c4", "class": "reference", "turn": 41,
      "literal": "report.md rev 14",
      "probe": "Where is the report being written, and at what revision?",
      "expect": "report.md, rev 14" }
  ]
}

Which conversations go into the set matters more than how many. A random sample of 200 production sessions is mostly short ones that never compact at all, and those contribute nothing except a comfortable denominator — half of them will pass every check because no compaction ran. Selecting adversarially instead gives a set that is smaller and far more informative: the longest sessions, the ones with the most compactions, the ones the arrival-time classifier flagged the most constraints in, and every session where a compaction incident was reported. Forty conversations chosen that way exercise more distinct compaction decisions than 200 chosen at random, and they keep the annotation cost within reach.

The reason that is not a paradox is that the unit of evidence is a compaction decision rather than a conversation. A session that never reaches the threshold runs no compaction, so it exercises the code under test zero times and passes every check by not being affected — it adds to the denominator and to the annotation bill and to nothing else. A session with several compactions exercises the code once per compaction, and a session full of constraints exercises it once per constraint per compaction. Counting conversations therefore counts the wrong thing, in a way that systematically favours the sessions carrying no information. Forty sessions selected for length and constraint density can contain more decisions than two hundred selected at random, and each of those decisions is one the change could plausibly get wrong.

Checking those items against a compacted context is two different operations. Anything the pinned ledger claims to carry verbatim is checked by substring match, which is deterministic, free, and has no false positives — either the string "never modify anything under /migrations" is in the assembled request or it is not. Anything that was legitimately summarised needs a judge, and the judge is asked a closed question against the compacted context alone, with three permitted answers rather than two.

JUDGE = """Here is the entire context an agent will see on its next turn.
Answer ONLY from this context. Question: {probe}
Reply with JSON: {{"answer": "<short answer>",
                   "verdict": "supported" | "absent" | "contradicted"}}"""

def check(fixture, compacted_ctx) -> list[Result]:
    out = []
    for item in fixture["must_survive"]:
        if item["literal"] in render(compacted_ctx):
            out.append(Result(item["id"], "verbatim"))
            continue
        v = judge(JUDGE.format(probe=item["probe"]), compacted_ctx)
        out.append(Result(item["id"], v["verdict"], answer=v["answer"]))
    return out

The three-way verdict is the part that carries the design. An item that is absent has been dropped, which costs the agent information and usually produces a question it should not have needed to ask. An item that is contradicted has been inverted, which is a different kind of failure and a much worse one, because the agent will act confidently on a false statement. Collapsing both into "not retained" and reporting a single recall number hides the distinction that matters most, so they are counted separately and gated separately: absences are budgeted, contradictions are not.

The third option is not a refinement of the judge; it is what makes the gate expressible. A two-way judge answers whether the item survived, and a single retained-items percentage follows from that. Any percentage has to have a threshold below one hundred, because a compaction that dropped nothing would not be a compaction, so the gate becomes "at least this share survived" — and under that gate an inversion and an omission each consume one unit of the same budget. They should not. An omission costs the agent information and typically produces a question it should not have needed to ask, which is a cost worth trading against a smaller request. An inversion gives the agent a false statement it will act on confidently, and there is no request size that trades against that. Two failures requiring opposite treatments cannot share a budget, which is why the judge has to be able to name which one it saw.

The report that replaces the three numbers#

Running that over the fixture set produces a table with prompt size in it rather than a prompt-size result with caveats attached.

Reading that table honestly means noticing something the scenario below does not dwell on: the incumbent does not pass either. Seventy-one of eighty is under the 95% gate, forty-four of fifty-one is under the 90% gate, ninety-six of a hundred and three is under the 95% gate, fifty-eight of sixty-one is under the 98% gate, and two contradictions is not zero. Five of the six gates are unmet by the summariser currently in production. That is not a defect in the table. A gate is a standard the system is being held to, and a fixture set introduced to a system that has never been measured this way will normally find the standard is not yet met — which is a backlog rather than a merge decision. What blocks a particular change is the rows that got worse under it: contradictions rising and open questions falling. Conflating "this change is unsafe" with "this system is not yet compliant" is how a gate becomes something a team learns to ignore.

Task success stays in the report, on the last line, described as reported rather than as a gate. It is worth measuring because a change that halves task success would be caught by it, and it is worth being explicit that it will not catch anything smaller.

Making the replay see behaviour#

Checking the compacted context is a static test, and it misses failures that only appear when the agent acts on it. The behavioural version runs each recorded conversation twice from the same fixed prefix — once with compaction disabled, which is the reference, and once with the compaction under test — then compares what the agent does on the turns after the compaction point.

Pairing does two useful things. It removes most of the variance from the success comparison, because the same conversation is compared against itself with the same seed and temperature 0, so a test over the discordant pairs — the conversations where one run succeeded and the other did not, which are the only ones carrying information about the change — has far more power than comparing two independent rates.

Where that power comes from is worth being precise about, because "paired tests are stronger" is a slogan otherwise. Most of the variation in a task-success rate is not caused by the compactor. It is caused by the conversations: some tasks are harder, some fixtures are ambiguous, some sessions would fail under any compaction at all. Comparing two independent rates means that variation appears in both measurements and has to be seen through. Running the same conversation twice puts the same difficulty on both sides of the comparison, so it cancels, and what remains is the effect of the one thing that differed. The conversations where both runs agreed then carry no information about the change — whatever happened, happened either way — which is why the analysis is over the disagreements. It is also why the disagreements are the right place to spend the review time. It also gives the review work a natural budget: on a set of 200 conversations the two runs typically agree on 180 or more, and only the disagreements need a person, which is an hour of work rather than a week.

The reference run has a limit that should be stated rather than assumed away. An uncompacted run is only available for conversations that fit in the window without compaction, which excludes precisely the longest and most compaction-sensitive sessions. For those, the reference has to be a recorded run that a human has already accepted as correct, which makes the comparison a regression test against a known-good trace rather than against ground truth.

Annotation coverage and the production signals#

They cannot see anything they do not encode. A conversation whose annotation lists four items will not reveal that a fifth was lost, and the items most likely to be missed by an annotator are the ones an agent complied with silently for fifty turns — the same property that made them invisible to the summariser. Generating candidate items automatically with the arrival-time classifier from the pinned-ledger chapter, run over the original turns, and having a person confirm or reject them takes annotation from around fifteen minutes per conversation to about two, which is the difference between a 50-hour project and a 6-hour one and is what makes the fixture set exist at all.

Both halves of that are worth reading. The arithmetic is straightforward — two hundred conversations at a quarter of an hour each is fifty hours, and the same two hundred at a couple of minutes is closer to six — but the reason the time falls is the interesting part, and it is not that the classifier is fast. It is that authoring and reviewing are different tasks. Authoring means reading sixty-three turns while holding in mind what kinds of thing bind later turns, and the items an annotator misses are exactly the ones the summariser misses, for the same reason: they were complied with silently and generated nothing. Reviewing means reading a proposed list and answering yes or no to each entry, which requires no recall at all. The classifier is not doing the judgement. It is converting the judgement into a form where a person's attention is spent confirming candidates rather than remembering to look for them.

They also cannot see failures in shapes nobody has recorded. That gap is covered from production rather than from the fixture set, with two online signals that cost almost nothing: a tool call repeated with identical arguments within five turns of a compaction, and a user message that restates an instruction they already gave. Both are cheap to detect, both are heavily concentrated in sessions where compaction went wrong, and both generate the conversations that become next quarter's fixtures.

What makes those two work in production, where no fixture exists, is that each is a consequence of a lost item rather than an observation of one. Detecting a lost item directly would require knowing what was in the conversation before the compaction, which is the thing an annotation supplies and production does not. A repeated tool call with identical arguments needs neither: it is two records in a log with the same fields, and the only reason an agent re-issues an identical call within a few turns is that it no longer holds the result. A user restating an instruction is the same shape seen from the other side — the user is a detector that noticed the agent stopped honouring something, and their message is the alarm. Neither signal says which item went missing. Both say that this session is worth annotating, which is all a fixture pipeline needs from them. Sampling twenty flagged sessions a week keeps the fixture set tracking the product instead of tracking the day it was built.

The one thing this framework deliberately does not measure is how well the summaries read. That quality has almost no relationship to whether the next turn behaves correctly, and it competes with the qualities that do, since decisive well-structured prose is produced by the same instincts that drop hedges and close open questions. Readability matters for the humans debugging the agent, which is an argument for logging a readable narrative alongside the state document rather than for putting one in the request.

A compaction change is safe when the items that bind later turns survive it, and that is a per-item measurement over conversations chosen for being hard rather than for being numerous. Pinned text is checked by substring and costs nothing; summarised material is checked by a judge that is allowed to say the item was inverted rather than merely lost; and the aggregate everyone wanted to lead with goes at the bottom of the report, where it can be read for what it is worth.

What none of this does is make the decision on a live session. The last chapter of this topic takes a coding agent ninety minutes in, holding a constraint from turn 3, a half-finished plan and sixty turns of file reads, sitting 8,000 tokens over budget, and works the compaction item by item.