Hierarchical compaction

The obvious compaction loop is recursive: when the threshold is hit, send the previous summary plus the turns since to a model and ask for an updated summary. It is cheap, it is what most implementations start with, and it has the property every engineer likes, which is a constant cost per compaction regardless of how long the session runs. It also mirrors the running-total pattern that works correctly everywhere else in software — fold the new data into the accumulator, discard the input, repeat.

The reason it fails here is that the fold is lossy and non-deterministic. The accumulator is not a summary of the session but a summary of a summary of a summary, and there is no invariant tying it back to what actually happened. By the sixth pass a sentence from early in the session has been rewritten six times, each time by a model that could see only the previous rewrite, and the original text has not been in the request since the first compaction.

The generations of one sentence#

Log every compaction output and the chain is recoverable. This is what six passes did to one sentence a user typed at turn 22 of a long migration session.

No single step in that chain is a hallucination. "Raised" is a defensible rewrite of "asked whether", "proposed" is a defensible rewrite of "raised", and by pass four the hedge has been gone for two generations and there is nothing in the input to reconstruct it from. Each pass is faithful to its input; the chain as a whole inverts the meaning of the original.

Two properties of that process make it worse than ordinary noise. The first is that the errors are directional. A summariser under instruction to be concise drops qualifiers before it drops nouns, because "not sure it's worth a migration" is nine words carrying an attitude while "convert orders.status to enum" is the actionable content, and every compression objective ever written rewards the second over the first. The second is that the errors are absorbing. Once "wants" has replaced "asked whether", no later pass can recover the question, because recovering it would require information that is no longer in the input. Confidence in this chain only ever increases; you will not find a case where pass five turned a decision back into an open question.

"Absorbing" is a strong word and the argument for it is short enough to check. Each pass sees exactly one thing: the previous pass's output. A pass can only write what its input supports, since anything else would be invention rather than summarisation, and a well-behaved summariser does not invent. By pass four the input says the user wants the conversion. Nothing in it distinguishes a user who wanted it all along from a user who asked a question that was progressively firmed up, because the record of the question is gone. So a pass that restored the hedge would be adding a fact it cannot support, which is the one thing the summariser is reliably good at not doing. The property that makes each individual pass trustworthy is the same property that makes the chain irreversible.

That is why this differs from ordinary measurement error, and the difference matters for what fixes it. Noise is symmetric and cancels: measure something six times with a wobbly instrument and the average is better than any single reading. Here every pass moves in one direction and no pass can move back, so repetition does not average anything — it accumulates. Adding more passes to a recursive compactor makes the estimate worse, monotonically, which is the opposite of how almost every other repeated process in a system behaves.

Measured across a replay of sixty long recorded sessions, hedge and qualifier retention — checking, for each originally hedged statement, whether the hedge is still expressed — ran at 88% after one pass, 51% after three, and 22% after six.

Those three numbers are more informative read against each other than read in turn. If every pass were an independent trial at the first-pass rate, three passes would leave a bit over two-thirds of the hedges standing. The measured figure after three passes is about half, which is worse — so the per-pass rate is not constant, it deteriorates. The step from three passes to six says it again: retention loses a larger share of what remains over the second set of three passes than over the first, so the later passes are at least as destructive as the earlier ones despite having far less left to destroy. That is what the absorbing property looks like when it is measured rather than argued. Each pass hands the next one a slightly weaker version of the hedge, and a weaker hedge is an easier thing to drop. Outright inversion, where a question or a rejected option is rendered as a decision, showed up in about 0.6% of tracked items per pass, which sounds small until you multiply it by the forty or so tracked items in a long session and the six passes: roughly one session in eight ended with at least one inverted commitment. That is the expected outcome of the design at that session length rather than a rare model error.

Summarising from originals#

The fix follows from where the information is lost. The original turns still exist; they were evicted from the context window, and there is no reason for them to have been deleted from disk. Writing every turn to an append-only log as it happens costs a few kilobytes per turn and makes the original text available to every future compaction, at which point each summary can be derived from source material rather than from the last paraphrase.

Every arrow in that diagram starts at original text. A block summary covers a fixed span of turns, is generated once when the span closes, and is never regenerated — turns 1 to 40 are summarised at turn 41 and that summary is immutable thereafter, so it can drift exactly once. When the accumulated block summaries themselves get too large, the span summary that replaces them is generated from the originals of turns 1 to 120, not from the three block summaries, so it is also one generation away from source.

def compact(session: Session, budget: int) -> list[Message]:
    log = session.durable_log            # every turn, verbatim, on disk
    blocks = closed_blocks(log)          # 40-turn spans, immutable once closed

    for b in blocks:
        if b.summary is None:            # generated exactly once, from originals
            b.summary = summarise(log.turns(b.start, b.end), style=STATE_SCHEMA)

    ctx = [session.system, session.pinned_ledger]
    ctx += [b.summary for b in blocks]
    ctx += log.turns_since(blocks[-1].end)          # recent turns, untouched

    if tokens(ctx) > budget:             # block summaries alone no longer fit
        span = summarise(log.turns(0, blocks[-2].end), style=STATE_SCHEMA)
        ctx = [session.system, session.pinned_ledger, span,
               blocks[-1].summary, *log.turns_since(blocks[-1].end)]
    return ctx

The important line is the argument to summarise in the if branch: log.turns(...), the originals, rather than the block summaries that are sitting right there and would be a hundred times cheaper to read. That substitution is the entire difference between one generation of drift and six.

Why immutability does the work is worth stating separately from the code, because it is the property the design rests on. A block summary is generated once, when its span of turns closes, and is never regenerated afterwards. So it is written from original text exactly once, drifts exactly once, and then stops changing — whatever happens later in the session, the summary of turns 1 to 40 is the same object it was at turn 41. The recursive design has no such object. Its accumulator is rewritten on every compaction, so the number of times a sentence has been paraphrased grows with the length of the session. Drift in the hierarchical design is bounded by how many summarisation steps stand between an original turn and the request, and that is one for a block and one for a span, regardless of whether the session runs to a hundred turns or a thousand.

Where the block boundaries fall is a real design decision rather than a detail of the loop. A fixed span of forty turns will sometimes cut through the middle of a unit of work — the agent is nine turns into diagnosing a failing test when the block closes, so half the diagnosis is summarised as a stalled investigation and the other half arrives in the next block with no statement of what it was investigating. Closing blocks at natural seams instead produces summaries that are individually coherent: a completed plan step, a resolved tool loop, the point where the user changed subject. A cheap and surprisingly effective rule is to close a block at the first turn after the span limit where there is no open tool call and the agent has just reported a completed step, with a hard cap so a session that never reaches a seam still closes eventually.

A mid-work cut is worse than an uneven block for a reason that has nothing to do with summariser quality. A summariser can only describe what it was given. Handed the first nine turns of a diagnosis, it reads a line of enquiry that stops without reaching a conclusion, and the honest account of that is a stalled investigation, which is what it writes. Handed the remainder, it reads work whose subject was established before the earliest message it can see, and the honest account of that is activity with no stated purpose. Neither output is wrong about its input. The fact that would reconcile them — that these are two halves of one piece of work — lives in the relationship between the two blocks, and no single call has both. Closing at a seam does not make the summariser better. It makes each input a complete unit, which is the condition under which a faithful account of the input is also a faithful account of the work.

What it costs to stop re-summarising summaries#

Reading originals repeatedly is more expensive than reading the last summary, and the amount is worth having in front of you before you argue about it.

The two totals in that table are worth building rather than believing, because the ratio between them is the price of the whole design. The recursive column is one multiplication: twelve compactions at about 13,000 tokens of input each is the 156,000. The hierarchical column is two products added together — seven block closes at about 48,000 tokens, plus two span rebuilds at about 360,000 tokens, which come to the 1,056,000. The span rebuilds dominate, and they dominate because each one reads the whole session from the start; that is precisely the thing being paid for. Seven times the input at the same per-token rate is seven times the spend, which is why $0.06 becomes about $0.42 and not something harder to predict.

The money is not the constraint; $0.42 of summariser spend sits inside a session whose main-model cost is two orders of magnitude larger. The forty-second call is the constraint, because a user is waiting on the turn that triggered it. Two adjustments handle that. Block summaries are generated when the block closes rather than when the budget is hit, which moves them off the critical path entirely — the block closing at turn 80 can be summarised in the background while the agent works on turn 81, and by the time the budget forces a compaction the summary already exists. Span rebuilds are rarer and can be scheduled the same way, triggered at a soft threshold well below the hard one so there is time to complete before anything is actually required.

Depth is not summariser quality#

It does not improve a bad summariser. A block summary generated once from originals by a model that systematically drops negative results will drop them just as reliably at depth one as at depth six; hierarchy limits how many times an error can be amplified, and does nothing about whether the first pass was any good. The measurements from the previous chapter — does the output carry findings, open items and references — are still the ones that decide summariser quality, and they should be run per block.

It also does not protect commitments, and it is important not to let the structural fix crowd out the simpler one. The orders.status sentence should never have been in the compactable span at all. It contains a hedge about a schema change, which makes it an open question under the pinned ledger from the previous chapter on classifying binding items at arrival, and a pinned open item reading "user asked whether orders.status should become an enum; expressed doubt about migration cost on a table that size; undecided" would have survived all six passes untouched because no model would have been asked to rewrite it. Hierarchy is the defence for the material you have accepted will be paraphrased; the ledger is the defence for the material that must not be.

The retention requirement rules this design out for some systems. Keeping every original turn on disk means keeping user-supplied content, which is exactly the material that data-protection commitments constrain: a product that promises deletion within twenty-four hours, or that handles health or payment data, may not be able to hold a durable log at all. Where that applies, the fallback is to bound the depth rather than eliminate it — retain originals only for the current block, freeze older block summaries so they are never re-derived, and accept one generation of drift per block with no compounding across blocks. That gives up the ability to rebuild a span from source, and it keeps the property that matters most, which is that no summary is ever the input to another summary.

Depth also cannot be driven to zero on a long enough session. Three hundred turns of a coding agent is roughly 400,000 tokens of original text, which does not fit in one summariser call, so a span summary at that scale has to be assembled from block summaries and is a second generation whether you like it or not.

The bound on depth is arithmetic rather than policy, which is worth seeing because it tells you where the design stops. Summarising from originals requires the originals to fit in the summariser's own context window, and a session's original text grows without limit while the window does not. Once the session outgrows the window, one call cannot see the whole session, so the top-level summary has to take summaries as input, and that is a second generation by construction. Nothing about a better harness avoids it. What the design controls is how many generations sit between an original turn and the request at the point the session ends, and the argument for hierarchy is that this count grows with how many levels the session has grown to need rather than with how many compactions have run. Twelve becoming two is that difference at three hundred turns, and the count rises by one only when a session grows long enough to require another level, not every time the budget is hit. What hierarchy buys is a change in how drift scales: twelve generations at turn 300 becomes two, and two is a number you can measure and reason about.

Every mechanism in this chapter and the three before it is triggered by something deciding that the context is now too large. That decision is usually made by a counter that has been quietly correct for months, and a counter encodes an assumption about how much a turn can cost — an assumption that one new tool can invalidate in an afternoon, in a way no amount of good summarisation repairs.