Dropping turns and rolling summaries

A model keeps no memory between calls, so every turn is sent as the whole conversation from the beginning, and the request grows with each exchange while the context window — the fixed number of tokens the model accepts in one request — stays where it is. The code that assembles the request, calls the API, appends the reply and goes round again is the harness, and it is the only component that can notice the transcript has outgrown the budget and act. It has two operations available. Eviction takes material out of the request and stops sending it, which is what happens when the oldest turns or the largest tool outputs are dropped outright. Compaction replaces a span of the transcript with a shorter rendering of that same span, usually a summary a model writes of the conversation so far, so those turns go on contributing at a fraction of their original size.

Most engineers meet the two from the outside before they implement either: a coding agent that pauses in a long session to announce it is compacting the conversation, an assistant that turns vague about something you told it an hour ago, a support bot that asks again for an order number you have already given. Inside the harness it is a function with a plain signature, taking a message list that is too long to send and returning one that fits. Claude's API offers a server-side version, and the agent frameworks all ship some form of history trimming or summarisation step. This chapter is about how that function should choose, because the choice is usually made on the wrong grounds — the reasoning offered for a summary is that it retains more information per token than deletion does, and that is not what it does.

Exactness and continuity are separate goods#

Take two implementations of "something leaves". The first, drop_oldest, walks the message list from the front and deletes turns until the request fits; it costs no money, finishes in under a millisecond, and is roughly forty lines including the guard that keeps tool calls paired with their results. The second, rolling_summary, takes everything older than the last six turns, sends it to a small model with an instruction to summarise the conversation so far, and replaces those turns with the 800 tokens that come back; it costs one extra model call per compaction, reads about 46,000 tokens on a long session, and adds two to four seconds to the turn the user is waiting on.

The belief that decides between them is usually the one in the introduction, and it is worth setting out properly because it is nearly right. Compare the two outputs by asking how much of the original conversation a reader could reconstruct from each, and the summary wins outright: from 780 tokens of prose a reader recovers the subject, the shape and the outcome of thirty-five turns, while from a deletion they recover nothing. On that measure a summary really does retain more per token, and anyone who has read both would say so. The step that does not follow is that reconstruction is the operation being performed. The next turn does not need to reconstruct the conversation. It needs one item, exactly, and a summary's information is spread evenly across everything that happened while the item needed is a handful of characters that either survived or did not. A representation can be better on average and useless at the one point of use.

Deleting the oldest turns is lossless on everything it keeps. A surviving turn arrives at the model as the exact bytes the user typed, with the exact figures, the exact hedges, and the exact ordering. What it loses, it loses completely and visibly: the early turns are simply not there, and nothing in the remaining context implies they ever existed. The agent does not misremember an agreed figure, because it has no representation of the agreed figure at all.

Summarising inverts both halves of that. Every turn in the compacted span contributes something, so continuity survives — the model still knows a negotiation happened, that concessions were made, that a tone was established. What it loses is exactness, and it loses it invisibly. A summariser writing "the agent agreed to refund most of the purchase price and waived some fees" has produced a sentence that is true, fluent, and useless for the only decision the next turn has to make. Worse outcomes are available: "around €900", or "the agent offered to waive the restocking fee", which turns a granted exception into an offer that might still be withdrawn.

Here is the material that makes the difference concrete, reduced to the four messages of a support conversation that carry the value of the whole thing.

[
  { "role": "assistant", "turn": 12,
    "content": "I can process this as a goodwill return even though it's past day 30. The refund would be €930 — that's the €1,240 purchase price less the 25% opening deduction our policy applies to unsealed appliances." },
  { "role": "user", "turn": 13,
    "content": "€930 works, but I'm not paying a restocking fee on top of that." },
  { "role": "assistant", "turn": 14,
    "content": "Understood — I've waived the €75 restocking fee for this return, and we'll cover return shipping as well. Should the original €18 delivery charge be refunded too, or would you rather I leave that on the order?" },
  { "role": "user", "turn": 15,
    "content": "Let me check what I actually paid for delivery, one sec." }
]

The refund figure in turn 12 is not a number the agent chose; it is a calculation the customer can check, which is why its exactness matters. The machine cost €1,240. Policy applies a 25% opening deduction to unsealed appliances, and a quarter of €1,240 is three hundred and ten euros. Subtracting that leaves €930. The customer accepted the figure in the next turn, and both parties can reconstruct it from the two inputs. A later message saying "around €900" is not a rounded version of that arithmetic, because no deduction rate applied to €1,240 produces it. It is a different claim, and it is one the customer can disprove with a calculator.

Nine turns of back-and-forth about collection slots follow, then the customer changes the delivery address, then a courier question, and by turn 41 nobody has returned to the €18. That is the state a compaction has to preserve. Run each strategy against it and the outputs diverge in a way that is easy to check.

drop_oldest (turns 1-35 removed, 34,900 tokens freed)
  → context now begins mid-thread at turn 36: "the courier slot on Thursday
    is fine". No refund amount. No exceptions. No open question.
  → next turn, the agent asks the customer what refund amount was agreed.

rolling_summary (turns 1-35 → 780 tokens)
  → "The customer is returning an espresso machine outside the return
     window. After some negotiation the agent agreed to a refund of
     approximately €900 and waived fees as a gesture of goodwill.
     Collection has been arranged for Thursday."
  → next turn, the agent confirms "€900" in writing to the customer.

The dropped version fails loudly, which is annoying and recoverable — the customer re-states the number, the agent looks incompetent, the conversation continues. The summarised version fails quietly and expensively, because the agent now states a wrong figure with the same confidence it would have stated the right one. Both compactions "worked" by the only metric either of them reports, which is that the request fit.

"Approximately €900" is not the summariser malfunctioning, and understanding why is what stops the fix from being "use a better summariser". A summariser is asked for a readable account of a conversation, and in a readable account of a negotiation, an approximate figure is good prose. "The agent agreed a refund of approximately €900" reads better than the exact figure and costs fewer tokens, and every objective a summariser is given — be concise, be readable, capture what happened — scores it at least as highly as the precise version. There is no term in the objective for this number will be quoted back to a customer in writing, because nothing in the transcript marks it as that kind of number. The failure is a correct optimisation of the wrong target.

The second half of the failure is on the reading side and is the part that makes it expensive. When the agent reads the summary at turn 42, "approximately €900" arrives as ordinary context, indistinguishable in form from the original message that said €930. A dropped turn leaves a hole the agent can notice; a paraphrased turn leaves a sentence, and a sentence that is slightly wrong looks exactly like a sentence that is right. That asymmetry — loud loss against quiet corruption — is the real content of the choice, and it does not appear anywhere in a token count.

Three profiles against the same conversation#

The two implementations are the endpoints of a range rather than the only options. The working answer for a conversation like this one is a hybrid that treats a small set of items as non-negotiable and compresses only the discussion around them.

The last row is the one that surprises people the first time they measure it. The three strategies recover almost the same number of tokens, because the bulk of a long transcript is discussion, and the items worth pinning in this conversation amount to a few hundred tokens: the €930, the two granted exceptions with the words "waived" and "we'll cover", and the unanswered question about the €18. Pinning them costs about 300 tokens out of 34,000 recovered. The choice between the strategies is therefore not a budget choice at all, and framing it as one is how teams end up defending a rolling summary on the grounds that it is more efficient.

The three figures in that row are one subtraction each, and reproducing them is what makes the point unarguable. Turns 1 to 35 weigh 34,900 tokens. Dropping them removes all of it, so the recovery is 34,900. Summarising them puts 780 tokens back into the request, so the recovery is 34,900 less 780, a little over 34,100. Pinning as well as summarising puts back the same 780 plus about 300 tokens of quoted items, so the recovery is a little over 33,800. Everything that distinguishes the three strategies costs about a thousand tokens between the extremes, against tens of thousands recovered by all of them.

Set that against what actually has to be freed and the framing collapses. The transcript is 63,400 tokens against a 60,000-token working budget, so the deficit is a little over three thousand tokens, and every one of the three strategies recovers roughly ten times that. There is no arm here that fits and no arm that does not. A budget can only ever answer the question "does this fit", and when all three candidates answer yes, the budget has been asked its only question and has run out of things to say. What remains is a question about which kind of loss the next turn can survive, and that question has to be settled on the material rather than on the arithmetic.

It is worth being precise about what a summary is actually paid for, because the accounting is easy to get backwards. The 800 tokens of summary sit in every subsequent request until the next compaction, so at 25 turns between compactions and roughly 1,100 tokens of new material per turn, the summary is re-read about twenty-five times and costs perhaps 20,000 cumulative input tokens beyond the one-off generation call. Against a 63,400-token transcript that is a rounding error, and against the cost of confirming a wrong refund figure in writing it is not worth discussing. The interesting cost of a summary is never the tokens; it is the probability that one of the sentences in it is subtly wrong and the agent has no way to tell.

The multiplication behind that is worth doing once, because it is the one people expect to be large. A summary is not read once. It is part of the request on every turn until the next compaction replaces it, so a summary of 800 tokens surviving twenty-five turns is charged as input twenty-five times, which is around twenty thousand tokens of cumulative input across the interval. Over that same interval the conversation adds roughly 1,100 tokens of new material per turn, so one ordinary turn puts more into the request than the entire summary weighs. Set the cumulative cost of the summary against the 63,400-token transcript it is compressing and it is a fraction of one compaction's worth of savings. The accounting that makes a summary look expensive is the accounting that forgets to compare it with anything.

The pinned form of the same compaction looks like this, and the shape matters as much as the content, because the pinned block is quoted rather than described.

=== Carried forward verbatim (do not paraphrase) ===
AGREED: refund amount €930 (turn 12, accepted by customer turn 13)
GRANTED: €75 restocking fee waived (turn 14)
GRANTED: return shipping covered by us (turn 14)
OPEN: is the original €18 delivery charge refunded? asked turn 14,
      customer said "let me check", never answered

=== Summary of turns 1-35 ===
Customer returning a €1,240 espresso machine 11 days past the return
window. Machine was unsealed but unused. Agent processed it as a goodwill
return. Collection booked for Thursday 14:00-18:00; delivery address
changed at turn 27 to the customer's work address (see turn 27 for the
full address). Tone has been cooperative throughout.

An agent reading that cannot invent €900, because €930 is present as a literal string it did not generate. It also cannot forget the €18, because the open item is labelled as open and carries the reason it is open. The summary underneath is allowed to be lossy, since the address is referenced rather than reproduced and the "tone has been cooperative" line is exactly the sort of gist a summary is good at.

The OPEN line is doing something structurally different from the other three and it is the one most often left out. The other three items exist as text in the transcript — a figure was stated, an exception was granted, the words are there to be copied. The open question is an absence: what makes the €18 unresolved is that no later turn answers it, and an absence has no location in the message list to be preserved from. Any process that works by keeping the important parts of what was said will lose it, not through carelessness but because there is nothing at that position to keep. The only representation available is to state it explicitly, which means somebody or something has to have noticed the non-arrival of an answer across thirty-five intervening turns. That is why open items are the class most reliably dropped, and why they are the one class a pinned block has to construct rather than quote.

Workloads that accumulate no state#

The same three options against a different workload give a different answer, and it is worth having a case where the honest choice is to do nothing at all. A document question-answering agent holds a 40-page contract in its context and answers questions about it: what is the notice period, does clause 9 survive termination, which entity signs. Each question is answered from the document, the previous questions do not constrain the next one, and the user asks eleven of them over twenty minutes.

Here the older turns have no bearing on the next answer. Dropping them costs nothing measurable, and a rolling summary is pure overhead — it adds a model call and two to four seconds to a turn, spends tokens describing questions nobody will ask again, and introduces a channel through which the model can acquire a wrong belief about the contract that the contract itself does not support. A summary that says "the agent explained that the notice period is 60 days" is now a second, lower-quality copy of a fact that is already sitting in the context in its original form. If the summariser gets it wrong, the agent has two sources that disagree, and no rule for preferring the document.

The last clause is where the harm actually lands, and it is worth taking one step further down. Everything in the assembled request arrives as the same kind of thing: text in a message list. The contract clause and the summary sentence about the contract clause are both prose in the context, and nothing in the representation records that one is the source and the other is a derivation of it. A person given a contract and a colleague's note about it knows which to trust without being told. The model has no equivalent, because the provenance that would settle it was never encoded — so when the two disagree, which one wins is a matter of which happens to be more salient at that point in the request. Adding a summary to a workload that already holds its source material does not add a second opinion. It adds a rival with no way to rank it.

There is a second reason to keep the no-op in the repertoire, which is that a compaction pipeline that never runs cannot break in production at three in the morning. Every summariser call is a model call with its own latency distribution, its own failure modes, and its own capacity to return something structurally invalid on a bad day. A team running rolling summaries on a stateless question-answering product has taken on an availability dependency and a correctness risk for a workload that would have been fine with a hard cut at the oldest turn. The default for a new agent should be to drop, measure how often the agent asks for something it has already been told, and add summarisation when that number is non-zero and rising.

Two properties make the judgement easier to apply than it sounds. The first is whether any turn contains a commitment — a number, a promise, a permission — that a later turn is expected to honour. The second is whether an item is still open, since an unanswered question is the one kind of past event that is entirely about the future. The refund conversation has four such items across forty-one turns. The contract questions have none. The general form of that distinction is the retrieval framing from topic 4.15, on selecting what goes into a request: when the answer to every question can be selected from a document at request time, the transcript is not carrying information and does not need protecting.

The choice between dropping and summarising is a choice about which kind of loss you can live with, and the honest version of it starts from the next turn rather than from the token count. Exactness is what a figure, a granted exception and a still-open question need; continuity is what tone, intent and the shape of a long negotiation need; and a transcript that contains both is served badly by any strategy that treats it as one thing. Pinning the handful of items that cannot be paraphrased costs a few hundred tokens out of tens of thousands, which is why the hybrid is the working answer far more often than the token accounting would suggest.

What the hybrid depends on is that somebody decided which items those were. In the refund conversation they were easy to spot, partly because the whole transcript was about them and partly because the reconstruction happened after the failure, with the answer already known. Doing it live is a different exercise: the compaction runs at turn 41 with no idea which of the preceding turns turn 42 will depend on, and the items that matter are frequently not the ones the conversation is nominally about. A user instruction stated once early and never mentioned again binds every turn after it, and it looks, to a summariser reading sixty turns of transcript, exactly like a thing that happened a long time ago.