History compaction and task boundaries

Conversations outgrow the context window, so most long-running agents acquire a piece of middleware that trims history: keep the recent turns intact, strip everything else down to what matters. Written before reasoning existed, that filter is usually a whitelist of block types — keep text, keep tool calls and results, drop the rest — and it is entirely correct for the great majority of turns.

It stops being correct on a turn that is part of an unfinished task. Chapter 5 established that an assistant turn containing a tool call is validated and continued as a unit, so removing a block from one is not trimming context; it is handing the provider a turn that does not match what the provider emitted, in the middle of work the model is still doing. This chapter is about the two pieces of knowledge a compactor needs to avoid that — which blocks are load-bearing, and where the current task begins — and about a security rule that reliably gets misapplied to the same code.

One belief lets this defect through review: that a compactor is a context optimisation, plumbing that trims what gets sent, motivated only by size. A model holds nothing between calls, so the message array is not a record of the conversation but the whole of the input the next completion is computed from. Editing it edits the question, and the only open issue is whether the part removed was load-bearing for the answer.

Why the same edit is safe on a chat turn#

Dropping a thinking block from an ordinary assistant text turn leaves a well-formed turn behind. The assistant said something; the deliberation that preceded it has no dependent that needs it; the provider has no reason to look for it and the model has nothing outstanding to continue. That is the great majority of turns in any conversation, which is why a compactor with that rule can run for months without incident and why it passes review.

The chain under that is short enough to check link by link. A model is called with a transcript and answers from it, so an edit to the transcript changes what it is answering. Deliberation attached to a finished answer has nothing depending on it: the answer it produced is already in the transcript, and the work it belonged to is over. Deliberation attached to an unanswered tool call is the plan the next step continues from, and chapter 5's rule — an assistant turn carrying a tool call is validated and replayed as the unit the provider emitted — means it is checked against what was issued, not against what looks well formed.

A tool turn is a different object, and the difference is not visible in the block types alone. A tool_use block whose result has already been answered by a final assistant message belongs to a finished task and can be trimmed like anything else; the same block in a task still in flight cannot. Block types do not tell you which of those you are holding. Finished is the whole of the difference, and finished is not a property of a block: it belongs to the span the block sits in, which either ends in a settled assistant turn or does not. Two identical thinking blocks can therefore have opposite fates, decided by what comes after them, and a filter reading only a block's type is reading a field that cannot carry that fact.

KEEP = {"text", "tool_use", "tool_result"}

def compact(history, keep_turns=10):
    old, recent = history[:-keep_turns], history[-keep_turns:]
    trimmed = []
    for message in old:
        blocks = [b for b in message["content"] if b["type"] in KEEP]
        if blocks:
            trimmed.append({**message, "content": blocks})
    return trimmed + recent

Read that with chapter 5's rule in mind and the defect is a single line. KEEP is a whitelist of block types, applied without reference to whether the turn it is filtering has an unanswered tool call in it, which is the only question that matters.

The arithmetic of the window decides how often the defect fires and why a test suite tends to miss it, so it is worth doing rather than sketching. Real conversations reach ten turns constantly, and a single agentic task inside one contributes two messages per tool step — the assistant turn carrying the call and the user turn carrying the result back. Multiply that against a window of ten: a six-step task is twelve messages on its own, two more than the compactor promises to leave alone, and the two that fall outside are that task's own earliest turns.

Two messages per step means half a window's worth of steps fills the window exactly, so any task longer than half the window is guaranteed to have its earliest turns rewritten while it is still running, and the longest tasks cross that line soonest and lose the most once they do. A window of ten turns reads as a conservative setting, which is the second belief this defect lives inside: it is conservative counted in turns and aggressive counted in tasks, where ten turns is five tool steps, and nothing in the code names tasks. Regression scenarios that run three or four tool steps each are eight messages at most, which never reaches ten, so staging loads the compactor and never invokes it. A compactor measuring in turns while the invariant it needs to respect is stated in tasks will start rewriting messages that belong to the request currently in flight, and it will do so first on the longest tasks.

A security rule about a different thing#

There is a second obstacle to fixing this, and it is usually the one that costs the most days. A security review will have concluded, in one sentence, that raw model reasoning must never be persisted or exposed. That sentence is right, and the reasoning behind it is sound: deliberation is unreviewed generated content that may restate personal data from the prompt, may contain discarded hypotheses about a customer stated in blunt terms, and has been through none of the filtering the visible answer goes through. Nobody wants to be the engineer who proposes writing it to a durable store.

The belief that stalls the fix is that a sentence forbidding persistence forbids holding the value at all. It is tempting because the sentence is short and because "in memory" appears in it as neither exception nor prohibition, so a reader under pressure cannot tell whether the omission was a decision.

What the sentence does not say, and what such a review is not asked about, is how long a value may live in the memory of the process that received it. The exposure being protected against has a shape: content escaping into places where people, other systems, or long-lived storage can reach it. Handing a block back to the API that generated it thirty seconds earlier is not in that shape. Neither is holding it in a request-scoped object for the duration of one task. Writing it to the conversation table, shipping it to the log aggregator, or including it in a support console payload very much is.

The test that shape implies is reachability, not medium — not what the value is held in, but whether a path exists from where it sits to a person, to a system other than the one that produced it, or to a store that outlives the work. A request body going back to the provider adds no reader, and a request-scoped object adds none either, since the only path out of it is code you wrote and there is none. A log line fails despite carrying less, because fan-out is the property being measured. Each row below is that one question asked of a different destination.

The productive move is to replace the one-sentence rule with a table of surfaces, because the rule is unarguable and useless while the table is arguable and actionable:

The third row is the one that needs a decision rather than a default. An agent whose tasks run inside a single request handler never needs reasoning to leave memory, and that is the easy case. An agent whose tasks run on a queue, where a worker can pick up step four of a task that step three ran on a different machine, needs the blocks in a store both workers can read. That store should be the task record rather than the conversation record, encrypted at rest, excluded from the pipeline that copies operational data into the warehouse, and cleaned up when the task reaches a terminal state — which is also, conveniently, the moment the provider stops needing the blocks.

The answer that comes up first in this discussion is to turn reasoning off for the agent, on the grounds that if the blocks cannot be stored the feature cannot be used. It makes the rejections stop within a deploy and it reads as the cautious choice, which is why it gets proposed. It also removes the capability from the surface with the strongest case for it, since multi-step tool use is where deliberation does the most work — the same agent without it made 5.1 tool calls per completed task instead of 3.4 — and it does so to satisfy a rule about what users and logs can see by changing what the model does. The mismatch is worth stating precisely: the rule's subject is a set of destinations — stores, logs, screens, warehouses — and what it constrains is what may arrive at each of them. A model configuration is not one of those destinations; it decides what gets produced, not where the product may go. The scope of the rule is the thing to fix, not the model configuration.

What the compactor has to become#

The replacement needs the two pieces of knowledge the original lacked. The harder one, the task boundary, has a clean definition: a task is finished when an assistant turn contains a final answer with no unanswered tool call, so the boundary is the last such turn in the history.

The instinct to resist first is finding that boundary by counting — keep the last so-many turns, raise the number until the errors stop. How many messages a task occupies is set by how many tool steps the model takes at run time, which is not knowable when the middleware is configured, so any fixed count is a bet that no task runs longer than it, and the bet is lost silently, on the longest tasks, in production.

def last_settled_index(history):
    """Index just after the most recent assistant turn that ended a task."""
    for i in range(len(history) - 1, -1, -1):
        message = history[i]
        if message["role"] != "assistant":
            continue
        types = {b["type"] for b in message["content"]}
        if "tool_use" not in types and "text" in types:
            return i + 1
    return 0


def compact(history, keep_turns=10):
    boundary = last_settled_index(history)
    live = history[boundary:]                    # in-flight task: never rewritten
    settled = history[:boundary]

    trimmed = []
    for message in settled[-keep_turns:]:
        blocks = [b for b in message["content"]
                  if b["type"] not in ("thinking", "redacted_thinking")]
        if blocks:
            trimmed.append({**message, "content": blocks})
    return trimmed + live

Three properties of that version are worth naming. The live segment is returned untouched, so no in-flight task can be corrupted no matter how long it runs — which means a fifty-step task is exempt from compaction entirely, and if that threatens the context window the answer is a task-level budget rather than a middleware that edits history under the model. The filter is now a denylist of reasoning types rather than a whitelist of known-good types, so a block type the provider adds next quarter is carried through instead of silently dropped. And reasoning is removed only from settled history, where chapter 5's rule says it has no remaining obligation.

The second of those has a reason under it. A whitelist decides a block's fate by whether the filter's author had heard of its type, which is a fact about the day the code was written; a type introduced after that day fails the question and is dropped without an error, because dropping is the filter working as designed. A denylist names the types that must go, so the same future block answers no and survives. The cost is the mirror image: a type that ought to be removed will be carried until somebody adds it to the list.

The definition of settled implemented above is also narrower than the English word. Reading an assistant turn as final when it carries text and no tool_use assumes the transcript is one conversation advancing one task at a time, in the order the provider saw it. A history interleaving two tasks, a text-only turn written as a progress note rather than an answer, or a transcript rebuilt from a store that does not preserve order will each put the boundary inside live work.

Tasks that never settle#

The exemption in that design is also its honest limitation, and it should be stated rather than discovered. A task that never settles is a task that is never compacted, so an agent running forty tool steps against large tool outputs will walk into the context window with nothing standing in its way. The trade is exact and worth stating as a trade: what the exemption buys is that in-flight work is never rewritten, and what it costs is any bound at all on how large in-flight work can grow, since the one mechanism that shortens the transcript has been told to keep away from the span that is expanding.

History compaction is the wrong instrument for that, because the thing consuming the window is the current task rather than the conversation around it. The instruments that do work operate inside the task: cap the number of steps, truncate or summarise individual tool results before they enter the transcript — a 40,000-token API response reduced to the fields the model asked for — and give the task a token budget that ends it with a partial answer rather than a provider error. Those are decisions about one task's shape, made by code that knows what the task is, which is exactly the knowledge a generic history middleware does not have. The missing knowledge is specific: cutting a tool result down to the fields that matter requires knowing which fields were asked for and what the next step will do with them, and a middleware holding a list of messages can see that a message is large without seeing whether the large part is the answer.

Some providers now offer compaction as a server-side feature — context-management strategies that clear tool results or thinking blocks according to rules the provider maintains, and server-side summarisation of older turns. Where it is available it is worth preferring, for the same reason you prefer the platform's TLS stack to your own: the invariants are the provider's to state and they change on the provider's schedule. That preference carries a condition, and it is the question this chapter has been asking: a strategy that clears tool results by age or position is stated in the same currency as the compactor above, so if it can reach into a task still in flight it carries the same defect under the provider's name. The task-boundary logic above is what you write when you are managing history yourself, and it doubles as a specification for what to check the provider's version does.

The general lesson is about where an invariant is stated. A compactor expressed in turns and block types cannot see the thing it must not break, because "unfinished task" is not a property of any single message — it is a property of a span of messages ending, or not ending, in a settled assistant turn. Any code that rewrites conversation history needs that span, and the cheapest way to get it right is to make the live span untouchable and do all trimming behind it.

That settles where reasoning may live inside the system. It says nothing about what leaves the system. The same conversation object is serialised for a support console, streamed to a browser, and written to an audit record, and each of those is a separate filter written by a different team — usually one that has never heard of block types and is deciding what to keep by picking the field that looks like the answer.