Compaction with tool calling

Compaction is the pass that runs before a call and cuts a transcript down until the request fits the budget, and the simplest version of it — keep the system message, drop the oldest remaining messages one at a time until the request fits — is correct for a plain chat transcript and only for a plain chat transcript. In a chat array every message stands on its own: a user message is valid wherever it appears, an assistant message is valid wherever it appears, and the array is well-formed as long as it is not empty. That property is what makes drop-the-oldest work, and it is a property of chat transcripts rather than of transcripts in general.

The property was never written down anywhere, and that is the whole of the difficulty. Drop-the-oldest is not a heuristic that happens to work most of the time; it is a correct algorithm with a precondition, and the precondition is that removing any message leaves an array the API will still accept. Nothing in the function states it, no test asserts it, and nothing fails at the moment it stops holding — the code goes on running against a transcript that has quietly acquired references in it. The sentence a reader would confidently write after using such a pass is that trimming the oldest message is safe, and they would be right about the system they wrote it for and wrong about the system it is running in six months later. What changed was not the compaction code. It was the shape of the thing being compacted.

This chapter is about what happens to that pass when the assistant gains the ability to call functions. It sets out the referential structure tool calling introduces and the specific rules each provider enforces, replaces the message with the exchange as the unit of eviction — the removal of material from a transcript to get a request back under budget — supplies a twenty-line validator that converts a class of production 400s into test failures, and covers the other paths in an agent loop that orphan a tool call without any compaction being involved. It ends on the remedy that gets proposed first in every incident review and what that remedy actually does to the failure rate.

Tool calling makes messages stop being independent#

An assistant turn that decides to call two tools emits two tool_use blocks, each with an identifier, and the next message must carry a tool_result for each of those identifiers. The pieces reference each other by id, and the API validates the references:

[
  { "role": "user",      "content": "which of my invoices are overdue?" },
  { "role": "assistant", "content": [
      { "type": "tool_use", "id": "toolu_01A", "name": "search_invoices", "input": {"status": "open"} },
      { "type": "tool_use", "id": "toolu_01B", "name": "get_customer",    "input": {"id": "cus_8812f"} } ] },
  { "role": "user",      "content": [
      { "type": "tool_result", "tool_use_id": "toolu_01A", "content": "[{\"id\":\"in_9921\"...}]" },
      { "type": "tool_result", "tool_use_id": "toolu_01B", "content": "{\"name\":\"Acme Ltd\"...}" } ] },
  { "role": "assistant", "content": "Two invoices are overdue: in_9921 and in_9930." }
]

Read that array as a piece of history rather than as four items in a list. The assistant turn in the middle of it is not describing work it did; it is requesting work, and it identifies each request with a string that has no meaning outside this exchange. The message after it is the answer, and it answers by quoting those strings back. So the third message is not merely adjacent to the second — it is unintelligible without it, in the strict sense that toolu_01A refers to nothing at all once the second message is gone.

Those four messages are one unit. Removing the second leaves two orphaned results; removing the third leaves an assistant turn that requested work whose outcome never arrived; removing only the toolu_01B result from the third leaves a partially answered turn, which is rejected just as firmly as removing all of it. The only cuts that produce a valid array are cuts at the boundaries of the whole group. What a violation looks like coming back from the API is this:

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "messages.0: Each `tool_result` block must have a corresponding `tool_use` block in the previous message."
  }
}

Nothing about that request was too long — it can be 47,000 tokens inside a 200,000-token window. It was malformed, because the cut landed at a position between an assistant turn and the results of the tools that turn had requested, and the surviving array began with a tool_result referring to a tool_use that no longer existed.

Why the provider refuses rather than coping is the step worth having, because every rule in this chapter is a consequence of it. The message array is not what the model reads. The provider renders it into one flat sequence of text, and the rendering of a tool_result carries the identifier of the call it answers, since that is how the model is shown which request a result belongs to when a single turn made several. If no matching tool_use appears earlier in that sequence, the rendered prompt states as a fact a piece of history that did not happen — a result came back for a call nobody made. The provider declines to build that prompt at all, which is why the failure arrives as a 400 from the API rather than as a strange answer from the model.

Why the identifier carries that binding, rather than position doing the work, is one more step down and it is a short one. A single assistant turn can request several tools at once, and their results come back together in one message. If results were matched to calls by their order, the loop would have to guarantee that it appends them in the order the calls were made — which it cannot do cheaply when the tools run concurrently and finish out of order. The identifier removes that requirement. Each result names its own call, so the order of the results carries no meaning at all and the loop is free to append them as they arrive. Having given position no meaning, the provider then has no fallback when one half of a pair is missing: there is no second signal it could consult, because the second signal was deliberately not created.

The belief that mechanism displaces is that a 400 on a long conversation is a size problem, and it is tempting for two reasons that are both nearly right. Compaction is a size mechanism, so the code that produced the malformed array was in the middle of a size calculation when it did so. And the failure only ever appears in sessions long enough to have been compacted, which is exactly the population where a window overflow would also live. The distinguishing evidence sits in the error body rather than in the token count: an invalid_request_error naming a tool_result is a statement about structure, and the request carrying it may be nowhere near the window.

The rules differ in their details between providers and agree on the substance:

The last row catches a different version of the same mistake: a truncation applied to the end of a transcript, which happens in retry paths and in code that drops a failed turn, can leave a tool_use as the final message with nothing answering it.

The unit of eviction is the exchange#

The correction is to stop treating the message array as a list of removable items and start treating it as a sequence of atomic groups. A group starts at a user message that is not a tool result and continues through every assistant turn and every tool result that follows it, up to the next such user message. Compaction chooses groups, which makes eviction an all-or-nothing decision taken on one group at a time.

That definition is not arbitrary, and following where it comes from is what makes it portable to a transcript shaped differently. The rule the API enforces is that every tool_result has a tool_use above it and every tool_use has a tool_result below it, so a legal cut is one after which no surviving block has lost its partner. References only ever run between an assistant turn and the message immediately following it, and they never cross a user message that is not itself a tool result — a person typing a new question introduces no identifiers and answers none. Such a message is therefore a position where no reference spans the boundary, which is exactly the definition of a safe cut. The group is not a convention somebody chose; it is the largest span between two consecutive safe cuts, which is why removing one whole group can never break anything and removing part of one nearly always can.

def group_exchanges(messages: list[dict]) -> list[list[dict]]:
    """Split a transcript into atomically removable exchanges."""
    groups: list[list[dict]] = []
    for m in messages:
        starts_new = m["role"] == "user" and not has_block(m, "tool_result")
        if starts_new or not groups:
            groups.append([])
        groups[-1].append(m)
    return groups

def compact(messages: list[dict], budget: int, count) -> list[dict]:
    pinned = [m for m in messages if m["role"] == "system"]
    groups = group_exchanges([m for m in messages if m["role"] != "system"])
    kept: list[list[dict]] = []
    used = count(pinned)
    for g in reversed(groups):                      # newest exchange first
        cost = count(g)
        if used + cost > budget:
            break
        used += cost
        kept.append(g)
    dropped = len(groups) - len(kept)
    if dropped:
        log.info("compacted", exchanges_dropped=dropped, exchanges_kept=len(kept))
    return pinned + [m for g in reversed(kept) for m in g]

Two consequences follow from group-level eviction and both are worth accepting deliberately. Granularity gets coarser: an exchange containing a 9,000-token tool result is now all-or-nothing, so freeing 400 tokens may cost 9,000. And the budget arithmetic becomes lumpy, which means a compaction pass can overshoot and leave the prompt smaller than necessary. In an agent transcript where exchanges average 2,400 tokens and the largest hold a 9,000-token file read, closing a 400-token deficit removes a whole exchange, so the average compaction frees six times more than it needed to — 2,400 divided by 400. Over a long session that overshoot compounds and the transcript ends up substantially smaller than the budget allows, which costs nothing in correctness and does mean the model is working with less history than it could have. The fix is not finer cuts but cheaper representations of the same exchanges, which is what the next chapter's digests are for.

That six is the ratio worth tracking whenever the average exchange size and the typical deficit are both known, because it is what the design is really costing: it says how many tokens of history a correctly written compaction pass discards for reasons that have nothing to do with the budget it was closing. A deployment whose exchanges are small relative to its deficits pays almost nothing for group-level eviction. One whose exchanges carry large tool results pays a great deal, and pays it on every compaction rather than occasionally.

Whatever the policy, the request should be checked before it leaves. A validator is twenty lines, runs in microseconds, and converts a class of production 400s into a failure that appears in tests:

def assert_well_formed(messages: list[dict]) -> None:
    open_ids: set[str] = set()
    for i, m in enumerate(messages):
        results = {b["tool_use_id"] for b in blocks(m, "tool_result")}
        if results - open_ids:
            raise MalformedTranscript(f"orphan tool_result at {i}: {results - open_ids}")
        if open_ids - results and results:
            raise MalformedTranscript(f"partially answered call at {i}: {open_ids - results}")
        if open_ids and not results:
            raise MalformedTranscript(f"unanswered tool_use before message {i}")
        open_ids = {b["id"] for b in blocks(m, "tool_use")}
    if open_ids:
        raise MalformedTranscript(f"trailing unanswered tool_use: {open_ids}")

The whole of that check is one variable. open_ids holds the identifiers the previous message requested and nothing else, because the rule is about adjacent messages rather than about the array as a whole — a call answered three messages later is as invalid as one never answered. Each iteration therefore compares this message's results against the previous message's calls, and then replaces open_ids with this message's own calls, so the set never has to grow. The three exceptions inside the loop are the three ways that comparison can come out wrong: results with no calls behind them, calls with only some of their results, and calls with no results at all. The fourth check is outside the loop because a trailing unanswered call has no next message to be compared against.

Run it in the request path behind a flag that raises in development and logs in production, since a malformed array the provider would reject is not made better by your own exception, and the log line tells you which compaction path produced it.

Compaction is not the only thing that orphans a call#

Once a validator exists, it starts catching arrays that compaction never touched, and the other producers are worth knowing because they are fixed in the tool-execution path rather than in the budget path.

The most common is a tool that raises. The agent loop appends the assistant message with its tool_use blocks, calls the function, the function throws, the exception propagates to a handler that logs and continues the loop — and no tool_result was ever appended. The next request has a trailing unanswered call. The same thing happens when one of three parallel tools times out and the loop appends only the two that returned, when a user cancels mid-turn and the partially built exchange is persisted, and when a retry after a network error re-sends a request whose assistant message was written to the store but whose results were not.

Every one of those is repaired the same way, by making the result mandatory rather than conditional on success:

async def run_tools(assistant_msg: dict) -> dict:
    results = []
    for call in blocks(assistant_msg, "tool_use"):
        try:
            out = await asyncio.wait_for(dispatch(call), timeout=30)
            results.append({"type": "tool_result", "tool_use_id": call["id"],
                            "content": truncate(out, 4_000)})
        except Exception as e:                       # timeout, crash, bad args
            results.append({"type": "tool_result", "tool_use_id": call["id"],
                            "is_error": True,
                            "content": f"{type(e).__name__}: {e}"[:400]})
    return {"role": "user", "content": results}      # always one per call

Returning an error as a tool_result rather than omitting it keeps the array valid and gives the model something to work with, which in practice is a large behavioural improvement on its own: an agent told that read_file raised FileNotFoundError: src/billing/ledgr.py corrects the typo and retries, whereas an agent whose call vanished has no idea anything went wrong. The truncate call on the success path matters for a different reason — a tool that returns 60,000 tokens of output puts the transcript over the window in a single step, and truncating at the boundary where the result enters is the only place you can do it without losing the pairing.

The pairing is what fixes the timing. Once a tool_result is in the transcript it is bound by id to a tool_use in the message before it, and shrinking it later means editing a message another message refers to. That is possible, but it has become a transcript-rewriting operation rather than a string operation, and one that has to be redone on every subsequent call. At the moment the result is constructed nothing refers to it yet, so a bound can simply be applied. The reason a bound is needed at all is that a tool result's size is decided by whatever the command happened to print, which is outside the agent's control and outside any model of how the transcript grows: one directory listing over a large repository can put more tokens into the array in a single step than the thirty steps before it did between them.

Raising the threshold changes the rate, not the failure#

The response that comes up first in an incident review is to compact later — trigger at 90 percent of the window instead of 70, or move to a model with a larger window so compaction fires less often. Both make the symptom rarer, and the arithmetic of how much rarer is what shows they are not fixes. Raising the trigger to 90 percent lowers the share of sessions that compact from 12 percent to perhaps 4, which turns one session in thirty-one into one in ninety-four. The failure has not changed in nature, its rate has dropped by a factor of three, and it now occurs only in the longest sessions, which are the ones hardest to reproduce and most expensive to lose. A 1,000,000-token window pushes it to one session in several thousand, at which point it becomes a mystery ticket that reappears every few weeks.

That second rate comes from the same multiplication with one term changed. Four sessions in a hundred compacting, against the same 27 bad cuts in a hundred, is a little over one session in a hundred, and one divided by that is about ninety-four. The factor of three is the ratio of the two compaction rates and nothing else, twelve over four, because the 27 percent did not move. Nothing about the transcripts changed. Only how often the code cut them.

Dropping one message is what causes the structural violation, and its probability is therefore proportional to how often you drop messages rather than to how long the transcript is. Anything that reduces the rate of dropping reduces the rate of the bug without touching it, which is exactly why the failure survived six months of chat-only traffic: chat sessions dropped messages constantly and had no referential structure to break.

A well-formed array is not a good array#

The validator accepts a transcript that has lost the exchange in which the user stated the account number, and the API will accept it too. Structure is a syntactic property and everything this topic has been building toward is a semantic one.

There is a second symptom that group-level eviction does not address, and conflating the two is why incident reviews on this go badly. Alongside the 400s, an agent will begin occasionally repeating a tool call it already made earlier in the same session — running the same search, fetching the same file — and that behaviour is not a structural violation. It is the correct behaviour of a model whose transcript no longer contains the result, because a whole exchange was evicted, and the model has no way to know that it once knew the answer. Evicting by group makes the API accept the request; it does not decide that a tool result the agent still needed was the wrong thing to remove.

The same distinction applies to summarisation-based compaction, which topic 2.3 covers as a way of preserving old turns. Replacing exchanges 1 through 20 with a summary message is exactly as capable of orphaning a tool_result as deleting them, so the summariser has to respect the same group boundaries, and the summary message it produces must be a plain user or assistant message rather than anything carrying tool blocks.

Tool calling turns a message array into a structure with references in it, and a compaction pass written against the older assumption produces requests that are well inside the window and rejected anyway. The exchange is the unit that survives that: a group starting at a real user message and running through every assistant turn and tool result until the next one, removed whole or not at all. A validator in the request path is what stops the rest of the agent loop from reintroducing the same defect through a timeout, a cancellation or a retry, and it costs twenty lines and microseconds per call.

What none of it does is choose. Given a set of removable groups and a deficit of 4,000 tokens, something has to decide which groups go, and the API has no opinion on that at all. That choice is where answer quality is won or lost, and it is the last thing this topic has to settle.