Trimming conversation history

Trimming is the operation that keeps a conversation's request body bounded: before sending, drop the oldest entries from the array so that what goes out is a suffix of the history rather than all of it. It is the standard answer to the quadratic input growth of the previous chapter, it does bound the cost, and the naive implementation of it — keep the last k messages — introduces a failure mode that the naive implementation of nothing else in this topic does.

The reason is a constraint established earlier: when a model calls a tool, the call and the result that answers it are two separate entries in the array, bound by an identifier the provider issued, and the provider validates that binding on every request. A tool_result whose id matches no tool_use earlier in the same array is rejected before any inference runs. A count of array entries is oblivious to that pairing, so a window that keeps the last eight entries will, on some conversations and not others, begin the array with an answer to a question that is no longer in it.

Where the cut lands#

Consider a conversation that is twelve entries long, where entries 6 and 7 are the two halves of a lookup. Keeping the last eight entries means starting at index 4, which is fine. Two turns later the same window starts at index 6 — still fine, by luck, because it starts on the assistant entry that issued the call. One turn after that it starts at index 7, and the array you send begins with a tool_result block answering a tool_use block that is no longer present:

index  role       content
  4    user       "and what about the express option?"
  5    assistant  text
  6    assistant  text + tool_use(id=toolu_01A9bC)     <-- the call
  7    user       tool_result(tool_use_id=toolu_01A9bC) <-- the answer
  8    assistant  text
  ...
      window of last 8 starts here ---------^ (cuts between 6 and 7)

The provider rejects it before running anything:

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "messages.0: tool_result block with tool_use_id 'toolu_01A9bC' has no corresponding tool_use block"
  }
}

Nothing about this is probabilistic, which is why it feels random. Whether the cut lands on a safe index or an unsafe one depends on how many entries have accumulated since the tool call, and that depends on how the user happened to behave — so the same code path fails for one customer and works for the next, and fails for the first customer again if they retry. Short conversations never trip it because the window never engages; a three-turn smoke test never trips it for the same reason.

The direct fix is not to reach for a smarter number. Eight is wrong for the same reason ten was wrong in the previous chapter's arithmetic: a count of entries is a unit that knows nothing about what the entries contain or which of them are bound to each other. Raising it to twenty moves the failure from 1.8% of conversations to 0.4% and makes it harder to reproduce.

That improvement is worth examining, because it is the kind that looks like a fix and is not. The window start is the array length minus the window size, so the window only engages once a conversation is longer than the window. Doubling the window roughly halves the proportion of conversations long enough to engage it, and then halves it again for the ones that engage it only briefly, since a conversation that is one entry over the limit has exactly one cut position to get wrong. What has not changed at all is the chance that any given cut lands on a tool result. A cut in a tool-using conversation is as likely to orphan a pair under a window of twenty as under a window of eight, because the tool exchanges are no rarer and the cut is no smarter.

So the larger number buys nothing except a smaller sample of the same defect, and it spends something real to do it: the fraction of conversations that fail falls into the range where a dashboard reads it as noise, and the reproduction that used to take a morning now takes a week. The lever that matters is not the size of the window. It is what the window is allowed to cut through.

Reading a malformed array from the error alone#

Before rewriting the trimming logic it is worth being able to name these failures on sight, because the error text is unusually informative and most teams skim past it. Nearly every validation error names the offending index, and the index plus the message is normally enough to identify the bug without opening a debugger.

Two habits make this table pay off. Log the full error body, not just the status code, because a dashboard that records 400 and discards the message throws away the index that identifies the bug. And log the array's length and the roles of its first and last three entries alongside it, so that a report of messages.0 can be matched against what position zero actually was.

Trimming on a budget, at boundaries#

A window that works measures in tokens, because tokens are the resource being bounded and the unit the context limit is expressed in, and it cuts only at positions where the array remains structurally valid. Walk backwards from the newest entry accumulating a token estimate, stop when the budget is exhausted, then move the start forward until it lands somewhere legal:

def trim(entries, budget_tokens, count):
    """entries: oldest-first list of stored turns. count: your token estimator."""
    kept, used = [], 0
    for entry in reversed(entries):
        size = count(entry)
        if used + size > budget_tokens and kept:
            break
        kept.append(entry)
        used += size
    kept.reverse()

    # Snap forward to a legal start: a user entry that is not a tool result.
    while kept and not is_clean_user_turn(kept[0]):
        kept.pop(0)
    return kept

def is_clean_user_turn(entry):
    if entry["role"] != "user":
        return False
    blocks = entry["content"]
    if isinstance(blocks, str):
        return True
    return not any(b.get("type") == "tool_result" for b in blocks)

The snap-forward loop is what the message-count version was missing, and it enforces two invariants at once. The array must begin with a user turn, and the first entry must not be a tool result, because a tool result at position zero is by definition an orphan. Dropping entries from the front until both hold is safe, cheap, and costs at most a handful of turns of context.

The first of those invariants looks like a formality and is not, and knowing why is what stops somebody removing the loop in a refactor. The array is rendered into a sequence of speaker spans, and the model's job is to continue it. An array starting with an assistant entry renders as a document in which the assistant spoke first, unprompted, about a subject nobody raised — a shape that essentially never occurs in the material these models were trained on. A provider that accepts it is not doing you a favour; it is handing the model a document with no precedent and asking it to continue. The rejection is the provider declining to run inference on a transcript that could not have happened, which is the same principle behind the orphan check rather than a second, unrelated rule.

The estimator matters more than it looks. Counting characters and dividing by four is fine for a budget with slack in it and badly wrong for code, JSON, or non-English text, where the ratio can be half that. If you are sizing the window close to the model's limit, use the provider's token-counting endpoint on the assembled array rather than an approximation, and treat the local estimate as a fast pre-filter.

The ratio moves for a reason that predicts which of your traffic will break it. A tokenizer is a fixed inventory of text fragments, built by finding the character sequences that recur most often in a body of text and giving each one its own entry. Common English words earn an entry apiece, so a word of several characters costs one token, and the characters-per-token ratio comes out around four. Text that does not look like that body of text gets no such bargain: tool_use_id, a UUID, a run of braces and quotes, an accented language written in a script the inventory covers thinly — each of those is assembled from several small fragments, so the same number of characters buys fewer characters per token. The estimator does not degrade gracefully as content shifts. It degrades exactly on the conversations that contain pasted logs and structured payloads, which are the conversations already closest to the ceiling.

The budget itself has to account for what is not in the array. The instruction field and the tool declarations are separate parts of the request body, they are never trimmed, and they count against the same context limit — so the budget you hand to the walk-backwards loop is the model's limit minus the instruction, minus the declarations, minus the space you are reserving for the reply. On a request with a 900-token instruction and 500 tokens of tool schemas, those two alone come to 1,400 tokens that the loop never sees and the model always reads. So forgetting that subtraction is an overshoot of exactly that size, present on every request, and it surfaces as an intermittent failure on the longest conversations for the same reason as everything else in this chapter: the overshoot only matters once the rest of the array has grown enough to use up the slack that was hiding it.

A refinement worth knowing is the head-plus-tail window: pin the first user turn, which often carries the task setup or the customer's original problem statement, and take the token budget from the newest end. It preserves the most useful early context for a modest fixed cost. The two things to get right are the same as before — the pinned head must itself be a legal start, so pin a user turn and never a tool result — and the seam. Splicing turn 1 directly onto turn 24 produces a conversation with a hole in it, and the model reads it as continuous; a one-line bracketed marker in the first entry of the tail, saying plainly that earlier turns were omitted, costs a few tokens and stops the model treating a twenty-turn gap as something that just happened.

Order the steps deliberately, because several of them interact. Assemble from the store, drop empty and soft-deleted entries, merge consecutive same-role entries so the roles alternate the way providers expect — user, then assistant, then user, with two stored user turns arriving side by side counting as a malformed array rather than a cosmetic wrinkle — then trim to budget, snap the start forward, validate, and send. Merging after trimming can undo an alternation the trim was relying on; validating before snapping reports problems you were one step away from fixing; counting tokens before dropping empties counts tokens you are about to discard.

Whatever you build, validate before you send. A dozen lines standing between your array and the wire converts a class of production 400s into a caught assertion in CI:

def validate(messages):
    assert messages, "empty array"
    assert messages[0]["role"] == "user", "first entry must be user"
    open_calls = set()
    for i, m in enumerate(messages):
        for b in m["content"] if isinstance(m["content"], list) else []:
            if b["type"] == "tool_use":
                open_calls.add(b["id"])
            elif b["type"] == "tool_result":
                assert b["tool_use_id"] in open_calls, f"messages.{i}: orphan result"
                open_calls.discard(b["tool_use_id"])
    assert not open_calls, f"unanswered tool calls: {sorted(open_calls)}"

That is the same check the provider runs, executed where you can see it, and it turns a customer-visible failure into a log line with a conversation id.

Testing a window against real conversation shapes#

Detecting it before the users do is a matter of testing the window against the shapes production actually produces. The reason that failure reached customers is that the window was tested on the wrong corpus. Nothing in the test suite had a tool call in it, so nothing in the test suite could produce the failure. Three cheap measures close that gap.

Replay real conversation shapes offline. Take a few thousand stored conversations, run the trimming function over every prefix of each — turn 1, turns 1–2, turns 1–3 — and run the validator on each result. This is pure computation with no API calls and no cost, it exercises every cut position that production will ever hit, and it is the single highest-value test in this area. Had it existed on the Monday, it would have flagged the orphan on the first tool-using conversation in the sample.

The claim that it covers every cut position is stronger than it sounds and rests on two properties, both worth checking rather than assuming. The trim function is a pure function of the array handed to it — which is available to you precisely because the endpoint is stateless and the array is therefore the whole input — so enumerating every prefix of a stored conversation enumerates every array that conversation could ever have produced. And a stored conversation is drawn from the same traffic as a future one, so the shapes in the sample are the shapes users make. The honest limit of the method is on that second property alone: it cannot exercise a conversation shape that has never happened, so a feature that introduces a new shape — a second tool, a retrieval step, an interface that lets a user edit an earlier turn — needs its shapes added to the corpus on the day it ships rather than after it has failed intermittently in production for a week.

Alert on 400s split by conversation shape. A flat 400 rate is a number nobody investigates until it is large; a 400 rate broken down by "did this conversation use a tool" turns 1.8% overall into 11% of tool-using conversations, which is unignorable. The same breakdown by turn index tells you whether the failure begins exactly where the window engages.

Watch the window's own behaviour as a metric. Emit the number of entries dropped per request and the position the window started at. A distribution that suddenly shifts after a deploy — because someone changed the budget, or because average turn size grew when a retrieval step started injecting documents — is the earliest possible warning, and it arrives before any request fails.

The context a window discards#

Trimming is lossy and no amount of care makes it otherwise. Anything outside the window is gone as far as the model is concerned, so a customer who gave their order number in turn three and asks a follow-up in turn thirty gets asked for it again. That is a product decision dressed as an infrastructure one, and it should be made by someone who knows which facts matter.

The usual next step — summarise the dropped prefix and prepend the summary — is genuinely useful and introduces two problems of its own. A summary is a lossy rewrite performed by the same class of system that is unreliable about details, so numbers, ids and negations are exactly what it tends to lose. And a summary replaces the entries it covers, which means any tool call inside that span is now described in prose while its id has vanished from the array, so summarisation must respect the same pairing constraint as trimming: summarise whole exchanges, never half of one. It also destroys the cached prefix, which as the previous chapter showed can be the more expensive of the two effects.

There is no policy that keeps everything, costs little and never fails. There is a policy that fails loudly in CI instead of quietly in production, and it is the same four steps every time: measure the window in tokens rather than entries, cut only where the array stays valid, run the provider's own validation locally before sending, and replay the whole thing against real conversation shapes so that the cut positions your users will produce are the cut positions your tests already produced. None of those four is expensive, and the first three fit in about forty lines.

One category of failure survives all of it, because it happens after the array is already correct. A worker builds a valid request, sends it, and never sees the response — the socket times out, the pod is evicted, the deploy rolls — and a queue does the sensible thing and runs the job again. The second attempt sends a byte-identical array, which is the one thing this topic has taught you to expect to behave identically, and that is precisely the problem.