Between iterations, only the message list survives

Something has to survive from one iteration of an agent's loop to the next, and in a tool-calling agent exactly one thing does: the message list. That is the ordered array of turns your code sends on every request — a user turn holding the question, an assistant turn holding whatever the model produced, and further user turns holding the results of the tools it asked for. A turn is one entry in that array with a role and a list of content blocks. A tool_use block is the model's request for one of your functions, carrying an identifier the model generated. A tool_result block is your answer to one such request, carrying the same identifier back.

Two bugs follow directly from getting that array wrong, and both present the same way: an invalid request rejected by the provider on a turn where every tool worked perfectly. Neither is a fault in a tool, a prompt or a model. Both are a loop assembling a list that does not obey the grammar the provider validates, and both are produced by code written to be careful about something else — about retries in the first case, about tool failures in the second. That class of bug is the reason to be precise about what an agent carries between iterations, and the answer is short: the whole message list, and nothing else.

Every iteration re-sends the entire conversation#

The model API holds no state between requests. Iteration four does not continue iteration three; it is a fresh HTTP request that happens to contain everything iteration three contained, plus two more entries. The loop function's local variables are not state in any sense the model can see — if a value is not in messages, the model has no access to it, and if a value is in messages, the model sees it on every subsequent iteration whether that is useful or not.

Here is what the list actually looks like partway through a turn, at the point where the model has requested one tool and the result has been appended:

[
  {"role": "user", "content": "Why was I charged twice for AB-40912?"},
  {"role": "assistant", "content": [
    {"type": "text", "text": "Let me look up that order."},
    {"type": "tool_use", "id": "toolu_01Wq8Z", "name": "get_order",
     "input": {"order_id": "AB-40912"}}
  ]},
  {"role": "user", "content": [
    {"type": "tool_result", "tool_use_id": "toolu_01Wq8Z",
     "content": "{\"order_id\":\"AB-40912\",\"charges\":[{\"id\":\"ch_9f\",\"amount\":4200},{\"id\":\"ch_a1\",\"amount\":4200}]}"}
  ]}
]

Three entries, and they come in a fixed grammar. The assistant turn holds the request. The user turn immediately after it holds the answers, one block per request, keyed by the identifier the model generated. Iteration two sends all three of these and gets a fourth entry back; iteration three sends all four plus its results; a turn that resolves in six iterations sends a thirteen-entry list on its final request. On one support assistant those thirteen entries came to about 9,400 input tokens against 1,200 on the first request, which is the ordinary shape of the growth and worth expecting rather than discovering.

Thirteen is not a number to memorise, and the rule that produces it is one line of arithmetic that settles every other case at the same time. The list starts with one entry, the user's question. Each completed round then appends exactly two entries and never more: the assistant turn, and the user turn of results that answers it. So after some number of completed rounds the list holds one entry plus two for each round, and that is exactly what the request following the last round carries. Six completed rounds gives one plus twelve, which is the thirteen above; ten rounds would give twenty-one; a round that has only half happened gives an even number.

Two useful things follow without running anything. The first is a check that costs nothing: an entry count that comes out even means an entry has been lost or a stray one added, and that holds whatever the conversation was about, because the opening user turn is the only entry without a partner. The second is the shape of the token growth. Entries accumulate in a straight line while the cost of sending them accumulates on top of everything sent before, which is why the last request in that turn weighed several times the first without weighing thirteen times the first: every request also carries the system prompt and the tool schemas, and that part is the same size on the first request as on the thirteenth.

The grammar has exactly two rules that the loop is responsible for. The assistant turn goes in exactly as it arrived, with its blocks unmodified and in order — you are storing the provider's own content array, not a summary of it or a re-serialisation of the parts you found interesting. And the very next entry is a user turn containing one tool_result for every tool_use in that assistant turn, matched on id, including the ones that errored, which is the same one-result-per-call requirement that governed parallel dispatch earlier.

Both bugs in this chapter are violations of one of those two rules, and both of them are produced by code that is trying to be careful.

A result with no turn that requested it#

A retry wrapper around the model call produces the first shape. Written out, the ordering fault is easy to see:

# Broken: results are appended before the call that might raise
for i in range(MAX_ITERATIONS):
    response = call_model_with_retries(messages)     # raises on the 3rd 529
    calls = [b for b in response.content if b.type == "tool_use"]
    if not calls:
        return text_of(response)
    messages.append({"role": "user", "content": dispatch_all(calls)})
    messages.append({"role": "assistant", "content": response.content})

The two appends are in the wrong order, which is invisible while every iteration completes: at the end of a successful iteration the list contains both entries either way, and only the interleaving differs. It becomes visible the moment something exits the loop body between them, and a retry that exhausts its attempts on the next iteration is enough — the list that gets re-sent has a tool_result user turn sitting directly after another user turn, with the assistant turn that requested it appended afterwards or lost entirely.

That is worth naming as a shape rather than as one bug, because it is the strongest form a hidden defect takes: a quantity somebody checks holds still while the property that matters varies. The quantity is the contents of the list at the end of an iteration, and it is identical under both orderings — same two entries, same blocks, same identifiers, same tokens on the next request. The property that varies is what the list looks like at the single instant between the two appends, and nothing in a passing test suite ever observes that instant. So the ordering reads as a matter of style, and objecting to it in review feels like fussing, right up until control leaves the loop body in between.

The wrong order is tempting for a reason worth stating plainly. Reading the code top to bottom, the results are the interesting output of the iteration and the assistant turn is bookkeeping, so a writer files the interesting thing first. The protocol reverses that ranking. The assistant turn is the question and the results are the answer, and an answer filed before its question is not an abbreviated record but an invalid one. What is actually true is that the assistant turn is available the instant the response arrives, before any tool has been dispatched, so there is never a moment at which holding it back buys anything.

The same shape arrives by several other routes. Persistence is the most common: a conversation store that writes tool results synchronously as they complete, because they are the expensive part and you do not want to recompute them, while assistant turns are flushed at the end of the loop. A crash between those two writes leaves a stored conversation that cannot be resumed, and the failure surfaces days later when a user replies to a thread. Streaming introduces it too, when the assistant turn is assembled incrementally and the code that appends the results runs off the parsed tool calls rather than off the completed message.

The one-in-thirty failure rate is characteristic and worth reading as a signal rather than a coincidence. A bug in the append ordering does not fire on every conversation; it fires on conversations where something interrupted the loop between the two writes, and the frequency of that is the frequency of provider hiccups, tool timeouts and process restarts combined. It is therefore invisible in development, invisible in a twenty-conversation test suite, and reproducible only by injecting the interruption on purpose. A cheap and effective test is a loop-body fault injector that raises after the results are appended on iteration two and then attempts to resume from the stored conversation; if the resume produces a 400, the ordering is wrong, and no amount of end-to-end happy-path testing will tell you that.

That rate is a sum, and nothing here decomposes it, which is worth admitting rather than leaving implicit. One conversation in thirty is quoted as the combined frequency of three sources, and none of the three was measured on its own. A combined figure of that kind predicts nothing about what fixing any single source would buy, and it cannot be compared against another team's figure, because their mix of the three is different. What would decompose it is a label written at the point the interruption fires: the retry wrapper knows when it has exhausted its attempts, the dispatcher knows when a tool timed out, and a process coming back up knows it restarted. Three counters turn one unexplained rate into three that can be worked on, and until they exist the only honest statement is the sum.

The fix in every case is to make the pair atomic, meaning both entries land or neither does — append the assistant turn the instant the response is in hand, before anything is dispatched, and treat that as the first statement of the iteration rather than a bookkeeping detail at the end. If you persist, persist both entries in one transaction, and if you cannot, persist the assistant turn first, because a conversation with an unanswered call is recoverable and a conversation with an orphaned answer is not.

A call id with no matching result#

The second shape is the mirror image, and its usual cause is an exception handler that logs and moves on:

results = []
for call in calls:
    try:
        results.append({
            "type": "tool_result",
            "tool_use_id": call.id,
            "content": dispatch(call.name, call.input),
        })
    except ToolError as e:
        log.warning("tool %s failed: %s", call.name, e)   # nothing appended
        continue
messages.append({"role": "user", "content": results})

When list_invoices throws, the assistant turn contains two tool_use blocks and the user turn contains one tool_result, and the request is rejected with the complementary message: tool_use ids were found without tool_result blocks immediately after. The instinct behind the continue is sound in most code — one failed lookup should not sink the whole turn — and it is exactly wrong here, because the protocol has no concept of a call that was skipped. Returning {"type": "tool_result", "tool_use_id": call.id, "is_error": true, "content": "invoice service unavailable (503)"} costs one line, keeps the request valid, and gives the model something it can act on, which usually means retrying that one call rather than restarting.

The instinct deserves the same treatment as the ordering fault, because it is also a belief that a defence works. Skipping a failed item and carrying on is the careful move nearly everywhere else: the loop degrades instead of collapsing, and the caller gets partial results rather than an exception. The reason it inverts here is a property of the protocol rather than a matter of taste. The assistant turn has already committed to a set of identifiers, the provider validates that the following turn answers all of them, and there is no encoding for a call that was not attempted — so the graceful degradation produces a request that cannot be sent at all, which is a harder failure than the one it was avoiding. The degradation the writer wanted does exist. It is an error result, and it is the same one line.

Three variants show up often enough to name. Dispatching only calls[0] and ignoring the rest, which works until the day the model requests two tools at once. Filtering results the loop judged uninteresting — dropping empty search results to save tokens is a real optimisation and it must drop the content, not the block. And truncating the list to control context growth by slicing off old entries, which will eventually cut between an assistant turn and its results unless the slicing is aware of pairs.

The third row is the quiet one. It does not always produce an error, which is worse, because a request that reconstructs the assistant turn from the fields the loop happened to care about will pass validation and change the model's inputs in ways nobody is looking for.

The second column of that table is best read as a set of gaps. The first row's gap is an assistant turn: the answers are present and the question that asked for them is not. The second row's gap is a result: the question is present and one of its answers is not. The third row has no gap at all — every entry is where it should be and the contents are wrong — which is exactly why it is the one that gets shipped.

Everything else is not state#

Once the list is understood as the whole of the state, a few consequences follow that are useful in design conversations.

Resuming an agent is a matter of restoring the list and nothing else. There is no session to reattach and no server-side conversation to look up. That is why an agent can move between processes, or be picked up an hour later by a different worker, provided the list was stored intact — and why a database schema that records last_tool_called, step_number, or plan_state is not resumable state, because none of those can reconstruct the request the model needs to see. Teams discover this when a "resume" feature produces a model that has forgotten what it just did, and the fix is always the same: store the list.

The list only grows. Nothing is removed, nothing is summarised, and every result you append is paid for again on every subsequent iteration of that turn. There are real techniques for managing that — pruning old tool results, summarising a long history server-side — and they all amount to editing the list under carefully chosen constraints, which is out of scope here. What matters for the definition is that unbounded iteration means unbounded context, so the fourth part of the agent is a bound on the state as much as on the wall clock.

Because the grammar is mechanical, it is also checkable. A twenty-line function that walks the list and asserts the two rules — every assistant turn with tool-use blocks is followed by a user turn, and the two sets of identifiers are equal — catches all three rows of the table above before the request goes out, and it catches them in the process that built the bad list rather than in a provider error whose message describes the symptom at index four. Running it as an assertion in development and as a logged warning in production costs microseconds against a request that takes seconds, and it converts a class of intermittent, hard-to-reproduce 400s into a stack trace pointing at the code that made the mistake.

Two instruments now report the same defects, and they share no step. The provider's validator runs on the far side of the network, after the request has been assembled, transmitted and paid for in latency, and it reports the symptom by index. The twenty-line walker runs inside your process, before anything is sent, and reports the cause by line number. Different code, different machines, different moment, different inputs — and they agree on the first two rows of the table. That agreement is what licenses treating the local check as a faithful stand-in for the remote one, which in turn licenses moving detection out of production and into a unit test without losing any coverage of those two rows.

The third row is where the agreement runs out, and it is worth being exact about what the walker has to do to reach it. A rewritten assistant turn is structurally valid, so the provider has nothing to object to; and a walker that asserts only the pairing rule has nothing to object to either, since the identifiers still match. Catching that row means asserting the first rule as well — that the stored assistant turn is the provider's own content array — and asserting it requires having the original to compare against. That is precisely what a loop which re-serialises blocks by hand has already discarded. So the check is available to code that keeps the response object until the entry is stored and unavailable to code that does not, and keeping it is a design decision rather than a detail: it is what makes the defence real instead of nominal.

And the list is not memory in the sense people usually mean. It survives one turn of one conversation because your loop keeps it in a local variable; it survives longer only if you write it somewhere. An agent asked the same question tomorrow starts from an empty list unless you built the storage, and everything the model "learned" in those thirteen entries is gone. Anything durable — user preferences, facts discovered last week — has to arrive through a tool that reads it, or by being loaded into the list before the loop starts. Confusing "the conversation is in the context" with "the agent remembers" is the most expensive misunderstanding in this area, because it makes people design memory layers to solve a problem they have not yet demonstrated they have.

The message list being the whole of the state is a stronger statement than it first sounds. It means there is nothing to reattach, nothing on the provider's side to synchronise with, and no hidden place a fact could be hiding. It also means every property of a turn, including the ones that break it, is a property of an array your own code assembled entry by entry. Two rules govern that array, and the two bugs here are one violation each, arriving from a retry wrapper and an exception handler that were both doing the sensible thing for any other kind of loop.

If the list is the whole of the state, then the list is also something you can write by hand. You do not need a model to produce an assistant turn containing a tool call — you need a dictionary with the right keys.