Debugging history assembly

History assembly is the step that turns stored turns into the array a chat request carries: a read scoped to one conversation, ordered by position, with a system message prepended and whatever compression the product applies. It runs on every call, it is usually fifteen lines long, and it is where almost every report of the form "the assistant forgot what I told it" actually originates. The two hypotheses — a wrong array, or a model that failed to use a correct one — are cheap to separate, because one of them leaves an artefact you can read.

That artefact exists because of a property of the chat endpoint rather than because anybody arranged for it. The provider retains nothing between calls, so the array your handler serialises is not a hint about what the model knew. It is the whole of what the model knew, and there was no second source. Anything the assistant could have used was in that value, and anything absent from it was unavailable at any price. One serialised array therefore settles one of the two hypotheses outright: if the order number is not in it, no claim about model quality can be true of that conversation, and if it is in it, the assembly hypothesis is dead.

The reason to reach for the assembly hypothesis first is that the two candidates fail in different shapes. Assembly is a handful of lines with several independent ways to go wrong — a filter clause, an ordering clause, a row limit, a soft-delete predicate, a prepended system message — and each of those, when wrong, is wrong on every call and removes material completely. Attention degrades rather than fails: it is a tendency that varies with position and length, it is measured across many inputs rather than observed in one, and it leaves the material in place. A report that one specific fact is missing every time from one specific conversation has the shape of the first kind of failure.

This chapter is about producing that artefact before anybody asks for it, and about the four or five bugs it exposes. Debugging apparent memory failures by reasoning about model behaviour is like debugging a page that renders wrong by reading the browser's layout algorithm before viewing source. The source is right there, it is one artefact, and it settles the question in a minute — but only if assembly was instrumented at the time, because by the time a ticket arrives the conversation is three days old and the request body was never kept.

The comparison to viewing source is an analogy, and the place it breaks is worth naming. A browser retains the document it rendered, so the equivalent of view-source can be run long after the fact and still answer the original question. Re-running assembly against the same conversation id looks like that move and is not, because assembly is a function of rows that have since changed — turns appended, a message edited, a row soft-deleted — and of code that has since been deployed over. Today's run answers what you would send now rather than what you sent then, and the two questions have the same shape and different answers.

Instrument assembly so the artefact exists before you need it#

The trail assembly needs to leave has five fields, and each of them answers a different question about a conversation that went wrong.

def assemble(conn, conversation_id: str) -> list[dict]:
    rows = conn.fetch(
        "SELECT id, seq, role, content FROM messages"
        " WHERE conversation_id = $1 AND deleted_at IS NULL"
        " ORDER BY seq ASC",
        conversation_id,
    )
    messages = [SYSTEM_PROMPT] + [{"role": r["role"], "content": r["content"]} for r in rows]

    log.info(
        "assembly",
        conversation_id=conversation_id,
        message_ids=[r["id"] for r in rows],
        seq_range=(rows[0]["seq"], rows[-1]["seq"]) if rows else None,
        count=len(messages),
        input_tokens=count_tokens(messages),
        assembly_revision=ASSEMBLY_REVISION,   # bumped whenever this function changes
        fingerprint=payload_fingerprint(messages),
    )
    return messages

The message id list is what turns a vague report into a diff. Two consecutive turns of a healthy conversation log id lists where the second is the first plus two new ids, and any deviation from that pattern is a bug with a name: an id that vanishes between turns is a filter problem, an id that appears twice is a duplicate write, an id that belongs to another conversation is a scoping problem, and a list that grows by four instead of two means something wrote turns you did not expect. The seq_range catches the case where the list looks plausible but starts at 17 because a limit clipped the front. The revision field exists so that when you compare a conversation from March against one from May you know whether the same code produced them, which matters enormously once somebody asks you to reproduce a past request exactly — chapter 8 leans on it hard.

The pattern of growing by two is worth deriving rather than memorising, because a deviation from it only means something once the baseline is understood. One turn writes exactly two rows: the user's message, inserted before the model call, and the assistant's reply, inserted after it. Assembly runs between those two writes, so the array it logs holds every user row up to and including the current one and every assistant row except the one that does not exist yet. Between one turn's log line and the next, exactly two rows have therefore appeared — the previous turn's reply and this turn's question — and the second id list is the first with two ids appended to its end. A list that grew by four means two turns were written where one was expected, a list that grew by one means a row that should have been written was not, and a list that shrank means something the earlier call could see is being excluded now.

The sequence range earns its place beside the ids for a reason the count cannot cover. A count is a single number and every plausible conversation has one, so an array of fifty messages reads as healthy whether those fifty are the first fifty of the conversation, the last fifty, or an arbitrary window from the middle. The range names both ends, and the front end is the one carrying the opening complaint and the order number. An array whose range begins at 17 is not a short conversation. It is a long conversation with its beginning removed, and those two situations are indistinguishable in a count.

The fingerprint — a hash of the canonically serialised array, introduced in chapter 1 as the way to prove no process is holding state — tells you whether two arrays are the same. It does not tell you what is in them, and for that you need the payload itself, which is where judgement about volume and privacy comes in. Logging every outbound array at 200,000 turns a month, averaging 12 KB serialised, is 2.4 GB a month of message content sitting in your log pipeline with whatever retention and access controls that pipeline happens to have, which for conversations containing addresses and order details is usually the wrong answer. Three practical postures work better: sample at 1% into a store with the same access controls as the messages table; capture in full for conversations explicitly flagged for tracing, so support can turn it on for a complaining customer; and always capture on the paths where something already went wrong, such as a model error or a moderation refusal.

The volume figure that forces those postures is worth taking in steps. Twelve kilobytes per turn multiplied by two hundred thousand turns is two million four hundred thousand kilobytes, which is the same quantity written as 2.4 GB; the multiplication is the legitimate one because the average is a size per turn and the count is a number of turns, so the per-turn unit cancels and what is left is a size. What the quantity is made of matters more than how large it is — message content, complete and in the clear, held wherever logs are held rather than wherever messages are held. Each posture then keeps the cases that answer a question and drops the bulk that answers none: a 1% sample characterises what a healthy array looks like, a per-conversation trace flag answers a specific complaint, and capturing on already-failed paths catches the conversations nobody files a ticket about.

None of that can be avoided by logging fingerprints more aggressively instead, and the reason is what a hash is. It maps an input of any size onto a value of fixed width, which forces many inputs onto each output and discards everything about the contents on the way. Two log lines carrying the same fingerprint carry the same array; two carrying different fingerprints carry different arrays and give no indication of where the difference lies. That is enough to answer whether two pods assembled the same thing, and it can never answer whether the order number was in either.

The classic bugs, and what each one looks like from the outside#

Start with the query, because that is where most of them live.

-- the bug
SELECT role, content
FROM messages
WHERE user_id = $1
ORDER BY created_at DESC
LIMIT 50;
-- rows are then reversed in application code

There is no conversation filter. The rows come back scoped to the user, so every chat that user has ever had is in scope, and the ordering by timestamp interleaves them. Someone who was arranging a return in one tab and asking about delivery windows in another gets an array where the two conversations alternate turn by turn, sorted by when each was sent.

Assembly for conversation c-7f3a returns all five rows, in that order. The model is shown a user who complains about a cracked order, is asked about the box, then abruptly asks about Tuesday delivery slots, receives an answer, then says the corner is crushed. What the user sees is an assistant that confidently refers to something they said in a completely different chat — mentioning delivery slots in the middle of a returns conversation — and, because the whole array is only 50 rows across all their conversations, an assistant whose memory of this conversation is also mysteriously short. Both symptoms come from the missing AND conversation_id = $2, and both disappear when it is added.

This bug survives testing because of what a test conversation looks like. A user with exactly one conversation gets the same rows from a user-scoped query as from a conversation-scoped one, since the two filters select the same set when the set has one member. The missing clause is therefore undetectable in any environment where an account is created for a single flow and thrown away afterwards. Exposing it needs a user with two conversations live at once, which is the ordinary condition of a real customer and an unusual condition of a fixture.

The LIMIT 50 deserves separate attention even after the filter is fixed. A limit taken from the recent end with a reversal in application code is correct in shape, and it is also a truncation policy that nobody wrote down: at turn 51, the conversation silently starts losing its beginning, which is where the order number and the original complaint live. That is the behaviour chapter 6 argues should be a measured decision rather than a constant someone typed while making a page load faster.

The mechanism of that truncation deserves stating, because the shape of the query conceals it. Ordering by timestamp descending and taking a fixed number of rows selects from the recent end of the conversation. Reversing the result in application code then puts those rows back into reading order, which restores the order of the rows that were fetched and can do nothing about rows that were never fetched. The array comes back well-formed, correctly sequenced, and beginning in the middle of a conversation, so nothing visible from inside the handler looks damaged. That is why the signal has to come from the sequence range rather than from any property the array declares about itself.

Duplicates are the second family. A client timing out at 30 seconds and retrying a turn whose model call took 34 seconds produces two identical user rows and, once both calls complete, two assistant rows. The array now contains the customer's question twice in a row, and models handle that badly in a specific way: repetition reads as emphasis or as a correction, so the assistant sometimes answers twice, sometimes apologises for having missed the first one, and sometimes treats the second copy as a restatement implying the first answer was wrong. The fix is an idempotency key on the write path rather than a de-duplication pass on the read path — a client-generated identifier sent with the turn and stored under a uniqueness constraint, so a retry of a request that already landed is recognised as the same write instead of a new one — because deduplicating at assembly hides a double-charged model call that is still happening.

The two-copy outcome follows from a deadline mismatch rather than from either side behaving incorrectly. The client's timeout expires while the model call is still running, so the client abandons a request the server is still executing. Nothing informs the server that the client has gone, so it finishes the call and writes both rows exactly as it would have. The retry then arrives as a new request to the same handler with no knowledge of the first, computes its own position, writes its own rows and makes its own model call. Two independent executions of a correct handler produce two of everything, which is why the defect is not in the handler.

That the two executions are indistinguishable is a second argument for the write path, alongside the double-charged call. A read-path deduplicator would have to decide whether two identical adjacent messages are a retry or a customer who genuinely sent the same sentence twice, and the stored text is the same in both cases, so the decision cannot be made from the rows at all. An idempotency key moves the decision to where the difference actually exists: a retry carries the key its original carried, a fresh message carries a new one, and a uniqueness constraint on that key rejects the second write without anybody having to infer intent from content.

Stale rows are the third. Products that let users edit a message usually store a new revision rather than mutating in place, and assembly that selects on conversation_id alone will happily include both the original and the edit, in sequence, so the model reads a customer who gave one order number and then gave a different one. Soft deletion has the same shape, which is why the query above carries AND deleted_at IS NULL — a clause that is invisible until the first time a user deletes a message and the assistant keeps referring to it.

What those two share is worth naming, because it predicts the rest of the family. Every clause in assembly's WHERE encodes a policy about which stored rows count as part of this conversation, and each policy stays dormant until a user exercises the feature that creates rows it was written to exclude. Until somebody edits a message there are no superseded revisions to filter out, and until somebody deletes one there is nothing for deleted_at IS NULL to do. A query missing either clause therefore passes every test that was written before the feature shipped, and fails on the day the feature ships to a user.

A fourth, quieter one is content truncation at the column or serialisation boundary. A content column typed varchar(2000), or a logging-and-then-sending path that reuses a truncating formatter, will silently clip long turns, and long turns are exactly the ones carrying pasted order details. The symptom is that the assistant remembers the beginning of what the customer said and not the end, which reads uncannily like a memory problem and is not one.

Whether a length bound clips or refuses is a thing to establish rather than assume, and it depends on where in the path the bound sits. A database column may reject an over-long write outright, in which case the failure is loud and arrives as an error rather than as a short message. A formatter, a serialiser or a field configured to truncate removes the tail and returns success, and only that second behaviour produces this symptom. Finding out which one is in front of you takes a deliberate over-long turn and a read-back of what was stored, and that is the demonstration worth asking for in review, because it decides whether this bug is possible in your system at all.

When it really is the model#

The array being correct does not end the investigation, and pretending otherwise trains a team to dismiss real failures. There is a well-documented effect where models attend less reliably to material in the middle of a long context than to material at either end, so a fact given at turn 12 of a 60-turn conversation can be present in the array and still be missed. Retrieval accuracy on a fact placed mid-context degrades noticeably as the surrounding context grows, and the degradation is gradual rather than a cliff, which is why it produces exactly the kind of intermittent report that looks like a bug in your code.

The belief that gets in the way here is that presence in the array is the same thing as use of it. Everything in the array is available to the model, and availability is where the guarantee stops. A continuation is produced by weighting all of the text in front of the model, that weighting was learned from data rather than specified by anyone, and nothing in the arrangement promises that a sentence will carry weight in proportion to how much a person reading the conversation would say it matters. Presence is necessary and it is not sufficient, and that is the whole of what this layer owes an engineer — the machinery underneath it changes with every model and the asymmetry does not.

The way to tell the two apart is the same artefact, read one step further: find the turn in the logged payload, note its position and the total token count, and check whether the reports cluster at long conversations. If the fact is at position 12 of 61 in a 38,000-token array, you are looking at an attention problem, and the remedies are different — restate extracted facts near the end of the array, pin them into a dedicated message, or shorten the window — and they belong to chapters 6 and 7. If the fact is absent, no remedy in that family will help.

The order in which the two hypotheses get tested is not a matter of taste. The array check is cheap, it runs against a log line that already exists, and it answers in both directions. The attention hypothesis is neither cheap nor directly checkable: it cannot be examined at all until the payload is in hand, and confirming it takes a cluster of reports across long conversations rather than one conversation read carefully. The sentence above is also a prediction rather than only a caution, and it is worth spending: every remedy in that family works by relocating a fact that is already present, so a team that ships one and sees the reports hold steady has learned which hypothesis was right without running the array check at all.

Instrumentation also has an honest limit. It tells you what you sent and it says nothing about whether what you sent was what should have been sent, so a conversation where assembly faithfully reproduced rows that were themselves written incorrectly logs perfectly clean. Two requests arriving on the same conversation within the same second can produce exactly that: rows that are wrong at the moment they are written, an id list that looks orderly, and a conversation missing a turn that both requests believe they stored.

The reason the log cannot reach that class is structural rather than a gap in the fields it carries. Every id in the list belongs to a row that exists, so a turn that was never written contributes nothing to the list and leaves no hole where it should have been. The sequence range shares the blindness: two requests computing the same next position leave one surviving row at that position between them, and the surviving rows still form an unbroken run. An unbroken run is exactly what a healthy conversation produces, so the artefact that settles every bug above has nothing to say about this one, and no additional field logged at assembly time would change that.

The instinct a team should build is to answer "the assistant forgot something" with a log line rather than with a hypothesis. Message ids, sequence range and fingerprint cost a few hundred bytes per turn and convert the most common support complaint in this product category into a mechanical comparison: ids that grew by two, a range that starts at 1, a fingerprint that is stable across pods. Every deviation in that pattern maps to a named bug in the table above, and the handful of reports that survive the comparison are the ones worth spending model budget on. The one class the log cannot reach is the conversation whose rows were wrong before assembly ever read them, which happens when two requests land on one conversation inside the same second, and the next chapter reproduces that on purpose.