A wrong rewrite is often a truncated history, not a bad rewriter

A conversational retrieval system answers a follow-up through four stages, and each one hands an artifact to the next. Chat state is the message list a request assembles out of storage and passes on. Contextualization — the rewriter — reads that list plus the turn just typed and returns a standalone query, meaning one that still identifies what is being asked about when it is read with no transcript attached. Retrieval takes the standalone query and returns passages. Generation takes the passages and writes the answer. A failed follow-up therefore leaves four artifacts behind, in a fixed order, and the diagnostic question is which of them is the first to be wrong.

Two of those four verdicts are easy to reach and two are not. A correct rewrite with wrong passages indicts ranking, and the trace says so plainly. A chat-state fault instead presents as a wrong rewrite, which is an artifact produced by the stage that is innocent, and a generation fault presents as a trace in which nothing upstream is wrong at all, which is a verdict reached by elimination and only as trustworthy as the eliminating. Between them those two account for the failures that get diagnosed as the wrong team's problem.

Why a wrong rewrite is weak evidence about the rewriter#

The rewriter is a function of two arguments. It is handed a list of earlier messages and the turn the user has just typed, and it returns a query string. There are exactly two ways for the return value of a function to be wrong: the function is wrong, or one of the arguments it was given was wrong. Nothing about the return value, read on its own, distinguishes those two cases. A rewrite that names the wrong entity is precisely what a good rewriter produces when it is handed a history in which the right entity does not appear.

Stated as a property of the artifact rather than of the function, that is why two days can go into the wrong file. The wrong rewrite is identical in both worlds — the same string, in the same log line, with the same visible defect in it, whether the rewriter is poor or the rewriter was starved. The quantity that separates the two worlds sits upstream and unrecorded, and the quantity everybody reads does not move between them at all. A signal that reads the same in one failing world as in the other carries no information about which you are in, however loudly it announces that something is wrong, and that is the whole case for instrumenting the boundary rather than the model.

An asymmetry in what gets logged is what keeps this from being obvious. Almost every system logs the rewrite, because it is one short string and it is obviously interesting; almost none logs the message list that produced it, because the list is bulky and looks like a duplicate of a store that already exists. So the only visible half of the pair absorbs the whole of the suspicion, and the argument that follows — the rewrite is wrong, the rewriter produced the rewrite, therefore fix the rewriter — is valid in form and unsound in exactly one place, which is the unstated assumption that the arguments were right.

The mechanism is worth spelling out one link at a time, because the conclusion is unintuitive while every link in it is dull. A product renders the conversation from storage, so the user sees every turn ever written. The request answering the next turn selects a subset of those to send, and the model conditions only on what it is sent, so anything outside the subset does not exist as far as the model is concerned. A referring expression such as the second one is resolvable only against an antecedent — the earlier mention it points back at — so an antecedent outside the subset leaves the model with nothing to resolve against. Its only available outputs are then to leave the reference unresolved or to guess at it, and both produce a wrong query from behaviour that is entirely correct.

This is the same shape as debugging a failure whose stack trace ends at an RPC boundary — a call into another process, where the trace stops at the point the call was made. The trace is real, every frame in it is accurate, and the fault is on the other side of a call the trace cannot see into. The remedy is the same as well: capture what crossed the boundary, and compare it with what the caller believed it sent.

The analogy breaks in two places worth naming. A stack frame is emitted by the runtime and cannot be wrong about itself, while a rewrite is emitted by a model that genuinely can be wrong on a perfectly good history; taken too far, the boundary picture exonerates the rewriter automatically, and one of the four verdicts below is that the rewriter really was at fault. The second break is about what is capturable. An RPC boundary carries a serialised payload with a schema, recordable whole; what crosses this boundary is a list the caller assembled by running a selection over a store, so "what the caller believed it sent" has to be reconstructed from fields somebody chose in advance to log.

The three ways chat state lies#

Chat state faults share a signature, which is that the rewrite is wrong in a way that is reasonable given the input the rewriter actually received. Distinguishing them from a genuinely bad rewriter takes one comparison — what went into the contextualizer against what the user can see on their screen — and there are three recurring causes.

The window is shorter than the conversation. Products render the full transcript because scrollback is cheap, and send a fixed number of recent turns to the model because context is not. Those two numbers are set in different files by different people, and the user's expectation is set by the one they can see. Chapter 5's argument for shrinking windows makes this more likely rather than less, which is why the scoped approaches there mark relevance rather than truncate.

Assistant turns are missing. This is the most damaging version because it leaves a history that looks populated. An assistant is what enumerates things — regions, plan tiers, clusters — since enumerating is what an answer does while asking is what a question does, so ordinals and definite references point at assistant output almost by construction. A history containing only user turns is therefore missing precisely the content the rewriter needs, while appearing to contain six turns of context. The causes are mundane: a client that persists user input synchronously and assistant output after the stream completes, a stream that was cancelled when the user navigated away, a tool-calling turn stored in a different table.

The state belongs to a different conversation. Shared session keys across browser tabs, a resumed session whose key was reused, or an out-of-order write from two concurrent requests all produce a history that is well-formed and not the user's. The tell is an entity in the rewriter's input that appears nowhere in the user's transcript, and it is unmistakable once you look for it and invisible if you only read the rewrite.

None of the three is a mistake anybody made, and separating the requirement from the decisions shows why. The requirement is that a follow-up be answered as though the whole conversation were available to the retriever. The decisions are local and each was correct where it was taken: render everything because scrollback is cheap, send a bounded number of recent turns because context is not, persist the user's message on arrival and the assistant's when the stream finishes, keep tool-calling turns in their own table, key a session on something the browser already holds. Each opens a gap between what the requirement assumes and what the rewriter receives, invisible from inside the component that opened it.

The severity of the missing-assistant case is worth quantifying, because it is easy to assume a half-populated history is half as good. Of the follow-up items whose antecedent lived in an assistant turn — 64% of the production sample, since ordinals and definite references almost always point at something the assistant enumerated — dropping assistant turns from the rewriter's input took recall from 0.86 to 0.29. The remaining successes were items where the user had also named the entity themselves at some point, which is the minority. A history stripped of assistant turns is not a degraded history; for this task it is close to no history at all, while the n_messages field reports six.

The size of that collapse is not a matter of proportion, and the argument for why settles every history-stripping case rather than the one measured. Removing assistant turns does not degrade the antecedent for those items; it deletes it, and a list that was never passed in has no partial version. So recall on that group cannot settle at some fraction of what it was. It falls to whatever share of the items carry a second, independent mention — the cases where the user named the entity themselves as well — which is a property of how people phrase their own questions rather than of the retrieval system. The surviving recall is a floor rather than a scaled original, and a better rewriter cannot raise a floor set by a duplication rate, because the input it would need is missing in exactly the items that fail. That is what licenses reading 0.29 as a statement about coverage: the system has not become roughly a third as good at the task, it has stopped attempting most of the task and is scoring well on the remainder.

Instrument the boundary, not the model#

def log_contextualizer_input(conv_id: str, msgs: list[Message]) -> None:
    log.info("ctx_input", extra={
        "conv_id": conv_id,
        "n_messages": len(msgs),
        "roles": Counter(m.role for m in msgs),          # {"user": 6, "assistant": 2}
        "first_ts": msgs[0].ts, "last_ts": msgs[-1].ts,
        "monotonic": all(a.ts <= b.ts for a, b in zip(msgs, msgs[1:])),
        "digest": hashlib.sha256(
            "".join(f"{m.role}:{m.content}" for m in msgs).encode()).hexdigest()[:12],
        "client_turn_count": current_request.headers.get("X-Rendered-Turns"),
    })

Five fields answer all three questions before anyone reads a transcript. A roles histogram of {"user": 6, "assistant": 0} is the missing-turns fault. An n_messages of 4 against a client_turn_count of 11 is the window fault. A monotonic of false, or a digest that differs between two requests in the same conversation with no user turn in between, is the session fault. The client_turn_count header costs the frontend one line and is the only one of these that cannot be reconstructed server-side, because the server has no other way to know what the user is looking at.

That last point is what makes the pair worth more than either field. Two instruments here share no step: n_messages is the length of a list the server assembled from its own store, and client_turn_count is a count the browser made of elements it drew on a screen. Nothing derives one from the other, and no single bug can move both in step. When the two agree, truncation is ruled out without anyone opening the conversation, and when they disagree the difference is the number of turns the rewriter was never shown.

The header is also a design decision worth naming as one. The requirement is to detect that the rewriter saw less than the user did; the server knows what it stored and what it sent and has no access at all to what was drawn on a screen, so no amount of server-side logging meets that requirement alone. A request header is the cheapest available answer to the gap rather than the only conceivable one, and treating it as a decision is what keeps it from being filed as unowned frontend work.

The digest earns its place by what it cancels. It is computed over role and content only, so message ids, timestamps and request ids all vary between two requests and none of them reaches the hash. Equality is therefore a statement about content and nothing else. Inequality across two requests in the same conversation with no user turn between them has one legitimate cause — the assistant's own reply from the previous request has since been persisted, which lengthens the history by one known turn — and the residual after allowing for that is a write from somewhere that is not this conversation.

[ctx_input] conv=9f21c4 n_messages=4 roles={user:2,assistant:2}
            client_turn_count=11 monotonic=true digest=7c1a4be09fd2
[rewrite]   query="What is the retention period for the second cluster?"
            resolved=[]  dropped=[]

The empty resolved array in that record is the second signal, and it is the reason chapter 2's structured output earns its keep here. The rewriter is reporting that it resolved nothing, which means it did not find an antecedent, which means either the reference was unresolvable or the antecedent was not present. Combined with n_messages=4 against client_turn_count=11, the verdict is settled without opening the conversation.

That combination is worth more than either signal alone because the two are of different kinds. The message count is a property of what was passed in, fixed before the model ran; the resolved array is the model's own report of what it found, produced afterwards. A fault in the selection step cannot fabricate an empty resolved, and a confused model cannot fabricate a message count of four, so their agreement is what makes this a verdict rather than a plausible reading.

Two beliefs stand between a team and that record, and both are tempting. The first is that a populated history is an adequate history, and n_messages is the field that encourages it. It counts list elements, so it reads six whether those six are three complete exchanges or six consecutive user messages with every assistant reply missing, while the property that decides the outcome varies wildly between those cases. A reading identical in the failing world and the healthy one is not a weak check; it is not a check. The role histogram is the repair precisely because it is the cheapest field that does move.

The second tempting belief is that reopening the conversation reproduces the fault. It cannot, because the message list is not stored anywhere; it is derived from the store by a selection step at request time. Re-deriving it later runs that same selection over a store that has been written to since — the assistant turn that was missing has finished streaming and been saved, and the window may have been widened by the fix somebody deployed in between. A replay keyed on the conversation id therefore tests the store as it is now, and only a replay keyed on the logged message list tests the state that actually failed.

Reproduction done that way is the step that turns a logged verdict into a fix, and it needs the state, not the conversation id. A replay harness that takes the logged message list, loads it into a fresh session, and re-runs the follow-up gives a case you can iterate on locally at the cost of one model call; the same harness is what promotes a production failure into an item in the follow-up set, since an item is exactly a prefix plus a follow-up plus a gold label. Teams that wire replay into the trace viewer — a button that copies the ctx_input payload into a scratch session — convert most of these investigations from an afternoon into ten minutes, and they accumulate eval items as a side effect of doing support.

When the answer contradicts passages that are right#

The fourth boundary produces a trace where everything upstream is clean. State holds the full conversation, the rewrite names Dublin, the top-5 passages are Dublin SLA pages, and the answer says Frankfurt. No upstream artifact explains it, which is what makes generation the verdict.

There is a version of this specific to multi-turn systems and it is worth knowing before you meet it. The generation prompt contains the conversation history, which contains the assistant's earlier answer stating that Frankfurt is covered by a 99.95% SLA. The model now has two sources: retrieved passages about Dublin, and its own prior statement about Frankfurt, rendered in the same prompt with the authority of a previous assistant turn.

The two sources do not arrive on equal terms, and that asymmetry is the whole mechanism. A retrieved passage is presented as material to consult. The earlier answer is presented in the assistant's own role, and a model trained to continue a transcript treats a prior assistant turn as something the transcript has already committed to rather than as one more document to weigh. Continuing coherently is what the training rewards, so the earlier claim exerts a pull no passage can. Models are strongly biased toward consistency with their own earlier output, and the answer therefore comes out hedged toward Frankfurt or blended between the two, citing the Dublin passages while reporting the earlier region's numbers.

Both repairs work on that asymmetry rather than on the model. On one system that measured it, 62% of contradiction cases were fixed by moving the retrieved passages after the conversation history and immediately before the question, which changes which block sits nearest the thing being answered. A further group resolved when the prompt was changed to state explicitly that retrieved passages take precedence over anything said earlier in the conversation, which replaces an implicit contest of authority with a stated rule. Neither repair is general. The mechanism they work on is the conversational one, where the competing claim is the assistant's own earlier output, so a reordered prompt does nothing for the causes in the paragraph below, and the residual the pair leaves behind is the contradictions that were never about the earlier turn.

The other generation causes are the familiar ones and are not conversational: the gold passage sitting at position 8 of 10 where attention is weakest, a passage set where four of five documents describe the wrong entity so the majority wins, or an instruction in the system prompt about being consistent that was written for tone and is being applied to facts.

The four verdicts side by side#

The rows are exhaustive for a reason that has nothing to do with having thought of every cause: they enumerate artifacts, not causes. A failed follow-up produces four artifacts in a fixed order, and any failure has a first artifact that is wrong, so walking them in order and stopping at the first break puts every failure in exactly one row. The rows are also pairwise distinct — take any two and at least one column reads differently — so the flowchart is a decision tree over the table rather than a heuristic laid beside it. Each question splits the remaining rows on one column, in artifact order, and a decision tree over a table with no two identical rows terminates in a single name. That is the formal version of the claim that the flow returns a verdict rather than an opinion, each question answerable from a logged field or a thirty-second replay. Working the questions in reverse — starting from the answer and reasoning backwards about what might have gone wrong — is what produces two days of prompt tuning on a config bug, because the rewrite is the first artifact anyone reads and it is genuinely wrong.

Reading the tell column across the four rows shows what the table cannot do. Three of the four tells are positive: a field to compare, a rule to apply against the input, a replay to run. Generation's tell is nothing upstream is wrong, which is an absence, so generation is the residual bucket and a residual bucket is only as good as the checks that drain into it. In a system that does not log what the contextualizer received, the first row's tell is unavailable, the first question cannot be asked, and every case belonging in the first row falls through to the bottom of the table instead. The table cannot support a generation verdict at all unless the boundary above it was instrumented; what it records in that situation is a chat-state bug with no evidence against it.

Two faults at once, and traces nobody recorded#

Two stages can be wrong at once, and the flow above will name only the earliest. A conversation with a truncated window and a badly ranked index produces a wrong rewrite and bad passages, and fixing the window reveals the ranking problem that was underneath it the whole time. That is the residual the row-per-artifact argument leaves behind: first wrong artifact is not only wrong artifact, and the table decides the first while saying nothing about the second. The discipline that survives it is to fix the earliest indicted boundary, re-run the same case, and re-diagnose from scratch; a diagnosis of a downstream stage made while an upstream stage is feeding it garbage is not evidence about anything.

The method also assumes the artifacts exist. Most systems log the final answer and the retrieved chunk ids, some log the rewrite, and almost none log what the contextualizer received, which is the field that decides between the two most commonly confused verdicts. Adding those fields costs an afternoon and is worth doing before the first escalation rather than during it, because the traces you need are the ones from conversations that already happened, and a field added today says nothing about the ticket already open.

Read the four artifacts in order every time, then, because the only thing a wrong rewrite tells you on its own is that some boundary above it needs checking. Chat state and generation make that concrete from opposite directions: one produces a guilty-looking artifact from an innocent stage, the other produces no incriminating artifact at all and is reached only by elimination. Both are decided at boundaries rather than by looking harder at a component.

What all of this adds up to is a small set of habits to carry into a system you are responsible for. Look at whether the string reaching your embedder is self-sufficient, and measure how much of your traffic it is not. Build a follow-up set from real transcripts, stratified by reference pattern, and gate on the strata. Log the rewrite as structured output, with what it resolved and what it dropped, and log what it received. Then, when a follow-up fails, print the four artifacts in order and let the boundary that breaks tell you which team's afternoon this is.