Attributing a failure to a stage

A trace is a per-request record of what each stage of a pipeline received and emitted — the question as it arrived, the candidates retrieval returned, the context that reached the model, the answer, and the parameters each call ran under. It is the difference between a failing question that can be assigned to a stage and one that can only be argued about, and it is written at request time or not at all.

This chapter is about turning a single failure into a stage assignment: the four artifacts a trace has to contain and what they cost to store, the decision procedure for one failing question, the isolation re-run that is the only experimental step in it, how to replay a trace without accidentally comparing two builds, and how the whole thing runs nightly over a set of failures instead of one ticket at a time. The last section is about the boundary of the technique, which is that a stage assignment narrows the search and does not hand you a cause.

What a trace has to record#

Here is a trace for a question that produced a wrong answer — the assistant said SAML group sync runs every 24 hours, and it runs every 4.

{
  "trace_id": "req-7f31c2",
  "question": "How often does SAML group sync run?",
  "model": "gen-model-v3", "temperature": 0.2, "prompt_hash": "9ac41b",
  "spans": [
    {"stage": "rewrite",  "in": "How often does SAML group sync run?",
                          "out": "SAML group synchronization frequency schedule", "ms": 210},
    {"stage": "retrieve", "k": 50, "ms": 38,
     "out_ids": ["c-4417","c-8102","c-2290","c-6651","c-9930","...45 more"]},
    {"stage": "rerank",   "in_ids": "…same 50…", "ms": 96,
     "out_ids": ["c-8102","c-2290","c-6651","c-4417","c-1177"]},
    {"stage": "generate", "context_ids": ["c-8102","c-2290","c-6651","c-4417","c-1177"],
     "context_hash": "d21f0e", "in_tokens": 3914, "out_tokens": 128, "ms": 2240,
     "out": "SAML group sync runs every 24 hours."},
    {"stage": "cite",     "cited_ids": ["c-2290"], "ms": 3}
  ]
}

The gold chunk for this question is c-3380, from docs/sso/saml.md, and it appears nowhere in the trace — not in the reranked five, not in the fifty candidates. The retriever never found it, so the failure belongs to retrieval or to ingest, and any change to the generator's instructions is addressing a stage that was handed no timing values.

Calling that a lookup rather than an inference is a claim about where the work happened, and the distinction is the point of the whole chapter. Nothing in the previous paragraph reasoned about what the model was likely to do, or about which stage tends to be at fault, or about how confident the answer sounded. It read a field. c-3380 is either in out_ids or it is not, and it is not, and that settles which stages could possibly have contributed to this failure. The reason teams argue about attribution instead is not that the reasoning is hard. It is that the field is missing, and once the field is missing the only remaining instruments are plausibility and seniority.

Four artifacts have to have been written down for that lookup to work. The question as received, before any rewriting, because the rewritten form is a stage output and confusing the two makes the rewriter invisible. The full candidate list from retrieval, all fifty IDs, not the five that survived — this is the field most often dropped and it is the one that separates "never found" from "found and ranked out". The context that reached the model, as IDs plus a hash of the concatenated text, so that you can prove which version of a chunk was used after the documents have been re-indexed twice. And the generation parameters: model version, temperature, and a hash of the rendered prompt template, without which a re-run is a different experiment.

The storage cost is modest and worth stating because it is the objection that usually gets raised. Fifty chunk IDs, five context IDs, hashes and timings come to roughly 1.4 KB per request as JSON; chunk text is not stored in the trace at all, since IDs plus the content hash let you resolve text from the store as long as you keep old chunk versions. At 40,000 requests a day that is about 56 MB a day, or 1.7 GB a month, which is nothing against the inference bill for the same traffic.

The two multiplications there are worth doing rather than skimming, because the objection they answer is usually raised as an order-of-magnitude worry rather than a calculation. A kilobyte and a bit, forty thousand times, is fifty-six thousand kilobytes, which is fifty-six megabytes. Thirty of those days is a little under one and three quarter gigabytes. The same day's traffic sends 40,000 prompts of a few thousand tokens each through a generation model, and one month of that is a bill people approve without discussion. Tracing is three orders of magnitude cheaper than the thing it is describing, and the reason it gets dropped is never the storage.

Teams that end up unable to attribute anything usually did not decide against tracing; they logged the question and the answer because that is what an HTTP handler logs by default, and the intermediates were never in the record.

The intermediates are also not recoverable later, and it is worth being precise about why, since "we can always re-run it" is the reflex that makes the omission feel survivable. Retrieval is a function of the question and of the index as it stood at that instant. Documents get re-ingested, chunk boundaries move, the embedding model gets upgraded, the reranker gets a new checkpoint. Re-running the question next week asks a different index the same question and gets whatever that index returns, which may be right or wrong and is in either case not what happened. A trace is not a convenience that saves you a re-run. It is the only copy of a state that no longer exists.

The decision procedure for one failing question#

The last branch is the one that earns its place, because it is the only step in the procedure that runs an experiment rather than reading a record. When the gold chunk did reach the context and the answer is still wrong, there are two very different situations and no log field distinguishes them. Either the model is capable of answering from that passage and something about the surrounding four chunks, or their order, or the prompt's framing prevented it — or the model cannot produce the right answer from that passage at all.

def isolate(question, gold_ids, generate):
    full = generate(question, context_ids=retrieve_and_rank(question))
    solo = generate(question, context_ids=gold_ids)          # gold chunk only
    return {"full": full.text, "solo": solo.text}

Two calls, about four seconds, roughly $0.002.

The logic of that pair of calls is worth making explicit, because it is the one place in this chapter where a conclusion is being drawn from a comparison rather than read off a record. The two runs share the question, the model, the prompt template and the sampling settings. They differ in the context and in nothing else. If the solo run comes back right, then the model was always capable of producing that answer from that passage, which means the capability was never the constraint and the wording was never the constraint either — both were identical across the two runs. What differed was the presence of four other chunks, so the four other chunks are what stopped it. If the solo run comes back wrong, the model cannot get from that passage to that answer under these conditions, and the candidates that remain are the prompt, the model and the label.

Two honest limits sit on that. The test says the other four chunks are responsible and does not say which of them, or whether it was their content or their position; splitting that further means more runs, dropping one chunk at a time. And it holds the passage constant while changing the amount of material around it, so a solo run is also a much shorter prompt, which is a second difference the design cannot remove.

On the SAML question the isolation test is not applicable, since the gold chunk was never retrieved. On a question where it is — "which endpoint replaces /v1/exports", where the gold chunk sat at rank 1 — the solo run returned the correct endpoint and the full run did not, which localises the fault to the other four chunks in the context. Reading them showed two migration-guide passages for a different endpoint family, both semantically close, both containing the phrase "remains supported". The wording of the system prompt was not implicated in any way, and a prompt change would have been a coin flip against a distraction problem whose real fixes are a tighter k, a precision-oriented reranker, or a deduplication pass over near-identical migration documents.

If the solo run is also wrong, the prompt genuinely is a candidate — along with two others worth checking before rewriting it. The chunk may not say what the label claims it says, which happens more often than teams expect and is caught by reading the chunk text rather than trusting the label. And the model may be at capability limit on that reasoning step, which is checked by running the same solo prompt against a stronger model; if that fixes it, the wording was never the constraint.

Replaying a trace faithfully#

The isolation test is a comparison between two runs, so it is only informative if the runs differ in the one thing you intended. Three sources of drift make that harder than it looks. Temperature, the sampling setting that governs how much randomness the model is allowed when choosing each token, is the obvious one: at 0.2 the same prompt produces different answers across runs often enough that a single solo run coming out correct is weak evidence, so replays should either run at temperature 0 or run three times and report how many of the three were correct. The prompt template is the second, because a replay executed a week after the trace was captured renders the current template, and the trace's prompt_hash is what tells you whether that matters — a mismatch means you are comparing two changes at once. The third is the indexed text itself, since chunk c-8102 may have been re-ingested with different content, which is why the trace stores a context_hash and the store keeps prior chunk versions.

Building the replay as a function of a trace rather than of a question is what makes all three checkable. The signature replay(trace_id, override={"context_ids": gold_ids}) reconstructs the exact call, applies one substitution, and refuses to run when the template or chunk hashes no longer match. Without that discipline the isolation test still produces a result, and the result is a comparison between two builds rather than between two contexts.

Turning the procedure into a nightly report#

Done by hand the decision tree is a few minutes per question, which is fine for an escalation and impractical for the 37 failures in a nightly build. Everything except the isolation branch is a lookup against fields already in the trace, so most of it automates directly.

def attribute(trace, gold: set[str]) -> str:
    candidates = set(trace["retrieve"]["out_ids"])
    context    = set(trace["generate"]["context_ids"])
    if not candidates & gold:
        return "retrieval_miss"
    if not context & gold:
        return "rerank_or_k"
    return "generation_candidate"        # needs the isolation re-run to split further

Two of the three branches are free and run over every failing question in the build. The third costs two generation calls each and is worth running only on the questions that reach it, which on this build was ten of thirty-seven, or about two cents and forty seconds — ten questions at four seconds and two-tenths of a cent apiece. Reporting the counts nightly turns attribution from something a person does when a ticket arrives into a distribution the team sees every morning.

The shares in that last column are the counts divided by the thirty-seven failures, and reading them in that direction is what makes the table actionable. Twenty-one of thirty-seven is a little under three failures in five, which is the retrieval row. Three of thirty-seven is a little under one in twelve, which is the prompt row. The distribution is not a ranking of stages by importance and it is not stable across products; it is this build's answer to the question of where the work is, computed from records that were going to be written anyway.

Three of thirty-seven failures are in the cell that a prompt change can address, which is a shape that recurs across systems of this kind. The one thing not to automate is the reading. The categories tell you which stage owns a failure; they do not tell you why, and a week of nightly counts with nobody opening the underlying traces produces a stable chart and no fixes.

A boundary is not a root cause#

The first row of the distribution above is not yet a diagnosis. "Not among the candidates" is a boundary assignment, and the causes underneath it — the chunk was never ingested, it was split across a boundary so neither half states the fact, the embedding model handles the vocabulary badly, the lexical half of the hybrid was disabled — are separate investigations. Assigning a failure to a boundary narrows the search from the whole pipeline to one stage's inputs and configuration. It does not hand you the root cause, and treating a boundary assignment as a root cause is how teams end up rebuilding a retriever when the actual defect was an ingest job silently skipping files above 2 MB.

What the assignment does buy is a reduction in the search space, and it is worth pricing that honestly rather than either overselling it or dismissing it. Before the lookup, a failing question could have come from any of five stages and any of their configurations. After it, four of the five are excluded and the remaining work is a list of a handful of candidate causes inside one stage, each of which is cheap to check because it is a question about that stage's inputs. That is most of the value of attribution and it is not all of the value anyone expects from it, which is where the disappointment comes from. It is also why the distribution above is the input to chapter 7, on sequencing component evals, rather than the output of this one.

There is also a sample-size discipline that the ticket-driven version of this work tends to lose. One escalated question tells you one thing about one question, and the pull to redesign a stage on the strength of a single vivid trace is strong. The counts above are worth acting on; a single row of them is worth investigating and not worth committing a quarter to.

The aggregate metrics cannot do this job either, and the reason is directional. The metrics of chapter 3, on retrieval recall and answer groundedness, give rates across a set. Rates tell you the size of a problem and cannot tell you anything about a specific request, because the mapping from a request to its contribution to the average is one-way once the intermediates are gone. Traces work in the other direction: they answer questions about instances and say nothing about frequency until you aggregate them, at which point you have reconstructed the component metric from the trace store, which is in fact how most teams end up computing recall in production — over logged candidate lists rather than by re-running retrieval.

That relationship is the reason tracing comes before eval infrastructure rather than after it. A component metric computed on a replayed pipeline tells you what the current build does on your eval set; one computed over traces tells you what the deployed system did on real traffic, on questions nobody wrote labels for, and it is the only one of the two that can be sliced by customer, by intent, or by the hour of an incident.

Attribution, then, is four fields in a log line and a decision tree over them, plus one two-call experiment for the cases the log cannot settle. It converts "the assistant hallucinates" into a count per stage, and a ticket into a stage assignment before anybody edits anything. With a distribution like the one above on the table, the next decision is what to do about it — and there is a wrong move that survives even a clear attribution, which is reaching for the prompt anyway, because the prompt is the part that can be changed this afternoon.