The prompt capture record

In a system with any sophistication at all, the string that reaches the model is built from parts with independent lifecycles:

• a system block, which is a template plus interpolated values, and which may be one of several variants if an experiment is running;
• retrieved context, which is whatever a search index returned for this query at this moment, in the order the reranker put it — a reranker being the second, more careful scoring pass that reorders the index's first-stage hits — from an index that is continuously being written to;
• tool results, which are the responses external systems gave when called — an order status, a stock level, a currency rate, all of which were true then and are not now;
• conversation history, which may have been truncated, summarised or compacted to fit the context window, the fixed number of tokens a single request is allowed to occupy, meaning the history the model saw is not the history in your database;
• runtime facts interpolated as text: the current date, the user's plan tier, which feature flags were on.

Every one of those is a variable that can change between the day a bad answer was produced and the day somebody tries to reproduce it, and every one is invisible if what you logged was {"question": "...", "answer": "..."}.

The chain that makes this fail is short and each link is checkable. The prompt is not something the user handed you; it is something your code built, from several sources, at one moment. Each of those sources has its own lifecycle and changes on its own schedule, with no coordination between them. Running the same build later therefore reads different sources and produces a different artefact — not occasionally, but by default, because nothing was holding any of them still. And a different artefact is a different request, so its answer is evidence about a request nobody complained about. Replaying the question does not re-run the request; it builds a new request from today's index, today's tool responses, today's flags and today's template, and then compares its answer to an eleven-day-old one. When they differ you learn nothing, and when they agree you conclude the bug is not real. Every investigation in the previous six chapters started from the assumption that the same request could be sent twice, and this chapter is about earning that assumption.

What a capture record contains#

The fix is to write down the assembled artefact, not its ingredients, and to write down enough of the ingredients to explain how the artefact was built.

{
  "capture_version": 3,
  "request_id": "req_01HQ8FZ4T2W6",
  "ts": "2026-02-17T14:22:08.441Z",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",

  "prompt": {
    "sha256": "9f2c4b1e7a…",
    "messages_ref": "blob://prompts/9f2c4b1e7a",
    "token_count": 3184,
    "components": {
      "system_template": "support-agent@v14",
      "system_vars_sha256": "1a77c0…",
      "history_turns": 6,
      "history_truncated_at_turn": 2,
      "retrieved": [
        {"doc_id": "kb-1187", "chunk": 3, "score": 0.812, "index_version": "kb-2026-02-16b"},
        {"doc_id": "sales-0442", "chunk": 0, "score": 0.774, "index_version": "kb-2026-02-16b"}
      ],
      "tool_results": [
        {"tool": "get_account", "args_sha256": "cc90a2…", "result_sha256": "77b301…",
         "latency_ms": 84, "upstream_version": "accounts-api/2026-02-11"}
      ]
    }
  },

  "model": {
    "requested": "gpt-4o",
    "served": "gpt-4o-2024-08-06",
    "system_fingerprint": "fp_a7d06e42cc",
    "params": {"temperature": 0, "top_p": 1, "seed": 42, "max_tokens": 600}
  },

  "serving": {
    "service_image": "sha256:7d1e…",
    "region": "eu-west-1",
    "prompt_template_commit": "a91f3c2"
  },

  "response": {
    "sha256": "3c81ff…",
    "body_ref": "blob://responses/3c81ff",
    "finish_reason": "stop",
    "usage": {"prompt_tokens": 3184, "completion_tokens": 118}
  }
}

Three decisions in that shape are worth defending.

The full message array is stored by reference and content-addressed. System blocks and retrieved chunks repeat across enormous numbers of requests, so a content-addressed blob store deduplicates almost all of it: at two million calls a day with 3,000-token prompts, storing every prompt inline is on the order of 24 GB a day, and deduplicating the shared prefixes typically cuts that by better than an order of magnitude. The record itself stays around 1 kB and lives in whatever you already query.

Both halves of that are arithmetic worth doing. Two million calls each carrying three thousand tokens is six billion tokens a day, and at roughly four bytes of text per token that is the 24 GB. The saving comes from what content-addressing means: a blob is keyed by the hash of its own bytes, so identical bytes written a million times are stored once and referenced a million times, and the reference is a few dozen bytes. A system block that is the same on every call therefore costs its own size once for the day rather than once per request, and the same is true of any retrieved chunk that keeps coming back. Getting better than an order of magnitude out of it does require addressing the repeating components individually rather than only hashing each assembled array as one object, since two prompts sharing a long preamble and differing in one ticket are two distinct objects at whole-array granularity.

Retrieval is captured as document IDs, chunk indices, scores and index version rather than only the concatenated text. The text is in the prompt blob; the IDs are what let you ask "was sales-0442 in this index on the 17th and is it now", which is a question no amount of re-running the query will answer. The distinction is between what the model read and where it came from. Text tells you the first and can tell you nothing about the second, because a passage carries no record of which document it was cut from or which build of the index served it. Re-running the query does not recover that either: the query answers from today's index, so it reports what would be retrieved now and is silent about what was retrieved then. The identifier is the only handle that survives the passage of time, and the scores beside it are what let you say whether a document was a marginal hit that a small change would have removed or a confident one that something upstream must explain.

Tool results are captured as hashes plus upstream version, with the bodies stored alongside if they are small. You are not going to re-call an order-status API and get February's answer, so the recorded result is the only copy of that input that will ever exist.

The hash needs one piece of discipline to be worth anything: it has to be computed over a canonical serialisation, by a single function used at both capture time and comparison time. A hash is an identity for the bytes it was given, and it is an identity for the logical value only if there is exactly one byte string per logical value. The moment the serialiser is free to choose — which order to write the keys, how much whitespace to use — that condition fails, and the digest becomes a fact about the serialiser's mood rather than about the prompt. Hashing json.dumps(messages) with default settings makes the digest depend on Python's dictionary insertion order and on the whitespace the serialiser chose, so two byte-identical requests can hash differently and two different requests can be made to agree by an unrelated refactor. Serialise with sorted keys and fixed separators, or better, hash the exact request body you are about to put on the wire, immediately before sending it. That last option is the most honest one available, because it hashes the thing that actually left your process rather than a reconstruction of it.

Capturing all of this for every call is usually more than you need and occasionally more than you are allowed. The split that works is to write the small metadata record on 100% of calls — it is a kilobyte, it contains hashes rather than content, and it is what the queries in chapter 6 run against — and to write the prompt and response blobs on a subset. Which subset is a policy decision worth making explicitly: a low background sample for trend work, plus 100% of anything the system already thinks is interesting. Requests that returned an error or a refusal, requests where a tool call failed, requests the user gave negative feedback on, and requests from any session that later opened a support ticket should all capture in full. That last one requires being able to backfill from a session ID, which in practice means a short-lived buffer that promotes to durable storage when a ticket arrives.

The split falls where it does because the two halves of the record have completely different cost profiles. The metadata record is a kilobyte of hashes and identifiers, it contains no customer content, and it is what every trend query in the previous chapter reads — so writing it on every call is cheap in storage and free of the obligations that content brings with it. The blobs are the assembled prompt and the full response, which are large and are made entirely of the content that carries those obligations. Sampling them is therefore not a compromise forced by budget; it is putting the expensive half of the record only where it earns its place. And the reason the policy has to include everything the system already thinks is interesting is that a background sample is chosen before anybody knows what will be asked about, so on its own it will miss the report you actually receive — which, by definition, came from an unusual request.

Streaming responses need one extra decision. The record should contain the assembled final message, not the chunk stream, because the assembled message is what your application acted on and what you will replay. Keep the stop reason and the token counts from the terminal event, and if the stream was cut off — a client disconnect, a timeout — record that explicitly, because a truncated response and a short response look identical once they are a string in a database and only one of them is a bug. The information that separates them was never in the text. It arrived alongside it, in the field saying why generation stopped, and that field exists precisely because the text cannot carry it — a model that finishes early and a stream that was cut both leave a shorter string and nothing in the characters distinguishes them. Discarding the terminal event is discarding the only copy of the distinction.

Classifying a discrepancy#

Using it is the other half of the exercise, because a record nobody reads is storage cost with no return. With a record in hand, "we cannot reproduce it" becomes a decision procedure rather than a mood.

The critical branch is the one that separates replaying the recorded prompt from rebuilding the prompt from today's inputs. They are different experiments and both are necessary. Replaying the recorded messages holds every input fixed and asks whether the model still behaves that way; rebuilding asks whether the inputs themselves have moved.

Between them the two cover the space, which is what makes the flowchart a procedure rather than a checklist. There are two places a difference can live — in what was sent, or in what the far end did with it — and each experiment freezes one of them so the other can be observed. Replay freezes the input and varies only the passage of time, so a difference is attributable to the model or the serving path. Rebuild freezes nothing on the far end and asks a question about your own pipeline alone, so a difference is attributable to an input. Neither alone is diagnostic, and running both in either order narrows the answer to one side of the boundary.

Running only the second is what pasting a customer's question into a console does, and it is why that check so reliably comes back clean. It is worse than incomplete, in fact: a console is a different entry point, so it may not even run the same assembly code, and a clean result from it is consistent with the bug being present in every path the product actually uses.

def replay(record: dict, client) -> dict:
    """Re-send the exact assembled prompt. Nothing is regenerated."""
    messages = blobs.get(record["prompt"]["messages_ref"])
    assert sha256_of(messages) == record["prompt"]["sha256"], "blob drift"

    body = {"model": record["model"]["served"], "messages": messages,
            **record["model"]["params"]}
    response = client.chat.completions.create(**body)

    return {
        "same_output": sha256_of_text(response.choices[0].message.content)
                       == record["response"]["sha256"],
        "served_then": record["model"]["served"],
        "served_now": response.model,
        "fingerprint_then": record["model"]["system_fingerprint"],
        "fingerprint_now": response.system_fingerprint,
    }

def rebuild(record: dict) -> dict:
    """Re-assemble the prompt from today's index, tools and template."""
    fresh = assemble_prompt(user_message=extract_user_turn(record))
    return {
        "same_prompt": sha256_of(fresh) == record["prompt"]["sha256"],
        "diff": component_diff(record["prompt"]["components"], describe(fresh)),
    }

Note that replay pins record["model"]["served"] rather than the alias. Sending the alias would let the provider serve whatever it points at today, which quietly merges the first branch of the flowchart into the second and destroys the experiment.

The final leaf deserves its own comment. "Serving variance" is only a conclusion if you have a number for it, which is what chapter 4's table was for. If your measured divergence rate for prompts of this kind is 2% and you have one unreproducible report, that is consistent. If your measured rate is effectively zero and you have thirty reports, something is wrong that this classification has not caught, and the honest move is to say so rather than to file everything unexplained under floating point.

The limits of capture#

Capturing everything does not guarantee reproduction. Chapter 3 was explicit that a hosted endpoint gives you no control over batch, kernel or device, so a byte-identical replay to a byte-identical build can still land on a near-tie and take the other branch. Capture converts unreproducible into reproducible or attributable, which is a smaller and more achievable promise, and it is the one that closes tickets.

Capture also has a cost that is not only storage. Assembled prompts contain everything the user said, everything retrieval pulled in, and everything your tools returned, which in most products means personal data, and sometimes special-category data. A capture store is therefore subject to the same retention limits, access controls, deletion-on-request handling and regional constraints as your primary datastore, and it is a genuinely bad idea to build one as a side effect of a debugging session. The practical shape is a short retention window on full prompt blobs — fourteen to thirty days covers essentially every report that gets raised — with the small metadata record kept longer, because that is what the trend queries of the previous chapter need and it contains hashes rather than content.

There is one class of finding capture produces that is worth expecting, because it surprises people: quite often the prompt turns out to be non-deterministic on your own side. Two requests that should have assembled identically produce different hashes, and the diff shows the same three retrieved chunks in a different order, or two tool results swapped because they were awaited concurrently and appended in completion order, or a set iterated without sorting. Each of those has a plain cause. Two chunks with equal relevance scores are ordered by whatever the sort does with ties, and an unstable sort does not promise the same answer twice. Two tool calls issued concurrently and appended as they return are ordered by which upstream service happened to be faster this time, which is a network measurement rather than a decision. A set has no order to iterate in, so the one you get is an implementation detail nobody promised to keep. None of that is the model's doing and all of it produces exactly the symptom this topic has been chasing — and every one of them is upstream of the model, so it shows up as a divergence in the first handful of tokens, which is the signature chapter 4 taught you to read.

A capture store makes it visible in a single query — group by the component fields, count distinct prompt hashes — and the fix is ordinary software engineering rather than anything to do with inference.

And this is not a substitute for tracing. The trace_id in the record exists so that the capture joins to the spans you already collect; latency, retries, upstream errors and queue time live there and should stay there. What the capture adds is the one thing a trace never holds: the exact bytes of a request whose inputs no longer exist.

Point all of this at the next unreproducible report, and the order is fixed. Pull the capture record. Compare the served model and fingerprint against today's; if they moved, you have your answer and the rest is re-baselining. If they held, replay the recorded prompt byte-for-byte and see whether the bad output comes back. If it does, this was never a non-determinism problem and you have a deterministic bug with a reproduction case. If it does not, rebuild the prompt from today's inputs and diff the components — index version, retrieved document IDs, template version, tool results, where history was truncated — and the difference that shows up there is what changed. Only when the model is identical, the prompt is identical and the output still differs are you looking at the mechanism this topic opened with.

By then you have a divergence rate to compare it against and a decision to make about whether reproducibility is worth 1.6× on the evaluation fleet. The twenty-run experiment from chapter 1 is the piece to run first, before any of this is needed, because it takes an afternoon and it converts every later argument from an opinion into a number.