Reproducing last week's answer needs what the alias resolved to

The escalation arrived on the 23rd, from an insurance broker with a regulated obligation to document what they tell their clients. On the 14th at 09:41 CET, one of their agents asked the billing assistant whether a mid-term adjustment could be backdated to the start of the policy period, and the assistant said yes, citing a clause. The clause exists and says the opposite. The agent acted on it, and the broker wants to know what happened, whether it is fixed, and how many of their people saw the same thing.

One piece of vocabulary decides how the rest of this reads. Providers publish a model under two kinds of name: a snapshot, which is a frozen set of weights with a dated identifier such as mid-2026-02-18 and never changes, and an alias, which is a moving name such as assistant-mid that points at whichever snapshot the provider currently considers the one you should be on. Code that names the alias is asking for whatever the alias resolves to on the day it runs, and the provider moves that pointer on its own schedule.

Nine days sit between the answer and the question. In that window the model alias assistant-mid moved on the 17th when the provider published a new snapshot, and the service does not pin snapshots because pinning was thought to delay improvements. The billing assistant's prompt is one of the ones that never got migrated to versioned artifacts, so it was edited in place twice, once on the 16th to shorten the tone preamble and once on the 20th to add a line about backdating that somebody wrote after this very complaint reached the team. And the flag assistant_v2, which controlled which cohort got the newer configuration, was switched off on the 15th and its key reused on the 19th for an unrelated retrieval experiment, because the key was free and the name was still descriptive enough.

Each of those three decisions had a defensible case behind it. Tracking the alias means picking up provider improvements without a change request, editing prompts in place is what the authoring tool has always done for a service nobody had time to migrate, and reusing a flag key kept the key namespace tidy. None of them looked like a decision about auditability at the time, and together they mean the answer to a regulator's question is currently a reconstruction from timestamps.

What the logs support and what they do not#

The request log for that day is complete in the sense that every request is in it. What it contains determines which sentences in the reply to the broker are statements of fact.

The flag row is the sharpest of these because it looks like evidence. A query joining requests to flag assignments returns assistant_v2: on for the 14th, and the same key now names a retrieval experiment with different variants, a different salt and a different targeting rule. Anyone reading the assignment table today, or any dashboard built on it, will interpret the 14th through the 19th's semantics. Flag keys should be archived rather than recycled for exactly this reason; the cost of a new key is nothing, and the cost of a recycled one is a permanent ambiguity in every historical query.

The envelope every request carries#

The fix is a fixed set of fields recorded on each request, alongside the trace, describing what the request bound to. Roughly 700 bytes:

{
  "request_id": "req_01HZ8K3M",
  "ts": "2026-03-14T09:41:12.338Z",
  "service_version": "billing-assistant@2026.03.09-1f3a2c",
  "prompt": {
    "name": "billing_assistant",
    "version": 48,
    "rendered_sha256": "9d5b1c…e7a0"
  },
  "model": {
    "requested": "assistant-mid",
    "resolved": "mid-2026-02-18",
    "provider_request_id": "req_mp_7Yh2",
    "params": { "temperature": 0.4, "top_p": 1.0, "max_tokens": 900 }
  },
  "retrieval": {
    "index_version": "policies-2026-03-11",
    "retriever_version": "hybrid@7",
    "docs": [
      { "id": "POL-4471#c12", "sha256": "3a70…91bb", "score": 0.82 },
      { "id": "POL-4471#c19", "sha256": "c412…07de", "score": 0.74 }
    ]
  },
  "tools": { "schema_sha256": "b1c9…4402" },
  "flags": {
    "billing_prompt.2026q1": { "variant": "v48", "rule": "rule:tier_pro",
                               "unit": "account_id", "salt": "bp.2026q1" }
  },
  "eval": { "suite": "billing@2026.03.05", "result": "pass" }
}

model.requested against model.resolved is the field that answers this escalation. The alias is what the code asked for; the snapshot is what served the request, and the provider returns it in the response body, so recording the resolution costs one line at the call site and is the only way the mapping survives the alias moving. Recording the alias alone records an intention that was already stale by the 17th.

rendered_sha256 is the backstop for everything else. It hashes the fully assembled message array — the template after interpolation, the retrieved passages, the tool definitions, the conversation history — so two requests with the same hash had identical input regardless of what any registry claims, and a request whose hash does not match what version 48 renders today is proof that something in the pipeline changed even if every version number matches. It is also what makes hash-only retention useful, which the cost section returns to.

The retrieval block is the part teams most often omit, and it is frequently the actual cause. The clause the assistant cited was in the context because the retriever put it there; whether the failure lies in the prompt, the model, the index build on the 11th or the retriever's ranking is undecidable without knowing which passages were supplied. Document ids alone are not enough, since documents are edited, so the content hash of each passage as supplied is what pins it.

Counting the affected users#

With the envelope in place, the population is a query rather than an estimate:

SELECT count(*) AS requests, count(DISTINCT user_id) AS users
FROM request_release_metadata
WHERE org_id = 'brk_2210'
  AND ts >= '2026-03-09' AND ts < '2026-03-16'
  AND prompt_version = 48
  AND model_resolved = 'mid-2026-02-18';
-- requests: 3,120   users: 41

That is the exposure, and it is not the answer to the question the broker asked. Sharing a configuration with a bad answer does not mean receiving a bad answer, so the population has to be narrowed to requests that could plausibly have hit the same failure and then measured. Filtering to requests whose retrieval context included a policy-terms document leaves 1,410. Human grading of a random 200 of those found 6 answers making an unsupported backdating claim, which is 3.0%, with a Wilson interval (a confidence interval that stays sensible when the counts are as small as six) of 1.4% to 6.4% — so between about 20 and 90 of the 1,410, from an estimated 8 to 20 of the 41 users.

Reporting that interval rather than the point estimate is the honest form, and a regulated customer will generally accept an interval with a stated method. What they cannot accept, and should not, is a number with no population definition behind it, which is what the pre-envelope version of this investigation produces: a filter on nothing more specific than "requests from your org in that week", a sample drawn from a population that mixes several configurations, and a percentage that describes none of them.

Grading 200 conversations by hand is two days of somebody's week, which is why teams reach for a judge model to classify the whole 1,410 instead. That works and needs its own accounting: a judge with 92% recall, meaning it catches 92% of the answers that really are bad, and 4% false positives, meaning it wrongly flags 4% of the good ones, applied to 1,410 requests at a true rate of 3% flags roughly 39 true positives and 55 false ones, so the raw flag count overstates the harm by more than double, and the estimate has to be corrected using the judge's measured error rates on a labelled subset. The usual arrangement is a judge over the full population to find candidates, plus human grading of a sample to calibrate it, and the judge's version identifier recorded alongside the result so that re-running the analysis in a month compares like with like.

Reproducing the configuration is not reproducing the answer#

Replaying with the recorded prompt version, the resolved snapshot, the recorded sampling parameters and the exact retrieved passages gives you the same inputs. It does not guarantee the same output. Temperature was 0.4, no seed was requested, and even at temperature 0 a provider's serving stack can return different tokens across time as batching, kernels and hardware change underneath a snapshot name. Snapshot pinning bounds the weights, not the numerics. A few providers expose a seed or a system fingerprint that lets you detect when the serving configuration changed; none of it makes a reproduction guaranteed, and this is genuinely unsolved rather than merely inconvenient.

The practical form is to replay repeatedly and report a frequency. Twenty replays at the recorded parameters, with the recorded context, produced the unsupported backdating claim seven times: 35%, with the sampling and the model held fixed. That is a stronger and more useful statement than a single reproduction, because it distinguishes a failure that is latent in the configuration from one that required an unlucky sample, and it gives a baseline against which the fix can be measured — the same twenty replays against prompt version 51 producing it zero times is evidence of a fix in a way that "we changed the prompt and it looks better" is not.

This also disposes of the tempting shortcut of re-running the question against today's configuration. If it reproduces, you have learned that today's stack has a problem, which is worth knowing and says nothing about the 14th. If it does not reproduce, you have learned that something changed between the 14th and now, which is the one thing you already knew and cannot characterise. Telling a regulated customer that their issue is resolved on that basis is a claim you cannot support, and it is the kind of claim that turns an incident into a finding.

What the discipline costs#

The storage is cheap. Seven hundred bytes per request at 100,000 requests a day is 70 MB a day and about 25 GB a year, which is not a budget conversation. The expensive part is the payloads: rendered prompts and responses contain customer content, and keeping thirteen months of them for reconstruction sits directly against the retention limits chapter 5 argued for. Splitting the two resolves most of the tension — retain the metadata envelope with its hashes for as long as your obligations run, and the payloads for thirty days. After thirty days you can still prove that a request used version 48 on snapshot mid-2026-02-18 with a specific rendered input, because the hash identifies it, even though the text is gone.

The rest of the cost is discipline that has to hold across teams: prompts are published rather than edited, aliases are recorded as resolved, flag keys are archived rather than reused, retrieval writes its index and document hashes into the same envelope, and the evaluation, canary and shadow stores all key on the same identifiers so that a comparison from chapter 5 and a canary arm from chapter 4 can be joined to a production request. One practical trap: none of these fields belong as labels on time-series metrics, where prompt version and resolved snapshot will multiply cardinality, the count of distinct label combinations for which the system keeps a separate series, until the metrics backend falls over. They live on traces and logs, and the aggregation happens in the query.

Everything in this topic exists to make one sequence possible. A change is published as an immutable artifact, selected by a flag that governs one decision, exposed to a cohort that resembles your traffic, watched by guardrails that abort without waiting for a person, reversible in twenty seconds through a path that shares nothing with your build pipeline, and recorded well enough that any request can be traced to what produced it. The way to find out whether you have that is to take the next prompt change on your list — the small one, the copy fix — and put it through the whole path, including the rollback, on purpose, at a time you have chosen.