Replaying a query today hides the version that produced the answer

Reproducing a defect is the ordinary first step in debugging, and it works because most systems are functions of their inputs and their code, both of which you can hold still. A retrieval system is a function of a third thing as well. Its answers depend on the state of a corpus — the body of documents it has ingested and indexed — and that state changes continuously, without a deploy, without a version number, and without anything in the request being different.

The consequence is a debugging rule that runs opposite to the usual one. When a report concerns a document that has been replaced, corrected or reindexed, a replay that returns the right answer is the expected outcome and not a refutation of the report. Working out what happened requires evidence about a past state of the corpus, and that evidence has to have been written down at the time, because by the time anybody looks the thing it describes has already changed.

Five explanations, and what separates them#

Read the second column rather than the first, because the five explanations differ in what layer moved, and that is what decides both the evidence and the remedy. Two of them are changes to the corpus itself, one is a change to a document's text, one is a change in ranking with no corpus change at all, and one is not a change of any kind — it is the system working as configured on two passages that were both retrieved.

The third row is already excluded by the ticket, because a correction in place would have changed the text the citation resolves to, and it still says $75. The first, second and fourth are all consistent with everything you have, and they call for different responses: the first is an ingestion latency problem affecting every recently revised document, the second means the fix already landed and the question is only how many others are pending, and the fourth means the ranking is unstable and the same query could regress tomorrow with no corpus change at all.

That fourth one is worth taking seriously rather than treating as a curiosity, and the mechanism behind it is a chain worth walking. A vector index rarely compares a query against every stored vector; it walks a graph that usually arrives at the nearest ones, which is where the approximate in approximate nearest neighbour comes from, and HNSW is the graph structure most vector databases use for that walk, with ef_search setting how many candidates the walk keeps in flight. The graph is built from a randomised insertion order, so two builds of the same collection are two different graphs. A walk over a different graph visits different nodes. When two candidates are nearly tied, the one the walk happens to reach first wins, and nothing about the query, the corpus or the code has changed.

An HNSW index with ef_search=64 over a collection containing two near-identical revisions of the same policy will therefore not reliably return the same one on every build, because the two candidates are separated by a cosine gap of 0.004. Rebuilding the same collection from the same source three times produced the 2025 chunk at rank 1 in two builds and the 2023 chunk in the third. Raising ef_search to 256 made it stable at the cost of 40% more query latency, which is a real trade and not the fix — the fix is that two revisions of one document should not both be candidates, which is chapter 3's filter. Widening the search lowers the probability that the walk misses a near-tied neighbour; it does not make the tie go away, and a tie between a current and a superseded revision is a corpus defect wearing a ranking costume. The point here is that ranking instability is a live explanation for a non-reproducible report and it is indistinguishable from the others without evidence.

The fifth is chapter 2's failure wearing this chapter's clothes. If both revisions were in the top-k on 4 March, the retrieval did its job and the generator picked, which means the ticket is about a missing conflict policy rather than about a stale index, and tombstoning the 2023 revision fixes this instance while leaving the mechanism intact.

What chunk IDs alone can and cannot tell you#

The logs on the inherited system recorded the query text, the answer, the citations, and the retrieved chunk IDs. That is more than many systems keep and it settles less than it appears to.

It establishes that chunk c_8841a2 was retrieved for that query, that the answer's claim traces to it, and — if the chunk text is unchanged since — that it stated $75 at the time. It does not establish that the chunk text is unchanged, because ingestion pipelines that update documents in place keep the chunk ID stable while replacing the text, so a chunk ID is a pointer to a mutable cell rather than a record of a value. It does not establish whether the 2025 revision existed in the index on 4 March, since the retrieval log lists what was returned and says nothing about what was present and not returned. It does not establish what the ranking was, only the membership of the top-k. And it does not establish which index build served the request, so a corpus change and a ranking change look identical from the log.

Line those four limits up against the five explanations and an exact correspondence appears. The citation's text excludes the third. The retrieval log's membership settles the fifth, since it lists which revisions came back. What is left is the first, the second and the fourth — and those are precisely the three that the four limits above make unresolvable. The evidence you have removes exactly the explanations you did not need removed, and leaves untouched exactly the set you did.

The investigation therefore ends in a judgement call, and judgement calls made under time pressure on a ticket that reproduces cleanly end in "not reproducible".

That is where the stumble sits, and it is a belief about a defence rather than a slip. Closing a ticket on a clean replay feels rigorous: the standard test was run and it passed. What makes it wrong is that the replay's outcome is a quantity that stays fixed while the dangerous property varies completely. It returns the right answer whether the underlying fault has been fixed for one document, fixed for a class, or is still live on forty other policies, because all three of those worlds produce the same current corpus for that one query. A test whose result cannot distinguish the states you care about is not weak evidence; it is no evidence.

The record that makes the next report a lookup#

Provenance has to be captured at answer time, because everything it describes is mutable and the whole difficulty is that it has already mutated by the time anyone looks. The chain is short and each link is checkable. A document's text, its version state and the set of documents in an index are all fields that later writes overwrite in place. Nothing in a vector store keeps the prior value of an overwritten field, because a vector store is not a version-control system and does not pretend to be. A query issued later therefore reads only the current value, whatever it was before. So any statement about a past state has to come from a copy taken while that state was current, which means the copy has to be written on the request path rather than reconstructed afterwards.

That is close to what a lockfile does for a build, and the comparison is worth making explicitly as an analogy rather than left to be inferred. A lockfile records the exact versions a build resolved, so that a later build can be made to resolve the same way. The place the analogy breaks is the part that matters here: a lockfile pins its inputs, and nothing in a retrieval system can pin a corpus, because the corpus is the product a hundred people are editing on purpose. A provenance record can only witness what the state was, never restore it, so it answers "what happened" and never "make it happen again".

The note at the bottom is the whole design in one line. The index has moved from one build to another between the answer and the investigation, and no query you can run today reaches the earlier one. Either something wrote down what that build contained, or that state is gone.

The record itself is small and flat.

{
  "answer_id": "ans_2026_03_04_7f31",
  "answered_at": "2026-03-04T09:14:22Z",
  "index_build_id": "B-1042",
  "retriever_config_hash": "sha256:9d2b…",
  "generator": {"model": "…-2026-01", "temperature": 0.2, "seed": 11},
  "conflict_policy_fired": null,
  "retrieved": [
    {"rank": 1, "chunk_id": "c_8841a2", "doc_id": "conf:FIN:221",
     "doc_family_id": "fam:travel-expense", "doc_version": 4,
     "version_state_at_answer": "unknown", "effective_from": "2023-02-01",
     "content_hash": "sha256:1f0c…", "score": 0.8412, "cited": true},
    {"rank": 2, "chunk_id": "c_b19e04", "doc_id": "conf:FIN:604",
     "doc_family_id": "fam:travel-expense", "doc_version": 6,
     "version_state_at_answer": "unknown", "effective_from": "2025-01-15",
     "content_hash": "sha256:77ad…", "score": 0.8376, "cited": false}
  ]
}

Two fields do most of the work. content_hash lets you assert that the text you are reading today is the text the model saw, without storing the text itself and without trusting chunk ID stability; when the hash differs, you know the document was corrected in place and the citation you are looking at is not the passage that produced the answer.

A hash earns its place here by being asymmetric in a useful direction. A content hash is a fixed-length fingerprint computed from a passage, such that any change to the passage produces a different fingerprint. Two equal fingerprints are therefore strong evidence that the text is unchanged, and two different fingerprints are proof that it changed — the second direction is the certain one, and it is the direction the investigation needs, because the question being asked is whether the citation in front of you can be trusted as the passage the model saw. The fingerprint is also short, so storing it costs almost nothing next to storing the passage, and it reveals nothing about the passage's contents to anyone reading the record.

index_build_id binds the record to a corpus state, which is only useful if build manifests are retained — a manifest listing doc_id, doc_version, version_state and content_hash for 60,000 documents is about 6 MB, so keeping the last 400 builds costs 2.4 GB and lets you reconstruct exactly which documents were candidates on any past date. That is the artefact that separates "the newer revision was not in the index" from "it was in the index and lost".

Those two statements are exactly the first and fourth explanations, and it is worth being precise about why a manifest settles them when the retrieval log cannot. The retrieval log is a list of what came back, so it can only report membership of the result. The manifest is a list of what was present, so it reports membership of the candidate pool. Absence from a result is consistent with both "not there" and "there and outranked"; absence from the manifest is consistent with only the first. Retaining manifests is therefore not a second copy of the same evidence but a different measurement, and it is the one the investigation was missing.

The storage cost of the records themselves is not a reason to hesitate, and the sizing chains from one figure to the next. Twelve retrieved chunks at roughly 200 bytes each plus the envelope is about 2.6 KB per answer; at 48,000 answers a month that is 125 MB a month, and thirteen months of retention is 1.6 GB. Each step multiplies the previous one by a count that is stated, and the sizing assumes a record holding a dozen candidates rather than the eight a single search returns in the diagram above, which is the right assumption for a system that also records what a reranker or a conflict policy dropped. Against the whole figure, the investigation above took two engineers a day and a half and ended without a conclusion.

Where provenance stops: a corpus that is uniformly out of date#

Provenance does not prevent anything. A logged stale answer is still a stale answer, and a team that ships provenance instead of version state has built an excellent record of a defect that continues to occur. Its value is entirely in converting an investigation into a lookup, which matters because the investigations are what stop teams from fixing the underlying corpus — three unresolved tickets teach an on-call engineer that these reports are not actionable.

It also cannot witness the case where the world changed and the corpus did not. If the meal allowance rose to $95 in a finance leadership decision and no document was ever updated, the provenance record shows a clean chain from a current, correctly stated, correctly cited document to a wrong answer, with nothing anywhere marked as suspect.

That limit is shared by every instrument in this topic, and it is worth stating as a property rather than as a caveat. Conflict detection compares two documents. Supersession compares two revisions. A version filter compares a document against its family. Provenance compares a stored state against a later one. All four are measurements of disagreement, and a corpus that is uniformly and confidently out of date presents no disagreement to any of them. That gap closes only with review dates and owners, which is a documentation practice rather than a retrieval one.

Putting the whole topic into an order for a system you have just inherited: turn on provenance logging first, since it is additive, cheap and changes no behaviour, and it makes every subsequent claim measurable. Add the cited-document version monitor from the previous chapter to get a baseline rate. Triage the document classes where staleness is dangerous, obtain supersession from owners where they exist and infer it where they do not, and tombstone with the filter rather than a score penalty. Configure a conflict policy per class before the detection starts firing, so the first conflict lands on a rule rather than on the generator. Scope any recency term to the classes with real revision lines, and ship it behind a shadow diff.

Then keep one habit as the standing default: a corpus bug that will not reproduce is a version question until the provenance says otherwise, and closing it on the strength of a clean replay closes the only kind of bug that fixes itself and comes back.