A revision id is what makes a quoted rule resolvable

A retrieval system answers a question by selecting chunks — the pieces a document was cut into for indexing — and handing their text to a model to write an answer from. When that answer quotes a rule, somebody eventually asks three questions about it: which document the rule came from, which version of that document, and whether the person who asked was entitled to read it. None of the three is a question about the answer's wording, and none of them can be settled by reading the answer more carefully.

What has to be stored beside a chunk's vector for those questions to have answers is the subject here, and one field carries most of the weight. A point, in a vector store, is a vector together with a payload — a set of stored fields that travel with it. A revision identifier is a payload field naming the exact state a document was in at the moment that vector was produced. It is the difference between a compliance question that resolves in two lookups and one that consumes two days and resolves in nothing.

Identity has to be stored, because it cannot be reconstructed#

A trace that logs text gives an investigator two things: the generated answer, and the text of the chunks that were retrieved. One chunk out of such a trace reads:

Recordings are retained for 24 months from the date of the call, after which
they are deleted unless subject to a legal hold or an active dispute.

That is the whole of what was recoverable. The failure is not that the trace was too small. It is that chunk text is not an identifier, and it is worth separating out what an identifier has to do, because chunk text fails at each part of the job for a different reason. An identifier has to name one thing rather than several. It has to name that thing as it stood at a particular moment, since the question being asked is about the past. And it has to still name something when somebody goes to look it up. Those are three requirements, not three phrasings of one.

Text fails the first because text is copied between documents, so it does not identify one. A full-text search of a document repository for the sentence above can return every document the passage was ever pasted into — a support operations handbook, a data retention standard, an onboarding deck — and the search has no basis for ranking one of them as the source, since the wording is identical in all three by construction. Copying makes the map from text to document one-to-many, and no amount of search skill inverts a one-to-many map.

Text fails the second because text changes between revisions, so a match against the current text tells you nothing about the revision that produced the vector. Matching a stored sentence against a document's current text compares the past against the present. A hit says the passage still exists today, which is not what was asked. A miss says the passage does not exist today, which is also not what was asked. The question was which state of the document produced this vector, and a revision that has since been edited leaves nothing of itself behind in the text that replaced it.

Text fails the third because text disappears when a revision deletes it, so the strongest evidence — the sentence that was removed — is exactly the case where reconstruction is impossible. The deleting revision takes away the only handle the trace was holding, and the resulting lookup has no domain: there is nothing current to match against, in any document. That is the worst case to lose, because the passages worth investigating are disproportionately the ones somebody thought worth rescinding. A sentence that has been deliberately withdrawn is precisely the sentence a reviewer will want to trace, and it is precisely the sentence text-matching cannot find.

Three requirements, three failures, one apiece, and nothing left over. That accounting is what licenses the conclusion rather than merely suggesting it: no residual case survives in which more careful text logging would have worked, because the three properties exhaust what an identifier is for and text breaks each of them on its own. The remedy is a different kind of field, not a more diligent handling of the same one.

What settles the question is a small set of fields stored on the point, none of them derived from the chunk's own text:

{
  "doc_id": "conf:112458",
  "doc_title": "Data Retention Standard",
  "revision_id": "conf:112458@v9",
  "revision_effective_at": "2024-06-01T00:00:00Z",
  "section_path": ["Retention periods", "Voice and call recordings"],
  "source_type": "confluence",
  "source_url": "https://wiki.internal/pages/112458?pageVersion=9",
  "acl_ref": {"system": "confluence", "space": "OPS", "restriction_id": "r-4471"},
  "chunker_version": "recursive-800-100-v3",
  "ingested_at": "2024-06-01T04:12:07Z",
  "content_sha256": "b41c7e…"
}

doc_id is the identifier the source system already uses — a Confluence page id, an S3 key, a contract record id in the contract lifecycle management system. It is not a hash of the title, because titles change and a title-derived id changes with them, silently forking one document into two. revision_id pins which state of that document produced this vector, which is the field the incident below actually needed: it distinguishes the standard at v9 from the standard at v10, and it survives the sentence being deleted, because the vector carries the revision it came from rather than pointing at whatever the document says today.

Two properties are what let a revision identifier do that, and a field can have one without the other. It has to change whenever the text changes, or two different states of the document share a name and the identifier inherits the one-to-many problem it was introduced to solve. And it has to resolve back to a version somebody can retrieve, or it names a state nobody can read. Each half fails in its own way when it is missing. An identifier that resolves but does not advance — a last-edited-by field, or a date rounded to the day — will happily report a stale vector as current, and it does so silently, because the comparison against the document's latest state comes back equal. An identifier that advances but does not resolve still proves staleness correctly and cannot say what the superseded state said, which is a real limit rather than a defect, and one this chapter comes back to.

The payload above satisfies both, and it does so by construction rather than by luck. revision_id is doc_id with a version suffix, so the document id can be recovered from the revision id without a lookup. source_url carries pageVersion=9, the same version the revision id names. source_type says confluence, which is the prefix on doc_id and also the system inside acl_ref. Four fields, written independently, agree about which system, which page and which version — and a payload whose four agreements still hold is a payload that can be checked without leaving it. That redundancy is what a connector bug shows up in first: a record where the URL and the revision id disagree about the version is malformed in a way that needs no external service to detect.

acl_ref is a pointer, not a copy. It records that this chunk inherits the restriction the source system maintains under r-4471; it does not record who was in that restriction at ingestion time, because the answer to that changes hourly and a copy is stale the moment it is written. Storing a pointer means the permission question is always answered against live state, and it means an index that is months old is still asking the right question — provided the restriction it points at still exists. A restriction that has been deleted, or an id the source system has recycled, leaves the pointer dangling, and a dangling pointer has two available answers. Choosing to deny rather than allow in that case is a decision, not something the pointer forces, and it is the kind of decision worth writing down next to the field rather than discovering during a review.

chunker_version and content_sha256 are the pair that makes the record self-describing. The hash identifies this exact chunk content, and the chunker version says which splitting configuration produced it — which matters because chunk boundaries move when the splitter is retuned, so a chunk-level identifier is only as stable as the configuration that generated it. Recording the configuration turns "the chunk ids changed" from a mystery into a diff.

That hash is also where an investigation goes wrong, and the mistake is a tempting one. content_sha256 looks like the identity field: it is long, it is unique-looking, and it does change when the text changes, which is the first of the two properties a revision identifier needs. What it lacks is the second, and it lacks it in the direction that matters. The hash is a function of the chunk text and of nothing else, so it moves when the splitter is retuned even though no document changed, and it stands still when the surrounding document changes around an untouched passage. It tracks the wrong variable in both directions: constant across the thing it would have to discriminate, and variable in response to a thing nobody asked about. chunker_version sitting beside it is the chapter's own admission of this. A hash answers "is this byte-for-byte the content I indexed"; it was never going to answer "which document, at which version".

Two lookups instead of a search#

With those fields, an investigation becomes mechanical. A retrieval trace stores the point ids that were returned, and each point resolves to a revision and a permission check:

def explain_retrieval(trace_id: str) -> list[dict]:
    trace = traces.get(trace_id)
    points = store.retrieve(ids=trace.retrieved_point_ids, with_payload=True)
    out = []
    for p in points:
        m = p.payload
        revision = docs.get_revision(m["doc_id"], m["revision_id"])
        out.append({
            "doc": f'{m["doc_title"]} ({m["doc_id"]})',
            "revision": m["revision_id"],
            "current_revision": docs.head_revision(m["doc_id"]),
            "is_stale": m["revision_id"] != docs.head_revision(m["doc_id"]),
            "section": " > ".join(m["section_path"]),
            "url": m["source_url"],
            "user_may_read": acl.check(trace.user_id, m["acl_ref"]),
            "revision_effective_at": m["revision_effective_at"],
        })
    return out

Each row that comes back carries the revision the vector was made from; the document's head revision — the state it is in today, as against the state that produced this vector; a flag saying whether those two differ; the section path; the source URL; and the answer to the permission question for the user who actually asked. Note which comparison decides staleness. It is revision_id against head, not revision_effective_at against a date somebody remembers, and that ordering is deliberate: a date has to be interpreted against a calendar and a timezone and a memory of when a policy changed, while two revision identifiers either match or do not. The effective date is stored so a human can read the row; the revision id is stored so a machine can decide it.

The two branches at the bottom of that diagram are the two halves of the question, and the payload has exactly one field for each: revision_id answers which version said this, acl_ref answers whether she was allowed to see it. Two questions, two fields, no residual — and the pair is instructive because the two fields are built on opposite principles. revision_id is a snapshot, frozen at ingestion, and it has to be frozen, since the question it answers is about a moment that has already passed and must not move. acl_ref is a pointer, resolved on every check, and it has to be a pointer, since the question it answers is about now. That gives a rule for any field you are tempted to add: ask whether the question it will be used to answer is about then or about now. A field answering a question about then must be copied; a field answering a question about now must be referenced. Getting that backwards is how an index ends up with a membership list from last spring.

Where resolvability gets lost#

Three things have to hold for this to work, and each is a place teams lose the property without noticing.

The retrieval trace has to record point ids, not just text. Logging the retrieved chunk text is the natural thing to do because it is what you want to read when debugging, and it reproduces exactly the failure this chapter opened with. Log the ids, the filter that was applied, the k, and the scores; the text is recoverable from the ids and the ids are not recoverable from the text.

Points have to be upserted with an id that is a function of identity rather than of insertion order. A UUID generated fresh on every ingest run means yesterday's trace refers to points that no longer exist, and the resolution fails at the first step. Deriving the point id from doc_id, revision_id and the chunk index gives you upsert semantics — a re-ingest of the same revision overwrites rather than duplicating — and gives old traces something to resolve against. Keeping superseded revisions' points for a retention window, rather than deleting them the moment a new revision lands, is what lets a trace from three weeks ago resolve at all.

Deletions have to leave a mark. When a revision removes a paragraph, the corresponding points either disappear or stop matching, and both of those are states in which a later question about them has no answer. Writing a tombstone instead — the point's payload retained with deleted_in_revision set and the vector dropped, kept for the same window as superseded revisions — costs a few hundred bytes per removed chunk and turns "this used to be retrievable and now is not" into a fact with a revision attached to it. It is also what lets you distinguish a chunk that was deliberately removed from a chunk that vanished because a connector failed, which are the same observation from the outside and require opposite responses.

Not every field needs an index, and treating them uniformly is how payload storage gets expensive. Vector stores build a separate index structure per field you declare filterable, and that structure lives in memory in most configurations; tenant_id and acl_group and effective_date earn theirs because filters reference them on every query, while source_url, ingested_at, chunker_version and content_sha256 are read only when a human is investigating and can sit unindexed as plain payload. On 2.4 million points the difference between indexing four fields and indexing eleven is measured in gigabytes of resident memory, and the seven extra indexes serve no query. Declare the filter fields, store the rest.

Those counts close exactly, and the closure is the useful part. Four indexed plus seven unindexed accounts for every stored field with nothing spare, because the partition is binary: a field is either referenced by a filter on the query path or it is not, and no third category exists. That is what makes the audit tractable. Run down your own payload, mark each field with the query that filters on it, and any field left unmarked is paying for an index that nothing reads.

And the fields have to be written at ingestion, from the connector, where the source system's own identifiers are in hand. A pipeline that receives text and a filename cannot invent a revision id, so the place to fix a missing field is always upstream of the splitter. Retrofitting revision_id onto an existing index means re-ingesting, which is why this is worth getting right before a corpus reaches millions of chunks rather than after.

Provenance ends where attribution begins#

A resolved trace does not tell you what the previous revision said. revision_id identifies the state that produced the vector; recovering the difference between v9 and v11 requires the document service to keep both, and plenty of source systems keep only head. Where the answer to "what changed" matters — the question left unresolved in chapter 4, where a filter over a single date field turns out not to express a comparison between two states of a document — the revision history has to exist somewhere outside the index, and the stored revision_id is the join key into it rather than the answer itself. This is the revision identifier that advances without resolving, met in practice: it proves the vector is stale and it cannot show you the text it was stale against.

It is a useful comparison for anyone reading a citation. A reference giving author and title but no edition has the same shortcoming, and the parallel holds for the same reason: the reader can see a work was cited and cannot see which words were in front of the person citing it. The analogy stops at one point, and the point is the one that bites here. A library keeps its superseded editions, so an edition number is always resolvable in principle; a document service configured to keep only head has destroyed the referent, so a revision identifier can be perfectly well-formed and still resolve to nothing. Retention of old revisions is a separate decision from recording which one was used, and only the second of the two is free.

Nor does it tell you which chunk the model actually used. The trace records four retrieved chunks and the answer quoted one of them; matching the quoted sentence back to a chunk works when the model quotes verbatim and stops working the moment it paraphrases or blends two sources. Attribution at that level needs something else — citation markers the model is asked to emit, or a post-hoc alignment step — and it is a separate problem from provenance. What the stored fields give you is the set of documents that could have produced the answer, resolved precisely, which was enough to close the incident above and is not the same as knowing which one did.

A resolved trace is a diagnosis#

Resolving a trace does not make the retrieval correct. Knowing precisely which stale revision was quoted is a diagnosis, and the fix was in the connector. Traceability shortens the loop between a wrong answer and its cause and does nothing to prevent wrong answers, which is worth saying plainly because "we added full metadata" sometimes gets recorded as a quality improvement in a way that later disappoints.

And it does not enforce anything. acl_ref sitting in a payload is a pointer that something has to dereference, and in the explain function above the thing that dereferenced it was a reviewer running a script after the fact. The check that matters happens before the search returns, in a filter derived from the asking user's live permissions, and nothing about storing the field causes that filter to exist. Chapter 4 made this point about tenant_id — that embedded text is what gets searched while payload fields are what get filtered — and it applies with more force here, because the failure is silent and the audit trail you have just built will faithfully record that an unauthorised chunk was retrieved.

A quoted rule becomes resolvable at the moment somebody stores a token saying which version it was quoted from, and stays unresolvable for as long as the only record is the words themselves. The three ways text fails — copied, changed, deleted — are not bad luck in one corpus; they are properties of text, and they are why the field has to be written at ingestion by the component that still has the source system's identifiers in hand. Everything downstream of the splitter is too late, because everything downstream of the splitter has only the text.

The tempting shortcut, once every chunk carries a classification and an audience, is to put those strings where the model can read them — Classification: Confidential | Visible to: finance-team, prepended to the text, right next to the heading path that has been earning its keep since chapter 1. It reads like defence in depth. It is worth working out exactly which component is supposed to act on that string.