A superseded guide can be the best textual match, and stay one

A document can be formally replaced and remain the best textual match for the question it no longer answers correctly. Supersession is the relation between a page and the newer page written to do its job — a relation that holds whether or not anybody deleted the old one, and that most wikis record, if at all, as a sentence a human is expected to read. Relevance ranking, the machinery that decides which passages a search returns for a question, has no access to that relation at all. Everything it can consider is contained in the query and the passage in front of it.

That gap has a practical shape. An assistant built over an inherited wiki will answer onboarding questions from whichever page reads most like an answer, and in a corpus assembled over years the page that reads most like an answer is frequently the one somebody wrote with care before it was replaced by a shorter successor. The replacement is correct and thin; the original is wrong and good. Nothing in the scoring path prefers correct over good, because nothing in the scoring path knows that one of the two documents replaced the other.

A relevance model scores a pair, and supersession joins two documents#

A cross-encoder reranker is the strongest instrument ordinarily available here, so it is worth being precise about what it takes as input. It is a model that reads the query and one candidate passage together, in a single sequence, and scores that pair directly, rather than comparing two vectors that were computed separately and never met. Reading them together is what makes it accurate: the model can notice that a phrase in the passage answers a phrase in the query, which a pair of independently computed vectors cannot represent.

Write out what that model is a function of, because the whole argument of this chapter falls out of it. Its inputs are a query and a passage. Its output is one number saying how well that passage answers that query. A second document appears nowhere in the input, so no property that is a function of two documents can influence the output. Supersession is exactly such a property: it is a statement about a pair of documents, and neither member of that pair is the query. The relation is outside the model's domain, and a term that is outside a function's domain cannot be recovered by tuning the function.

The consequence is easy to check and unpleasant to accept. Given the query "how do I get access to staging" and two candidate passages, a cross-encoder is asked which passage better answers the question. A 2019 guide that contains the phrase, the steps and a worked example better answers the question than a replacement whose corresponding line reads "Request environment access via the Access Portal" followed by a link. Both documents were judged on the only axis a relevance model has, and the older one won on merit. There is no reranker, no query rewriting and no embedding upgrade that changes this, because the property that makes the 2019 guide wrong is not a property of the query-document pair. It is a property of the document's relationship to another document, and relevance models cannot see relationships.

An evaluation set makes the size of the effect visible, and it is worth reading as a gap rather than as two levels. Across a 100-query onboarding evaluation set, the 2019 guide ranked first on 78 queries. Adding a cross-encoder reranker over the top 50 candidates — the standard move when the ordering looks wrong — left it ranking first on 71 of them. The difference is seven queries, and those seven are characterised: each contained a term that exists only in the new document. Everything else stayed exactly where it was, and the mean reciprocal rank of the new guide, which averages one divided by the position it lands at across the query set, improved from 0.21 to 0.26.

That the seven movers are all vocabulary cases is the finding, not an aside. Where the two documents used different words, the reranker could act, because the difference was visible inside a query-passage pair. Where they used the same words about the same procedure, it could not, because there was nothing in either pair to distinguish them. The reranker behaved correctly throughout, and a stronger one would behave correctly in the same way.

Version state as an attribute the query can filter on#

If the ordering cannot be repaired, the candidate set has to be. The superseded revision must not enter the candidate set for ordinary queries at all, which requires the index to carry version state as a filterable attribute rather than as prose inside the body text.

{
  "chunk_id": "c_8841a2",
  "doc_id": "conf:ENG:4417",
  "doc_family_id": "fam:eng-onboarding",
  "doc_version": 3,
  "version_state": "superseded",
  "superseded_by": "conf:ENGX:9102",
  "effective_from": "2019-04-02",
  "effective_to":   "2023-06-15",
  "doc_class": "runbook",
  "content_hash": "sha256:1f0c…",
  "text": "Getting access to staging is a two-step process. First…"
}

version_state takes one of current, superseded, draft, retired, or unknown, and unknown is not a placeholder — it is the honest value for most of an inherited corpus and it needs to be filterable, because the decision about what to do with unknowns is a decision, not an oversight. The doc_family_id is the identity that persists across revisions, and it is the field that makes superseded_by meaningful; without it you have two unrelated documents, one of which happens to point at the other.

Retrieval then filters rather than reweights.

from qdrant_client import models

def search(query_vec, k=8, include_historical=False):
    states = ["current", "unknown"]
    if include_historical:
        states += ["superseded", "retired"]
    return client.query_points(
        collection_name="corpus",
        query=query_vec,
        limit=k,
        query_filter=models.Filter(must=[
            models.FieldCondition(
                key="version_state",
                match=models.MatchAny(any=states),
            )
        ]),
    )

Two things about that snippet are load-bearing. unknown is included by default, because excluding it on an inherited corpus removes 90% of the index on day one; the migration path is to move documents out of unknown class by class rather than to change the default. And include_historical exists as a parameter because there are legitimate queries for the old text — a compliance question about what the onboarding process required in 2021, an incident review reconstructing what someone was told — and those queries need a route to documents that ordinary queries must never see. A route the caller opts into, and that is logged, is a very different object from a document that leaks into every result.

Those two lines are worth separating into the parts you must have and the parts you chose. The requirement is that version state be an attribute the query engine can filter on, and that a historical route exist for the readers who genuinely need the old text. Including unknown by default and expressing the historical route as a boolean parameter are design decisions, taken because an inherited corpus starts almost entirely unknown and because an opt-in argument is auditable. A different corpus, one whose ingestion has always set state, would be entitled to default the other way.

Delete, demote, or leave a marked-as-dead record in place#

Deleting the 2019 guide is the intuitive move and it is genuinely tempting, because deletion is the only intervention that cannot be outweighed by a strong enough textual match. It is also frequently not available and, where it is available, more expensive than it looks. The alternative it gets weighed against is a tombstone, a marked-as-dead flag on a record that stays in place, which here means the document keeps its row and its vectors while the filter above stops it reaching any ordinary query.

Read that table down its second column rather than across its rows. Three of the four approaches answer no to whether a strong enough textual match can defeat them, and one answers yes. That single cell is the argument of this section: every other difference between the rows is a matter of cost and recoverability, while the demote row differs in kind, because its outcome depends on a quantity nobody controls.

The demote row is where most teams land first, because a fixed penalty — subtract 0.15 from the score of anything superseded — is a two-line change that requires no schema work. Its behaviour follows from one line of algebra. Call the margin the amount by which the superseded document's score exceeds its replacement's on a given query. Subtracting a constant from the superseded score reverses the ordering on that query exactly when the margin is smaller than the constant, and leaves the ordering untouched when the margin is larger. A fixed penalty therefore does not fix a corpus; it partitions a query set at the value of the penalty.

The onboarding numbers are that partition, and they close with no residual. On the onboarding set, the 2019 guide beat its replacement by a margin ranging from 0.03 to 0.22 depending on the query, so a 0.15 penalty fixed 61 of the 78 queries and left 17 unfixed. Those two counts sum to exactly the number of queries the old guide had been winning, which is what tells you the partition is complete: every query the old guide won fell on one side of the penalty or the other, and there is no third category. The penalty sits inside the observed margin range, so both sides are necessarily occupied.

The stumble here is worth naming, because it is the move almost everyone makes next. The instinct is that the penalty was merely mis-tuned, and that a larger constant would clear the remaining queries. It is a tempting belief for a good reason: the change is two lines, it needs no ingestion work, and it visibly fixed most of the query set on the first attempt. It is wrong because a constant is held fixed while the margin varies freely, and the margin is the dangerous quantity. The same penalty simultaneously demoted a superseded API reference below a genuinely irrelevant blog post on a different set of queries where the margin ran the other way. There is no constant that is correct across a corpus, and tuning it per class turns a two-line change into the same schema project you were avoiding, with a worse guarantee at the end. A filter is binary, so its behaviour is stateable: this document cannot be returned by this class of query, full stop.

Tombstoning is, as an analogy, the same idea databases have used for decades. A soft-deleted row keeps its identity and its history while disappearing from ordinary reads, and the discipline that makes it work is that every read path filters, so the exception is the query that asks for deleted rows rather than the query that forgets to exclude them. The analogy holds precisely as far as that discipline does, and the place it breaks is worth stating, because it is where the misconception forms. A relational database has one read path through a query planner that can enforce the predicate centrally; a retrieval system usually has several — the assistant, an evaluation harness, a batch summarisation job, an internal search box — each holding its own client. The corpus version of the discipline is therefore the states list above living in one function that every retrieval path goes through, with the historical route as an explicit argument. The moment two retrieval paths exist and only one filters, the tombstone is decorative.

The reason to prefer it over deletion is that superseded documents have real consumers. Auditors ask what the control said during the period under review. Legal holds forbid destroying documents that are in scope for a matter, and in a regulated corpus the retention schedule may mandate keeping the 2019 guide for seven years regardless of what anyone wants. Post-incident reviews reconstruct what an engineer would have been told at the time, which is a question about the corpus as it was, and answering it from the current corpus produces a confidently wrong reconstruction. Deleting is also unrecoverable in exactly the case where you most want recovery: when the supersession relation turns out to be wrong and the "replaced" document was actually the current one for a subset of teams.

Where version state is written, and where it drifts back#

Version state cannot be computed at query time, because the query knows nothing about document families. It has to be written at ingest, which imposes three requirements on the ingestion path. Document identity has to be stable across revisions and across source systems, so that a page moved from one Confluence space to another keeps its doc_family_id — most sources give you a page ID that survives moves, and the ones that do not need a content-plus-title fingerprint carried in a mapping table. Version state has to be a mutable field on existing points rather than something baked into the vector, so that marking 4,000 documents superseded is a payload write. And there has to be a path that updates state without re-embedding.

client.set_payload(
    collection_name="corpus",
    payload={"version_state": "superseded",
             "superseded_by": "conf:ENGX:9102",
             "effective_to": "2023-06-15"},
    points=models.Filter(must=[models.FieldCondition(
        key="doc_id", match=models.MatchValue(value="conf:ENG:4417"))]),
)

The third requirement is the one that decides whether corpus maintenance is a routine operation or a quarterly project, and the reason is a ratio rather than an absolute cost. Re-ingesting a document means recomputing its embeddings, which is both wall-clock time and money paid per token. Writing a payload field changes bytes in an index that is already built, touches no vector, and costs nothing per token. A team facing seconds will mark documents superseded as they find them; a team facing tens of minutes and a bill will batch the work into a project, and a project is a thing that gets deferred.

There is a subtlety in where the state lives. Version state is a property of a document, and the index stores chunks, so every chunk carries a denormalised copy of its document's state and every state change has to fan out across all of them. The fan-out is what makes the payload-update path essential, and it is also the most common source of a half-tombstoned document. Follow the sequence: a re-chunking run creates new chunks; new chunks get whatever payload the ingestion code sets; ingestion code that hard-codes a default sets that default rather than the document's current state; therefore a document tombstoned in March comes back to life in June when someone changes the chunk size. Each link is checkable, and the failure is invisible because nothing errors.

Guarding that means the state lives in a document table that ingestion reads from, and the chunk payload is derived from it on every write, never authored independently. A reconciliation job that compares chunk payloads against the document table nightly and reports drift costs almost nothing and catches this class of regression within a day.

Two current documents, and a question about the past#

What this machinery does not do is resolve disagreement between two documents that are both current, which is the entire subject of the first two chapters. Tombstoning has no opinion about an SSO handbook and an SSO runbook that state different session lifetimes, because neither replaced the other; they are two live documents that happen to overlap, and a version filter passes both through untouched. The instruments are complementary rather than alternatives, and a team that ships version state expecting it to quieten conflict detection will find the conflict rate unchanged.

The second limit is a query the filter answers wrongly by answering it correctly. A question that is genuinely about the past while being phrased like a present-tense question will be served from current documents, and there is no reliable way to detect the difference from the query text alone. "What is the onboarding process" and "what was the onboarding process in 2021" are separable, because the second carries a tense and a date. "Why was I told to email IT for staging access" is not separable, because it is a present-tense question about a past event, and it will be answered from current documents that no longer contain the instruction the user is asking about. The include_historical route exists for this case and cannot be triggered automatically by anything the system can see.

Both limits share a shape worth carrying forward. A version filter answers the question "is this document in force now", and it answers it well. Every question it fails is a question that was not that one — which of two live documents governs, or what was in force at a past moment the query does not name.

Every intervention in this chapter consumes superseded_by and doc_family_id, and the whole chapter has assumed those fields are populated. In the 2019 guide's case they can be, because the replacement was formal: there is a Confluence banner, a migration ticket, and a person who can tell you which page replaced which. That is the easy case, and it is not what most of a 60,000-page corpus looks like. The relation the filter reads has to come from somewhere, and in an inherited corpus nobody ever wrote it down.