Most corpora never declare which document replaced which

A supersedes relation is an edge between two documents saying that the second was written to replace the first. Every remedy in this topic consumes one. A version filter needs to know which revision is current before it can exclude the others; a class-scoped freshness term needs to know which documents form an ordered line; answer-time provenance needs a family identifier before it can say whether a cited document has a newer sibling. All of them read a field, and in an inherited corpus that field is empty.

It is empty in a specific and slightly surprising way, which is why the first move is a count rather than a design. The relation is usually not merely unpopulated but unrepresentable: the source system has no place to record it, so what exists instead is a scattering of human-readable hints in prose, titles and link panels, each of which means something slightly different and none of which a query engine can filter on. Building the relation means turning those hints, plus the text of the documents themselves, into an edge with a direction and a confidence attached.

What an inherited export actually declares#

The first query anyone should run against an inherited corpus is a count. On a 60,000-page Confluence export, the number of pages carrying any structured statement that they were replaced by another page was zero. Not low — absent. There is no such field in the schema. What exists instead is 8,100 pages with entries in a "Related pages" panel, which is untyped and bidirectional and includes children, siblings, and pages someone linked once in 2020; 2,300 pages whose body text contains a string like "superseded", "deprecated", or "see instead", in forty different phrasings, sometimes inside a code sample; and 400 pages whose title ends in "(old)".

Read those three counts as a gap rather than as a total. Together they cover a minority of the export, and each of the three carries a different kind of error. The "Related pages" panel is untyped, so an entry is evidence of some relationship and not of replacement, and it is bidirectional, so it does not even carry a direction. The prose markers carry a direction but sit in free text, in dozens of phrasings, sometimes inside a code sample where the word is about something else entirely. The title suffix is unambiguous and rare. What none of them supplies is the pair: a marker saying a page is old does not say which page is new.

So Engineering Onboarding (2019, space ENG) and Getting Started (Engineering) (2023, space ENGX) are two rows with different IDs, different titles, different spaces, and no edge between them. Every mechanism from the previous chapter — tombstones, current-only filters, the historical opt-in route — reads a field that does not exist. The relation has to be constructed, and constructing it is where the duplicate half of this topic does its real work: near-duplicate detection is not being used here to remove redundancy from a result set, it is being used to recognise that two documents are revisions of the same content.

Building the relation out of three signals#

The candidate stage exists because the honest version of the job does not finish. Comparing every document against every other means comparing each of sixty thousand documents against every one of the others and counting each pair once, which is the number of documents times one less than itself, halved — for a corpus this size, close enough to half the square of the document count that the halving is the only correction worth making. That is 1.8 billion pairs, and even at 50 µs per comparison that is 25 hours of CPU per run. The two arithmetics agree: the pair count follows from the corpus size alone, and the running time follows from multiplying that pair count by a per-comparison cost measured separately. Nothing is shared between the two steps, which is what licenses reading 25 hours as a property of the problem rather than of one implementation. Blocking is therefore not an optimisation; it is the difference between a nightly job and no job.

MinHash turns each document into a short signature with the property that two documents sharing a large fraction of their word sequences produce signatures that agree in a large fraction of their positions. LSH banding then splits each signature into groups of consecutive positions and treats two documents as candidates if any one group matches exactly, which is the mechanism from topic 8.18. Banding 128-permutation signatures into 16 bands of 8 rows — sixteen groups of eight positions, which uses all one hundred and twenty-eight of them exactly once — gives a roughly 50% detection probability at Jaccard 0.55 and rises steeply above it, and produced 4,700 candidate pairs in under four minutes.

That steep rise is the property being bought, and it is worth seeing why banding produces it. A pair is a candidate if any band matches, so raising the number of bands raises the chance of at least one hit; a band matches only if all of its rows agree, so lengthening the bands makes each individual hit harder. The two effects pull in opposite directions as similarity rises, and the result is a curve that is nearly flat for dissimilar pairs and nearly flat again for very similar ones, with a sharp transition between. The band and row counts choose where that transition sits.

What differs from deduplication is the shingle size and the threshold. Revisions of a document share long verbatim runs — whole paragraphs survive a rewrite — so 5-word shingles with a Jaccard floor of 0.35 works better here than the tighter settings you would use to catch exact copies, and the recall you want is high because a missed pair costs you a supersession relation while a spurious pair costs you a verification step. The asymmetry in that last clause is the design decision: the stage is deliberately tuned loose because its errors are paid for downstream at a much lower rate than its omissions.

Verification runs embeddings over the survivors and keeps pairs that clear both a Jaccard floor and a cosine floor, which removes the classic MinHash false positive of two documents sharing a long boilerplate header. Union-find over the surviving edges, which walks the kept pairs and merges any two documents joined by one into a single group, produced 1,180 families covering 3,900 documents, so about 6.5% of the corpus sits in a family with more than one member. That share is the family coverage divided by the corpus size, and it closes: the two counts on either side of the division are both stated, and nothing else enters. Families of two are the common case; the largest was eleven, being a deployment runbook that had been copied per-service.

from datasketch import MinHash, MinHashLSH

def shingles(text, n=5):
    w = text.lower().split()
    return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}

lsh = MinHashLSH(threshold=0.35, num_perm=128)
sigs = {}
for doc in corpus:
    m = MinHash(num_perm=128)
    for s in shingles(doc.text):
        m.update(s.encode())
    sigs[doc.id] = m
    lsh.insert(doc.id, m)

candidates = set()
for doc_id, m in sigs.items():
    for other in lsh.query(m):
        if other != doc_id:
            candidates.add(tuple(sorted((doc_id, other))))

Title and path heuristics catch a family that clustering misses, which is the case where a replacement was written from scratch and shares little text with what it replaced. Normalising titles by lowercasing, stripping years, version markers, and parenthetical suffixes turned "Engineering Onboarding" and "Getting Started (Engineering)" into strings that still do not match — that pair only clusters because 40% of the new page was pasted from the old one. Where normalisation does work is on the mechanical patterns: "Deploy Runbook v2" against "Deploy Runbook", "Expense Policy 2024" against "Expense Policy 2022", "Copy of Incident Template". Those three patterns alone produced 610 additional candidate pairs on the corpus, with a manually checked precision of 0.91, which is high because the naming conventions are unambiguous when they are present at all.

Curator mappings are the third signal and by far the highest quality. A spreadsheet with two columns, old_page_id and replaced_by_page_id, sent to the eleven teams that own the document classes where staleness is dangerous, came back covering 740 documents in three weeks. It is slow, it does not scale to 60,000 pages, and it is authoritative in a way nothing inferred can be, which is why it is worth spending the three weeks on the subset where being wrong is expensive rather than trying to cover the corpus.

Direction is a separate inference from grouping#

Knowing that two documents are the same content does not tell you which one is in force, and the obvious signal is unreliable in a specific way. Comparing updated_at says the 2019 guide was last modified on 2024-11-02, because somebody fixed a broken link, and the 2023 replacement was last modified on 2023-08-14. On modification date alone, the direction inverts and you tombstone the current document.

The reason that happens is worth stating as a chain rather than as a caution. A modification timestamp records an edit event. An edit event is an act performed by a person on a page. People edit old pages for reasons unrelated to the page's authority — fixing a link, correcting a name, updating a template — and they do not edit current pages when nothing about them has changed. Therefore the timestamp measures editor attention rather than currency, and there is no reason for editor attention to run in the same direction as supersession.

Three signals combined get this right more often than any of them alone, and what makes the combination work is that they fail for unrelated reasons. Content lineage asks whether one document's distinctive material is largely contained in the other, since a replacement typically absorbs most of its predecessor and adds new sections, which shows up as an asymmetric containment ratio — 0.78 of the old document's shingles present in the new one against 0.41 the other way. Read that pair as a gap: neither number means anything alone, and it is the difference between them, running in one direction, that carries the claim. Inbound link migration asks where the rest of the corpus points now, and a document whose inbound links dropped from 34 to 3 over eighteen months while a sibling's rose from 0 to 29 has been replaced regardless of what its timestamps say. First-publication date, as opposed to last-modified, is more stable than updated_at and is available in most source systems even when it is not in the export you were handed.

There is a case where all three fail together, and it is the one to check for before trusting the combination. A replacement written from scratch shares little text, so containment is symmetric and near zero. A replacement nobody has linked to yet moves no inbound links. And a document whose first publication predates the thing it replaces — a page that existed for another purpose and was repurposed — inverts the date signal too. The three signals are independent in their errors but not in that scenario, which is why a low-confidence family is recorded rather than acted upon.

The signals get combined into a confidence, and the confidence decides the action rather than the relation being treated as binary.

The last column sums to exactly one hundred percent with no residual, and that is what licenses reading the table as a partition: every family sits in exactly one band, the bands are exhaustive, and no family is being counted twice or falling through. Without that property the shares would be four unrelated measurements and the arithmetic below would not be available.

Choosing the threshold from blast radius, not from an error rate#

The threshold question decides whether this work helps or hurts, because the two errors are not symmetric and neither is visible in the same way. A missed supersession leaves the status quo, which is the failure the whole topic is about — bad, known, and already happening. A false supersession suppresses a document that was never replaced, and that failure produces no error anywhere: the document exists, the query returns other things, the user gets an answer built from something else, and nobody files a ticket saying "the retriever silently excluded the right page".

That is the stumble in its strongest form, and it is worth stating as a property rather than as a warning. The observed error rate stays flat at zero while the number of wrongly suppressed documents varies freely, because the two are not connected by any reporting path. A team watching its ticket queue for evidence that the threshold is too aggressive is watching a quantity that cannot move. The threshold therefore has to be chosen from what a wrong suppression would cost, which is knowable in advance, rather than from how often one has been observed, which is not.

The arithmetic of that cost is available from the table. The auto-tombstone band holds about a fifth of the families, and a fifth of eleven hundred and eighty is roughly 250 families. At 0.97 with that band auto-tombstoned, an audit of 100 sampled decisions found 3 wrong, so applying that rate across the band gives around 7 documents wrongly suppressed. Both steps use figures already on the page and nothing else, and the second depends on an assumption worth naming: the audit sampled decisions rather than documents, and a family may hold more than one document — the largest held eleven — so the estimate is a floor rather than a bound, and it is a good floor only because families of two are the common case. Whether that is acceptable depends entirely on which documents they are, which is why the auto-tombstone band should be scoped to document classes where the blast radius of a wrong suppression is small, and curator mappings should carry the classes where it is not.

The 0.85-to-0.97 band is where most of the value sits and it is easy to under-build, because "queue for owner review" sounds like a placeholder for work nobody will do. It becomes real when the queue item is a decision with two buttons rather than a research task. The item shows the two documents side by side with a diff, the signals that produced the score, the inbound link counts, and one question — did the second of these replace the first? — and an owner clears twenty of them in half an hour. Presented instead as a spreadsheet of 400 page IDs, the same work never happens. During the review period those families sit demoted rather than tombstoned, which is the one place a score penalty is the right instrument: it is explicitly a temporary hedge on an uncertain relation, not a substitute for state.

There is a cheap safety net worth building at the same time, and it works by finding a reporting path where the previous paragraph said there was none. Every auto-tombstone writes a row with the family, the signals, the confidence and the date, and a monthly job reports any tombstoned document that is still receiving traffic in the source system — people opening it in Confluence, linking to it, editing it. A superseded page that thirty people opened last month was probably not superseded, and that signal costs one query against the wiki's analytics API. The retrieval system cannot see a wrong suppression, but the wiki can see that humans are still reading the page.

Partial supersession, and the edge this relation cannot represent#

Partial supersession is the honest limit. A 2023 standard replaces sections 4 and 7 of a 2019 policy and leaves the remaining nine sections in force, which is extremely common in policy corpora and is not representable as a document-level superseded_by edge. The reason is structural rather than a gap in the implementation: the edge joins two documents, and the fact to be recorded joins two sections, so the representation is one level too coarse for the claim. Tombstoning the 2019 policy removes nine live sections from retrieval; leaving it current keeps two dead ones. Section-level supersession is the correct model and almost nobody has the metadata for it, so the practical compromise is to keep the document current, attach the conflict detection from chapter 1, and let the resolution policy handle the two sections that now disagree — which converts a corpus-structure problem into a per-answer one and accepts the cost of that.

The relation this work builds is also not a knowledge graph, and the temptation to grow it into one should be resisted for as long as possible. It has one edge type with a direction and a confidence, it is consumed by exactly one filter, and it can be rebuilt from scratch in an afternoon. Each of those three properties is what makes a wrong edge cheap: one type means there is one thing to get wrong, one consumer means there is one place to look when it is wrong, and rebuildability means a bad run is undone by rerunning rather than by repair. Add a second edge type and all three properties weaken at once. The write-up From a stale README to a security research intelligence platform is worth reading for how far the same primitives go when document identity, deduplication and version tracking are treated as the core of the system rather than as maintenance.

The confidence attached to every edge is also not a probability, and treating it as one is the third limit. It is a score assembled from signals whose errors are correlated in the one case named above, calibrated against a single audit on a single corpus, and it is used only to select a band. It supports the comparison between bands and it does not support a statement that a given edge is right ninety-seven times in a hundred.

None of this stops a team from reaching for the thing that requires no relation at all. If newer documents are usually the right ones, and the whole difficulty here is establishing which document is newer than which, then a scoring term on absolute age looks like it gets the same benefit for one line of code and no ingestion work at all. The relation is what makes "newer" mean anything, and a term that skips it is measuring a different quantity entirely.