Contradictions, duplicates and stale documents in the corpus

Read these first

• Retrieval strategies for RAG
• Embedding-based deduplication and near-duplicate detection

What you will know

• Detect two retrieved documents that contradict each other

  Semantic similarity can hide incompatible dates, permissions, or version numbers; lexical overlap is not evidence that the claims agree.

• Decide what the system does when they do

  A fixed policy makes the behavior auditable: some domains need authoritative sources, others need the newest version, and safety-critical conflicts should reach the user.

• Handle a superseded document that still retrieves well

  Version metadata, tombstones, or pre-retrieval filters may be necessary because embedding relevance will continue to favor a popular obsolete passage.

• Use recency as a ranking signal where it applies

  A timestamp helps only when freshness correlates with correctness; applying it blindly can demote durable policies or favor an unreviewed draft.

• Trace a confident answer back to a stale document and show the resulting error

  The dangerous case is a plausible response whose citations look valid, making freshness failures hard to detect without replaying provenance and version selection.

Chapters

1. 1

   Cosine similarity cannot tell agreement from disagreement

   Two retrieved chunks state the SSO session timeout as 8 hours and 12 hours, and a near-duplicate detector set at 0.95 reports nothing worth acting on. Two passages agreeing about a number and two contradicting each other look identical in embedding space, because the digit is a couple of tokens in an otherwise matching sentence, and lowering the threshold only flags more pairs that agree. Detection has to compare extracted claims, and this chapter shows where that fails too.

   Read chapter
2. 2

   Silently picking one version is a policy nobody wrote down

   Once a conflict is detected something must happen, and leaving it to the generator produces behaviour nobody can state in advance, that varies run to run and cannot be reviewed after an incident. Work through a contractor production-access rule where a wrong permit and a wrong denial both carry cost, and the policy that follows is surfacing both sources. Authoritative-source and newest-version rules fit other domains, and the point is choosing beforehand.

   Read chapter
3. 3

   A superseded guide can be the best textual match, and stay one

   A 2019 onboarding guide replaced in 2023 keeps ranking first because it is longer, better written and phrased like the way people actually ask, and a cross-encoder reranker does not move it. Nothing inside relevance ranking can fix a document that should not be a candidate at all. Covers version state in the index, why tombstoning beats deletion in a corpus you may need for audit, and what that demands of ingestion.

   Read chapter
4. 4

   Most corpora never declare which document replaced which

   A guide formally replaced in 2023 is the easy case. Across 60,000 inherited wiki pages a 2019 page and a 2023 page are two unrelated rows, and every remedy in this topic — tombstones, current-only filters, class-scoped recency, answer-time version provenance — consumes a supersedes relation that nobody ever wrote down. Here you build one: near-duplicate clustering across revisions of the same content, which is where the duplicate half of this topic actually lives, together with title and date heuristics and curator-supplied replacement mappings. The last question is how much precision you need before tombstoning on an inferred relation rather than a declared one, because suppressing a document that was never really replaced is its own outage.

   Read chapter
5. 5

   Recency helps only where newer correlates with correct

   A global recency boost drops a stable 2018 security policy below a 2024 draft that was never approved, and starts citing recent meeting notes for long-settled procedures. Freshness proxies correctness only within document classes that genuinely supersede one another, so the signal belongs scoped to those classes rather than applied corpus-wide. You will also see why this particular regression generates no user reports and moves no dashboard.

   Read chapter
6. 6

   Faithful, correctly cited, and wrong about this year's policy

   Context recall 0.93, faithfulness 0.95, citation accuracy 0.94, and users are acting on policies replaced over a year ago. Each of those metrics scores the answer against the passages that were retrieved, and a stale passage genuinely does support the answer built on it. This chapter explains the blind spot to a team proud of its dashboard, then sets out the first move when re-ingesting 60,000 documents is out of scope for the quarter: triage to the document classes where staleness is dangerous, obtain or infer supersession for that subset alone, tombstone or filter it, and stand up the one monitor the dashboard lacks — the age and version status of the documents your answers actually cite.

   Read chapter
7. 7

   Replaying a query today hides the version that produced the answer

   A user reports a wrong expense limit, you replay the query and get the right number, and the original citation resolves cleanly to a passage stating the wrong figure. Non-reproducibility is the expected symptom of a stale-document failure rather than evidence of health, because the corpus has moved underneath the bug. Here you work out which explanations chunk IDs alone can separate, and the document-version and index-build provenance that turns the next report into a lookup.

   Read chapter