Faithful, correctly cited, and wrong about this year's policy

A retrieval-augmented generation system is usually watched through three or four metrics. Context recall asks whether the retriever found the passages a human annotator marked as the right evidence. Faithfulness asks whether every claim in the answer is supported by the passages the generator was handed. Citation accuracy asks whether each citation points at a passage that says what the answer attributes to it. Answer relevance asks whether the answer addresses the question. All four can sit high for months while an assistant hands people policies that were replaced over a year ago.

That is not a failure of any of the four, and treating it as one is why teams spend quarters tuning the wrong thing. Each metric compares two objects that both live inside the system, and the fact that is wrong lives outside it — in whether the document the system reached for is the one currently in force. A measurement whose two endpoints are both inside the box cannot report on something outside it, and four such measurements cannot either.

Every one of those metrics is conditioned on retrieval#

The three metrics differ in what they compare, and none of the three comparisons involves the world.

Read that table down its second column rather than across its rows. Every entry in it names two objects, and in every case both objects are artefacts of the system: retrieved chunks and gold chunks, answer and context, citation and passage, answer and question. The fourth column is then not a list of separate weaknesses but the same absence written four times.

Every dotted edge in that diagram has both of its endpoints inside the system. The one edge that would connect the corpus to the state of the world is the one nothing measures, and no arithmetic over the others reconstructs it. That last clause is the load-bearing one and it is not a rhetorical flourish: combining measurements can only produce a function of the quantities measured, and the quantity in question appears in none of them, so no combination — no weighting, no ratio, no composite score — contains it.

Faithfulness is the one to make concrete when explaining this to the team, because its failure is not a weakness in the metric but a consequence of its definition. Faithfulness asks whether every claim in the answer is supported by the retrieved context. The answer said $75. The 2023 revision of the Travel & Expense Policy says, in plain text, that the daily meal allowance is $75. The claim is entailed by the context with no hedging, so faithfulness scores it 1.0, and it scores it 1.0 correctly. A metric designed to catch a generator inventing facts is working exactly as intended on an answer where the generator invented nothing.

Context recall is the more surprising one and it is worth walking through slowly, because teams assume it covers document selection. Follow the sequence that produced the label. The gold chunks for the expense query were labelled by an annotator who searched the corpus for the meal allowance, found the 2023 policy at rank one, and marked its chunk as relevant. The 2025 revision was in the corpus and ranked seventh, and the annotator, having found a clear authoritative-looking answer, did not scroll. The gold label is now the stale chunk, so a retriever returning the stale chunk scores recall 1.0 and a retriever returning the correct one scores 0.0.

The consequence is stronger than a blind spot and it is the thing to say out loud. The evaluation set does not merely fail to catch the fault; it encodes it, and it will penalise the fix. An engineer who ships the correct filter watches context recall fall and, absent this explanation, concludes the filter is wrong. That is a common enough pattern that any judged set older than the corpus problem should be re-audited before it is used to evaluate a corpus intervention.

The stumble underneath all of this is a belief that a defence works: that a green dashboard is evidence of correctness because it is a measurement rather than an opinion. It is tempting for a good reason — those numbers took two quarters to reach, they are accurate, and they do catch the faults they were built for. It is false because every one of them is a quantity that stays high while the dangerous property varies freely underneath. Nothing about a corpus drifting out of date moves any of the four, in either direction, and the annotator case shows the coupling can even run backwards.

The common structure across all four rows is that each metric scores the answer against the retrieved set, so together they measure the quality of generation conditional on retrieval and the quality of retrieval conditional on the corpus. The probability that the corpus state used was the corpus state in force is not conditioned on anywhere, and there is no combination of these metrics that recovers it.

The metric the board is missing#

The missing measurement does not require judgments and does not require an LLM. For every answer served, log the version state and age of every document cited, then report the fraction of answers that cite at least one document whose family has a newer revision, or whose state is superseded, draft or retired.

-- cited-document version status, last 30 days
SELECT
  date_trunc('day', a.answered_at)                        AS day,
  count(*)                                                AS answers,
  count(*) FILTER (WHERE s.stale_citations > 0)           AS with_stale,
  round(100.0 * count(*) FILTER (WHERE s.stale_citations > 0)
        / count(*), 1)                                    AS pct_stale,
  round(avg(s.max_cited_age_days))                        AS avg_oldest_citation_days
FROM answers a
JOIN LATERAL (
  SELECT
    count(*) FILTER (
      WHERE d.version_state IN ('superseded', 'retired', 'draft')
         OR d.doc_version < f.max_version
    )                                    AS stale_citations,
    max(date_part('day', a.answered_at - d.effective_from)) AS max_cited_age_days
  FROM citations c
  JOIN documents d          ON d.doc_id = c.doc_id
  JOIN doc_family_max f     ON f.doc_family_id = d.doc_family_id
  WHERE c.answer_id = a.answer_id
) s ON true
WHERE a.answered_at > now() - interval '30 days'
GROUP BY 1 ORDER BY 1;

Two conditions in that filter are doing different jobs and both are needed. The first tests the document's own declared state, which catches a document somebody has marked dead. The second compares the cited document's version against the highest version in its family, which catches a document nobody marked at all but which has been quietly overtaken. A corpus part-way through a supersession project will have documents in each category, and a monitor testing only the first would report progress that is really the absence of labels.

Backfilled over 30 days of logs on the inherited system — 48,000 answers — that query returned 11.4% of answers citing at least one document with a newer revision in its family, and a mean oldest-citation age of 620 days. It ran in nine seconds. The whole visible defect was sitting in the log tables the entire time, unqueried, because nobody had asked a question whose answer was not about the generator.

That number is also the one to put in front of the team, because it converts an anecdote about three users into a rate with a denominator. Three complaints in a week reads as noise; 5,470 answers in thirty days built on superseded documents does not, and it makes the size of the remediation arguable rather than a matter of taste. The two statements are the same measurement: the rate applied to the answer count over the window produces the absolute figure, so the rate is not an abstraction laid over the complaints but the complaints counted properly.

The rate is a screen rather than a verdict, and it is worth calibrating once before anyone treats it as a defect count. A hundred flagged answers sampled from that thirty-day window, with the cited revision diffed against the current one by hand, came out as 58 where the change was material — a different number, a different owner to contact, a procedure with a step removed — 33 where the revision was cosmetic, and 9 where the two revisions could not be told apart without asking the owning team. Those three counts sum to exactly the hundred sampled, which is what makes them a partition and lets the material share be used as a multiplier. Cosmetic revisions are common in policy corpora because re-approval cycles bump a version and a review date without touching the body, so roughly a third of what the monitor flags would not have produced a wrong answer. The materially stale rate is therefore around 6.6% rather than 11.4%, and the honest way to report it is both numbers with the sample behind them.

The error runs in the other direction as well, and that one is harder to see. Sampling a hundred answers the monitor scored clean, from the same policy classes, turned up four that were wrong for the same reason: the meal allowance had moved out of the Travel & Expense Policy into a new Global Reimbursement Standard with its own family identifier, so the cited document was the current revision of a family that no longer governs the question. Supersession across families is invisible to a query that compares a document only against its own siblings, which means the monitor under-reports in exactly the classes where reorganisation is most likely. Neither adjustment changes what to do next, and both change what to claim: the monitor is a trend line whose composition needs re-checking each quarter, and a rate that halves is evidence of progress rather than proof of correctness.

Ranking document classes by citation share against cost#

Re-ingesting 60,000 documents this quarter is not on the table, and it is also not what the situation requires, because the harm is not distributed evenly across the corpus. Rank document classes by the product of how often they are cited and what a wrong answer from them costs.

That table closes twice, and both closures are worth checking before trusting anything derived from it. The citation shares sum to exactly one hundred percent, and the document counts sum to exactly the corpus size. So it is a partition taken two ways — every answer's citation falls in one class, and every document falls in one class — and the arithmetic below is therefore a statement about the whole corpus rather than about a sample of it.

The top four rows are 3,100 documents, 5% of the corpus, and 46% of citations. Both of those follow from the columns: the four document counts sum to that subset size, which against the corpus total is the stated share, and the four citation shares sum to the stated share of citations. The gap between those two numbers is the entire argument for triage — a twentieth of the documents carries close to half of the attention — and it is a gap that only exists because the table is a partition on both axes. Note that one later row happens to carry the same document count as the four-row subtotal; the two are unrelated and should not be read as a check on each other.

That subset is the one to obtain supersession for, using the curator mappings from chapter 4 where an owning team exists and the inference pipeline where it does not, and it is a three-week job rather than a quarter.

The order matters more than the contents. Turn on the cited-document monitor first, because it is a query against data you already have and it establishes the baseline that every later change is judged against. Then triage. Then obtain supersession for the triaged subset, tombstone it, and watch the rate move. A team that does the tombstoning first has no way to demonstrate that three weeks of work accomplished anything, which is how corpus hygiene projects get cancelled at the halfway point.

The boost that would have hidden it, and what the monitor still cannot see#

The team's proposal will be the global recency term, and the strongest version of that case deserves stating rather than dismissing. It is one configuration line and ships this week, against an ingestion and supersession project measured in months. It requires no new metadata, which matters when 29% of the corpus has no reliable class label. It targets exactly the observed failure, since in every one of the three complaints the correct document was newer than the cited one and present in the index at a lower rank. And on many corpora it would help on net, because the classes with genuine revision lines are the ones users query most.

The reason to decline it is not that it never helps. It is that its regressions are invisible to the same four metrics that failed to catch the original fault, so after shipping it you have no instrument that can tell you whether the trade was good. The demotion of durable policies beneath drafts and meeting notes described in the previous chapter produces no user reports and no dashboard movement, which means the visible failure has been exchanged for a diffuse one at an exchange rate nobody can measure. A mitigation whose downside is unobservable by the monitoring that missed the upside is a second undetected fault, not a stopgap. Given the cited-document monitor above, the argument changes shape — the class-scoped version from the previous chapter can be shipped with a shadow diff and a rate to watch — but the sequencing is fixed: the instrument comes before the intervention.

What the monitor does not do is tell you a document is wrong. It reports version status, so a current document containing a stale fact — the policy changed and nobody edited the page — scores clean, and that is a real hole with no cheap instrument. It also reports at the population level, so when the next individual complaint arrives, the rate tells you the class of fault is plausible and tells you nothing about that particular answer.

Those two limits are the same limit seen from two ends. The monitor compares a document against its own family and reports a population rate, so it is blind both to a fact that changed without any document changing and to which particular answer among thousands was the one that went wrong. It narrows where to look and it never finishes an investigation.

Working a single report back to the version that produced it turns out to be its own problem, and it fails in a way that looks, at first, like good news. The query that produced a wrong answer last week will frequently produce the right answer today, and a clean replay is the least informative evidence available.