A citation asserts which passage, and concatenated context cannot check it

An answer that carries [p-3310] after a sentence is making a second assertion alongside the sentence itself: that this particular passage is where the sentence came from. That assertion has a reader — somebody who clicks the link to check — and a failure mode of its own, which is that the passage says something else.

Nothing built so far in this topic can detect it. Faithfulness, from chapter 2, splits the answer into claims and asks whether the retrieved passages support each one, and every implementation in common use joins those passages into a single block of text before asking. That join is what makes the question answerable in one judge call, and it is also the step that throws away which passage is which. This chapter builds the metric that keeps the boundaries: for each claim, does the passage the answer named entail it.

Support anywhere versus support from that passage#

A specimen makes the gap concrete, and it is worth reading the citations rather than the prose.

Q: "How long do we keep audit logs, and can I extend it?"

retrieved:
  [p-3310] security/audit-logs.md  "Audit log retention is 90 days on the
           Business plan and 365 days on Enterprise."
  [p-3311] security/audit-logs.md  "Retention cannot be configured from the
           dashboard. Contact your account manager to discuss extensions."
  [p-3312] billing/enterprise.md   "Enterprise agreements may include custom
           data retention terms negotiated at contract signing."

answer:
  "Audit logs are retained for 90 days on the Business plan. [p-3310]
   Retention can be extended through a custom term in an Enterprise
   agreement. [p-3310]
   Changing it is not possible from the dashboard. [p-3312]"

faithfulness (concatenated context): 1.00

Every claim in that answer is supported by something in the retrieved set, which is what the 1.00 says and it is not wrong. The second claim is supported by p-3312 and cites p-3310, and the third is supported by p-3311 and cites p-3312. An agent clicking either link reads a passage that does not say what the sentence above it says, decides the assistant is unreliable, and stops trusting the sourced ones too.

Written as a set operation, faithfulness asks whether there exists some passage p in the retrieved set P such that p entails claim c. A citation makes a stronger assertion: this claim comes from this passage. The metric for it asks whether the specific passage the answer named entails the claim — not whether some passage does. Those two questions come apart in one direction only, and the asymmetry is worth deriving rather than accepting, because it is what makes a high faithfulness score compatible with total attribution failure. If the named passage entails the claim, then a passage entailing the claim exists, since the named one is a member of the set. So every claim passing the citation check passes faithfulness. The converse does not hold: a claim can be entailed by some member of the set and not by the one it names, which is what the second and third claims above do. An answer can therefore score 1.00 on faithfulness with every single citation pointing at the wrong source, and no configuration of faithfulness detects it.

That is the shape of belief worth naming here, and it is what kept the failure alive on the pipeline in this chapter. Faithfulness was being treated as a defence against bad sourcing, and the property it would have to be sensitive to is which passage supports which claim. Concatenation removes that property before the judge sees anything. The score reads the same in the world where every citation is correct and in the world where every citation is scrambled, because the text handed to the judge is identical in both — the passages are the same passages, in the same block, and the IDs are not part of the comparison. A number that cannot move when the fault appears is not evidence about the fault, however carefully it is computed.

The distinction is the one between a bibliography and a footnote. A bibliography at the end of a paper asserts that these works informed this text, and it is satisfied collectively; a footnote asserts that this sentence comes from this page of this work, and it is satisfied or violated one sentence at a time. Nobody would accept a reviewer who checked a paper's footnotes by confirming that each claim appeared somewhere in the bibliography's collected works, and that is the check a faithfulness score over concatenated context performs. The comparison is an analogy and it breaks at the point where the work gets done: a scholar attaches a footnote as a separate deliberate act, after writing the sentence, whereas a generator emits the ID and the prose in the same pass, so there is no moment of attribution to audit and no author to ask. That is why the repair has to be a measurement rather than a review step.

Citation accuracy as its own metric#

The unit is a claim, as it was in chapter 2, and the additional input is the set of passage IDs the answer attached to that claim.

@dataclass
class CitedClaim:
    text: str
    cited_ids: list[str]

def citation_accuracy(claims: list[CitedClaim], passages: dict[str, str],
                      judge: Judge) -> dict[str, float]:
    cited = [c for c in claims if c.cited_ids]
    correct = 0
    for c in cited:
        # the judge sees ONLY the passages this claim names
        shown = {pid: passages[pid] for pid in c.cited_ids if pid in passages}
        if judge.entails(c.text, shown):
            correct += 1
    return {
        "citation_accuracy": correct / len(cited) if cited else float("nan"),
        "citation_coverage": len(cited) / len(claims) if claims else float("nan"),
    }

The line that does the work is the shown dictionary, which is built from c.cited_ids rather than from passages. Everything else in this chapter follows from that one restriction of what the judge is allowed to read.

Two numbers come out because two things can go wrong and they need separate fixes. Accuracy is the fraction of cited claims whose named passages entail them. Coverage is the fraction of claims that carry a citation at all, and it exists because the two ratios have different denominators: accuracy divides by len(cited) and coverage divides by len(claims). A model that has learned citations are checked will stop citing, which shrinks accuracy's denominator without shrinking coverage's, so accuracy can rise while the answers get less attributable. On the pipeline below, coverage was 0.74, meaning a quarter of the assertions in the answers were unattributed and therefore contributed nothing to the accuracy denominator. Reporting accuracy alone lets a system reach 1.00 by citing one claim per answer.

Getting to a list[CitedClaim] is where most of the implementation effort actually goes, because the model attaches citations to sentences and the metric needs them attached to claims, and those are not the same units. Decomposition rewrites a sentence into one or more standalone claims and in doing so throws away the character offsets that the [p-3310] markers were sitting next to. The fix is to decompose per sentence and carry the sentence index through, so each claim inherits the citations of the sentence it came from.

def cited_claims(answer: str, judge: Judge) -> list[CitedClaim]:
    out = []
    for sentence in split_sentences(answer):
        ids = re.findall(r"\[(p-\d+)\]", sentence)
        clean = re.sub(r"\s*\[p-\d+\]", "", sentence)
        for claim in decompose(clean, judge):        # chapter 2's splitter
            out.append(CitedClaim(text=claim, cited_ids=ids))
    return out

Inheriting the sentence's citations means a sentence carrying two claims and one citation produces two claims that each cite that passage, which is the right reading of what the answer asserted and is also how the over-broad case creeps in. It also means the sentence splitter is now part of your measurement, so a splitter that breaks on the period in "02:00 a.m. UTC" attaches citations to fragments and depresses the score for reasons that have nothing to do with the pipeline. Running the extractor over 50 answers and reading the output before trusting any aggregate is twenty minutes that saves a week.

The bottom of that diagram is what makes the metric diagnostic rather than merely another score, and it is the asymmetry above drawn as a flowchart. A failed citation splits into a claim that is grounded in the retrieved set and points at the wrong member of it, and a claim that nothing supports. The second branch is where the two metrics overlap: those claims fail faithfulness as well, so faithfulness already reports them. The first branch is invisible to faithfulness by construction, and it is the population this metric exists for. Those two have different owners — the first is an attribution problem in how the generator or the post-processor attaches IDs, the second is ordinary hallucination — and lumping them together produces a number that moves for two unrelated reasons.

There is a third failure that neither branch catches unless you ask for it. A claim that cites five passages, one of which supports it, passes the entails check as written, because the judge is shown all five and finds support among them. That is the over-broad citation, and it is the shape a model naturally drifts toward when it is told to cite its sources and penalised for missing attributions — citing everything satisfies both instructions at once. Whether to count it as correct is a decision about what your citation means: if the UI highlights a single source per sentence, require exactly one cited ID and score a multi-citation claim as failed; if it lists sources, keep the any-of rule and report the mean number of IDs per claim alongside accuracy, since a rise from 1.2 to 3.4 is the metric being gamed. Note what a claim citing all of the retrieved passages would reduce to: the judge is then shown the whole set, which is the concatenated-context question again, and citation accuracy collapses into faithfulness. The over-broad case is not a separate defect so much as a slide back to the metric this chapter is distinguishing itself from.

The two scores in the third row come from the same three claims. All three are supported somewhere in the set, so faithfulness is three of three; exactly one of them cites the passage that supports it, so citation accuracy is one of three, which is 0.33. The fifth row is the sharpest test of the difference: shuffling which ID sits next to which sentence changes nothing about the concatenated block and therefore cannot move faithfulness, while it changes every comparison citation accuracy performs.

The last row is the reason both belong on the dashboard. An answer that is fully grounded and wrongly attributed is more dangerous to a user than an obviously unsupported one, because the citation invites the reader to stop checking.

What the metric does not judge is whether the cited passage was the best available source. A claim about Business-plan retention that cites a passage which mentions retention in passing, while a more authoritative passage sits at rank 2, scores as correct, because entailment is a yes-or-no relation and the metric is built on it. Nor does a perfect citation score say anything about whether the claim is true, since a passage can entail a claim and be out of date. And a claim that genuinely requires two passages jointly — the retention period from one and the extension route from another — has no correct single citation, so a system that answers multi-hop questions well, the ones whose answer has to be assembled from more than one passage, will show a citation accuracy ceiling below 1.00 that no amount of attribution work removes. That ceiling is a property of the question mix rather than of the pipeline, which means a citation accuracy target has to be set against the share of multi-hop questions in the traffic or it is a target nobody can hit.

A first score of 0.55, and what it might be#

Running the metric on 200 answers produced a citation accuracy of 0.55 with coverage at 0.74. That number has two candidate explanations that lead to completely different work, and it is worth naming both before touching anything.

The pipeline may genuinely be attributing claims wrongly. Generators asked to cite inline frequently reproduce the ID of the passage they were reading a sentence ago, or the first ID in the prompt, or the ID of the passage most similar to the whole answer rather than to the claim. Prompt format contributes: IDs placed only at the top of each passage, far from the sentence that matters, are attached to claims less reliably than IDs repeated at the start of every paragraph, and a model asked to write the answer first and add citations afterwards attributes worse than one that emits them inline.

The checker may be the thing that is wrong. Aligning a decomposed claim to the passage that supports it is itself a fallible model call, and it fails in specific ways here. Chunks produced with overlapping windows mean two passage IDs contain the same sentence, so a claim citing p-3310 is marked wrong when the checker found its support in p-3311 and the two overlap by 40 tokens. Decomposition can also produce a claim that merges material from two passages, which no single cited ID can entail. And a judge shown one short passage rather than the whole context is a stricter judge — the same claim, the same supporting sentence, verified in isolation, is marked entailed less often than when it sits in a longer block, which is a direct consequence of the design decision that makes this metric work at all.

Notice that all three of the checker's failure modes push the score in the same direction, which is down. Overlapping chunks, fused claims and a stricter isolated judgement each turn a correct attribution into a reported error, and none of them turns a wrong attribution into a reported success. So 0.55 is a lower bound on the true attribution quality rather than an estimate of it, and the honest reading of a first run is that the pipeline is no worse than the number says. That is enough to justify investigating and not enough to justify choosing between the two stories.

A citation, then, is a claim about provenance, and provenance is exactly the information a concatenated context destroys. Measuring it means restricting what the judge may read to the passages the answer named, reporting coverage alongside accuracy so that a model cannot improve one by abandoning the other, and reading a failure as either a mis-attribution or an ordinary hallucination rather than as one undifferentiated number.

Both stories predict a score near 0.55 and neither can be ruled out from the aggregate, which is the position every metric in this topic eventually puts you in. Faithfulness, answer relevance and citation accuracy are all verdicts produced by a model, on a corpus and a task it was never validated against, and a verdict of that kind is a reading from an instrument whose own error nobody has measured yet.