Mark the spans a source supports, and decide what the unsupported sentence gets

The first version of span marking in Atlas underlined every sentence that came back entailed and coloured the underline by score. On a typical five-sentence answer that meant three solid underlines, one dashed, and one bare sentence, and the handler feedback arrived within a day: it looked, in one handler's words, like a ransom note, and the marks had no information in them because almost everything was marked. Sixty-one percent of sentences were fully entailed, so the dominant visual signal in the answer pane was the ordinary case.

Inverting it fixed the density problem and improved the semantics at the same time. Entailed prose renders plain. Partial support gets a subtle marker on the specific clause that outran its source. No support gets an explicit mark that names itself. On the same traffic that took the marked share of answer text from 71% to 4.4%, and it changed what a mark means from this was checked to this needs your attention, which is the meaning a reader can act on.

<p class="answer">
  A policyholder may appeal a declined subsidence claim within 60 days of the
  decision letter.
  <mark data-support="partial" data-sources="hchm_14_3_c1,apnote_2_c4">The appeal
  must be submitted in writing and include any new structural evidence obtained
  since the original survey.</mark>
  <mark data-support="none">If the 60-day window has lapsed, a handler can grant a
  one-time 30-day extension where the policyholder can show that the decision
  letter was delayed in transit.</mark>
</p>

Below the sentence, the marks stop being cheap#

Sentences are a convenient unit because the entailment pass of chapter 1 operates on them, and they are frequently the wrong unit. The second sentence above is a single sentence carrying two claims: that the appeal must be written, which clause 14.3 supports, and that it must include new structural evidence, which appears nowhere in that clause and was assembled from an unrelated procedure note. A reader who sees the whole sentence marked partial knows something is off and does not know which half.

Getting below the sentence takes a claim-decomposition pass: split each sentence into atomic claims, check each against retrieval, then map the results back onto character ranges in the original text. On Atlas it added about 180ms and $0.0009 per answer, and it moved localisation from a sentence to a clause on 78% of the mixed-support cases. It also introduced a failure of its own that took three weeks to find, which is that decomposition dropped a negation on roughly 3% of sentences — the extension may not be granted retrospectively decomposed into a claim about granting extensions retrospectively, which the corpus supports, so the sentence came back entailed when its actual meaning was contradicted. That class of error is invisible in aggregate metrics and catastrophic in individual cases, which is a reasonable argument for keeping sentence-level marking in a high-stakes surface until the decomposition pass has been evaluated on negation and quantifier cases specifically.

How the marks are drawn interacts with all of this more than it looks. Three renderings were tried on the same data. A background tint on the marked span is the most noticeable and the most disruptive to reading, because coloured prose is harder to read at speed and handlers were reading forty answers a day. A dotted underline is quiet enough to live in body text and easy to miss on a laptop screen in a bright office, which two handlers demonstrated by not seeing it. A margin rail — a short vertical bar beside the line, with the mark's meaning available on hover — survived both objections, kept the prose clean, and had the additional property of being positionally stable, which matters for the streaming case below. The final design used a rail for partial support and a rail plus an inline label for the unsupported case, on the argument that the two states are not the same severity and should not be the same weight.

The marks are themselves probabilistic, and pretending otherwise is the same error as the fluent prose it is meant to correct. Measured against 600 hand-labelled sentences, the entailment pass ran at 0.88 precision and 0.79 recall on the unsupported class. One unsupported sentence in five was not marked. That number belongs in whatever the team tells its users about the feature, and it is the reason the mark reads not found in your documents rather than false.

The decision nobody writes down#

Once a sentence is known to have no support, something has to happen to it, and the range of options is wider than most teams consider before the first incident forces the question.

Atlas ran the regenerate path first, because it feels like the responsible option: retry with an instruction to use only the retrieved text, and if the retry comes back clean, show that. It succeeded 64% of the time, which left a fall-through that still needed one of the other policies, so the team had to make the decision anyway and had bought a 2.1-second delay on 15% of answers to defer it. Regeneration is worth keeping where the unsupported claim is peripheral and the answer is otherwise expensive to reproduce; it is not a policy on its own.

What the product settled on was a policy keyed to what the answer is for, written down in one place rather than distributed across three components.

STAKES = {
    "policy_lookup": "medium",     # handler reads, then acts
    "letter_draft": "high",        # text leaves the building
    "internal_summary": "low",     # orientation only
}

def render_policy(surface: str, profile: SupportProfile) -> Action:
    stakes = STAKES[surface]
    if stakes == "high" and profile.unsupported_sentences:
        # nothing unverified is allowed into a customer-facing draft
        return Action.SUPPRESS_WITH_STUB
    if stakes == "medium" and profile.unsupported_share > 0.5:
        # more than half unsupported is not an answer with a caveat
        return Action.REFUSE_WITH_PASSAGES
    if profile.unsupported_sentences:
        return Action.ANNOTATE
    return Action.RENDER_PLAIN

The stub matters more than it looks. A suppressed sentence with nothing in its place produces an answer the reader believes is complete, which is a new and worse version of the original problem — the interface is now asserting completeness it cannot back. A stub reading 1 sentence removed: not found in your documents costs a line, tells the reader the corpus may be incomplete, and gives them the option to ask again with different wording, which handlers did on 19% of stubs.

Attribution arrives after the words do#

All of the above assumes the marks and the text appear together, and in a streaming interface they do not. Attribution is the mapping from each span of the answer back to the passage that supports it, and it is computed after the span exists, which is the whole of the problem here. Tokens land at roughly 45 per second, so a five-sentence answer is fully on screen at about 4.1 seconds, while the entailment pass for a given sentence can only start once that sentence has finished. On Atlas the marks trailed their text by 240–310ms.

That interval is a state, and it has to be rendered as one. The rule is that the unresolved state must never look like the resolved-clean state, because the resolved-clean state is the one that says we checked this. Neutral, visibly pending text is honest; plain text that later grows a warning mark is not, since for a quarter of a second the interface told the reader something it did not know.

The Unverified state is the one that gets left out and the one that decides whether the whole mechanism is trustworthy. When the entailment service is slow, down, or rate-limited, the default has to be that the answer degrades to visibly unchecked rather than silently to clean. A system that renders unmarked prose when its checker is unavailable is at its least reliable precisely when it looks most confident, and that is the failure mode users never forgive because it is invisible until it is expensive.

The wire format is unremarkable once the states exist. Text deltas and attribution events share a stream, and attribution events reference sentence IDs that the text events have already established.

event: text
data: {"sentence_id":"s1","delta":"A policyholder may appeal a declined "}

event: text
data: {"sentence_id":"s1","delta":"subsidence claim within 60 days of the decision letter."}

event: sentence_end
data: {"sentence_id":"s1"}

event: text
data: {"sentence_id":"s2","delta":"The appeal must be submitted in writing "}

event: attribution
data: {"sentence_id":"s1","support":"entailed","entailment":0.94,"citations":["cit_4a81"]}

event: sentence_end
data: {"sentence_id":"s2"}

event: attribution
data: {"sentence_id":"s2","support":"partial","entailment":0.61,"citations":["cit_4a82","cit_4a83"]}

event: done
data: {"label":"partly_supported","unresolved":[]}

Two design consequences follow from that ordering. Any control that commits the answer to something — copy, insert into a letter, apply — should stay disabled until the done event carries an empty unresolved list, because a handler who copies at 3.8 seconds copies text whose last sentence has not been checked. A 300ms disable is imperceptible; if the checker's tail latency puts that closer to two seconds, the disable becomes a visible stall and the honest options are to make the check faster or to let the copy proceed with the unchecked sentence explicitly flagged in what gets copied. The second consequence is that marks must not reflow the text when they settle. Underlines, background tints and margin rails can be applied without changing layout; inline badges and appended labels cannot, and a paragraph that jumps 18 pixels as each sentence resolves reads as instability rather than diligence.

What marks are not#

A supported span is a claim about retrieval, not about the world. Every sentence in an answer can be perfectly entailed by a passage that was superseded by an endorsement nobody uploaded, which is exactly the corpus failure from chapter 3, and the answer will render entirely plain. Marks narrow the space where fabrication can hide and leave the space where the source itself is wrong untouched.

They are also a poor fit for surfaces where there is no source to attribute to. A tone-rewriting feature, a summarisation of the user's own text, or a brainstorming assistant has no corpus, and grafting support marks onto them produces either universal marking or universal absence, both meaningless. The uncertainty problem is real on those surfaces and takes a different form, which is usually about scope and reversibility rather than provenance.

That is the shift the next chapter makes. Everything so far has assumed the user reads the output and decides. When the output is written into a document, a record or an outbound action, the reading happens after the writing, and a mark on a sentence that has already replaced the paragraph it was supposed to improve is a report rather than a safeguard.