Withholding the reasoning is not withholding the evidence

The change that fixed the Analytics-seats incident took an afternoon and added 1,900 input tokens to each critic call. Meridian's writer already retrieved a bundle before drafting: the account's usage rollup, its plan record, and the current entitlement matrix pulled from the billing service. That bundle was being passed to the writer and thrown away. Passing the same bundle into the critic's message — as documents, with identifiers, not as narration — moved the pricing class from 3 caught out of 34 to 31 out of 34 on the same evaluation set. The three that still slipped were cases where the entitlement matrix itself was stale, which is a different bug in a different system.

Cost went up by roughly $0.006 per review at $3 per million input tokens, and p50 latency by 0.4 seconds, most of it prefill. Set that against the $2,160 in credits from a single batch and the arithmetic is not interesting. What is interesting is that the separation survived the change completely intact: the critic still has no idea what the writer planned, why it chose the recommendations it chose, or that any of the text was produced by a model at all.

Both of those figures are worth a moment, because each is a whole argument compressed into a number. The added cost is the token count multiplied by the price: 1,900 tokens is roughly two thousandths of a million, and two thousandths of three dollars is a little over half a penny. The recovered catches are the gap between the two fractions: twenty-eight of the thirty-four pricing defects moved from missed to caught, which is close to all of the thirty-one that any version of this design could reach, since the other three fail for a reason living upstream in the billing data. So the ceiling on this class under this design is thirty-one out of thirty-four while the entitlement matrix goes stale, and grounding took the critic essentially to that ceiling for the price of half a penny a review.

The line runs between derivation and inputs#

Chapter 1 built the split by removing things from the critic's context, and it is easy to carry away the rule "the critic sees less." The rule is narrower. What must be withheld is the writer's derivation — the plan, the intermediate reasoning, the assistant turns in which the draft's choices were argued for — because that material is what converts criticism into justification. The writer's inputs are in a different category entirely. They are the ground truth the draft is supposed to be faithful to, and a reviewer who cannot see them is reviewing a text for internal coherence and nothing else.

The distinction between the two categories is not a matter of taste, and there is a test that separates them. A piece of context is a derivation if it asserts that the draft is right; it is an input if the draft's correctness is measured against it. The plan asserts that the recommendations follow from the goal, so its presence in the critic's window supplies an argument the critic has no reason to reopen. The entitlement matrix asserts nothing about the draft at all — it states what the plan includes, and the draft either agrees with it or does not. Adding the first gives the critic something to be persuaded by. Adding the second gives it something to check against. Those are different operations and only one of them was ever the thing the split was defending against.

The analogy from chapter 1 holds all the way through here. A reviewer on a pull request does not want to read the author's private notes about why they chose this abstraction, because that is the thing they are meant to assess freshly. They very much do want the ticket, the API contract the change is written against, and the schema the migration touches. Withholding those does not make the review more independent; it makes it uninformed. A critic without the price list is a reviewer who has been shown a diff and denied the interface it implements.

The dashed edge is the only thing being blocked, and everything else that reached the writer reaches the critic as well. Drawn this way the earlier design looks like what it was: a reviewer cut off from the evidence in an attempt to cut it off from the argument.

Make every finding carry two quotations#

Grounding the critic — putting the documents its verdicts are supposed to be checked against into its context window, so that a claim can be compared with a source instead of with an expectation — raises a question the ungrounded version could not even pose. Once the sources are in the window, a finding can be checked, and the cheapest way to make it checkable is to require the critic to quote both sides — the span of the draft it objects to and the span of the source that contradicts it, tagged with the source's identifier.

FINDING_SCHEMA = {
    "type": "object",
    "properties": {
        "category":    {"type": "string"},
        "severity":    {"type": "string", "enum": ["high", "medium", "low"]},
        "draft_span":  {"type": "string", "description":
                        "Verbatim text from the report, 5-40 words."},
        "source_id":   {"type": "string", "description":
                        "Identifier of the document the span contradicts, "
                        "or 'internal' if the conflict is with the report "
                        "itself."},
        "source_span": {"type": "string", "description":
                        "Verbatim text from that document, 5-40 words."},
        "explanation": {"type": "string", "maxLength": 400},
    },
    "required": ["category", "severity", "draft_span",
                 "source_id", "source_span", "explanation"],
}

Read the schema for what it enforces and what it merely requests, because the difference decides how much of it you can rely on. The enum on severity is enforced: a value outside the three listed is a schema violation and fails before any of your code sees it. The maxLength on explanation is enforced the same way. The word range on the two span fields is not — it sits inside a description, which is text handed to the model as guidance and checked by nobody. So a forty-word limit on a quoted span is a request, and a critic that returns a three-hundred-word draft_span produces a valid object. That is worth knowing before writing code that assumes short spans.

The schema is worth something on its own, because a critic asked to supply a contradicting source span writes different findings from one asked to supply an opinion. It becomes worth considerably more once you check the quotations mechanically, which takes about fifteen lines.

def verified(findings, draft: str, sources: dict[str, str]) -> list[dict]:
    kept = []
    for f in findings:
        if f["draft_span"] not in draft:
            log.info("dropped: draft_span not in draft: %r", f["draft_span"])
            continue
        body = draft if f["source_id"] == "internal" else sources.get(f["source_id"])
        if body is None or f["source_span"] not in body:
            log.info("dropped: source_span not in %s", f["source_id"])
            continue
        kept.append(f)
    return kept

On the first grounded run over 120 reviews the critic emitted 214 findings and this function dropped 19 of them: 4 where the quoted draft text had been paraphrased rather than copied, and 15 where the quoted source span did not appear in the named document. Those 15 were the interesting ones. Every one was a finding whose objection sounded specific — an overage rate, a contract clause, a retention figure — and whose supporting evidence the critic had supplied from the same prior that produced the original error in chapter 2. Requiring a literal quotation turns that failure into a substring test, and a substring test costs microseconds and cannot be talked around.

The denominator on that rate is smaller than it first looks, and getting it right changes how alarming the number is. Roughly two fifths of Meridian's findings name internal as their source, which makes their source span a quotation from the draft rather than from a document — a span the critic is copying out of text it was given, which is a much easier operation than reproducing a line from a retrieved file. So the fifteen fabrications come out of the other three fifths, somewhere near a hundred and thirty findings, which puts the fabrication rate among the findings that can actually fabricate closer to one in eight than to the one in fourteen the raw ratio suggests.

There is a residual on the substring test that is worth stating before anybody relies on it too heavily. It proves the quoted span exists in the named document, and that is all it proves. It says nothing about whether that span contradicts the draft span, so a critic can satisfy the check by quoting a real sentence that is simply irrelevant to its objection, and the finding survives. What the check removes is invention; what it cannot remove is misreading. The reason it is still worth the fifteen lines is that invention was the failure actually happening, at a measurable rate, and misreading leaves a trail a reader can follow because both quotations are in front of them.

Exact-match dropping is slightly blunt on real text, since a source that has been reformatted between retrieval and prompting will fail the test on whitespace alone. Normalising both sides — collapsing runs of whitespace, unifying quote characters and dashes — before comparing removed all four of the paraphrase drops that turned out to be formatting artefacts and left the fifteen fabrications untouched. The function above is the version before that normalisation, so a reader lifting it will get the blunter behaviour and should add the normalisation on both sides of each comparison rather than on one.

That result is the reason the normalisation is safe to add. A change to a matching rule that recovers findings will usually recover some bad ones with the good, and the thing to check is which. Here the four recovered were formatting differences on spans the critic had copied correctly, and the fifteen stayed dropped because no amount of whitespace collapsing makes a sentence appear in a document that does not contain it. The two failure modes are separated by the normalisation rather than traded against each other, which is a property worth verifying on your own drop log rather than assuming.

What belongs in the critic's window#

The bundle is not "everything you have". Meridian's account records include a 40-page master services agreement, and putting all of it in the critic's context added 34,000 input tokens per call, raised p50 latency by 6 seconds, and lowered catch rates on two classes, because the entitlement matrix that actually decided most findings was now a small fraction of a large window. The working rule is that the critic gets what the writer's claims are supposed to rest on, retrieved with the same query, and nothing that merely might be relevant.

Price that experiment against the one that worked and the contrast is stark in every column. The agreement costs around a tenth of a dollar per call at the same input rate — roughly seventeen times what grounding on the retrieved bundle costs — for fifteen times the latency increase, and it made the catch rates worse rather than better. The mechanism is dilution rather than confusion: the model's attention over a long window spreads across everything in it, so a small decisive document sitting inside a large indifferent one competes for the same budget as forty pages of boilerplate. Adding context is not a monotone improvement, and this is the cheapest available demonstration of it.

The last row is a small thing that shows up in traces. Critic prompts that begin "review the following AI-generated report" produce noticeably softer verdicts on judgement classes than ones that say "review the following report", and the softening is consistent enough across runs to be worth the two-word edit.

Why this beats buying a second provider#

The teammate who proposes switching the critic to another vendor's model is responding to a real observation — the critic missed a class of defect and shares weights with the writer — with a fix that does not address the mechanism. Another provider's weights were trained on the public internet too. They contain a general prior over SaaS pricing and no row of your plan_entitlements table, which was last edited in a pricing meeting in October. Run the ungrounded critic on a competing frontier model and the pricing class lands in the same place the larger same-provider critic did: 4 or 5 out of 34, at higher cost, higher latency and a second vendor relationship to manage.

The two moves are not addressing the same defect, and it clarifies things to say what each one is for. Grounding supplies knowledge that no model has, and it is the only available fix for any claim whose truth lives in your systems. A different model decorrelates judgement, which matters for defects that depend on how the model reasons rather than on what it knows, and chapter 6 measures whether that decorrelation is real on a given task rather than assumed from the logo. Ordering them is straightforward: grounding costs an afternoon and $0.006 per review, so it is the first thing to try, and its result changes what the second question is even asking. A critic that has the price list in front of it and still waves through a wrong price is failing for reasons a different model might genuinely fix; a critic without the price list is failing for a reason no model can.

Hamel Husain's write-up Can I use the same model for both the main task and evaluation? reaches the same ordering from the evaluation side: the question of whether the judge should be a different model is usually asked before the cheaper questions about what the judge can see have been settled, and once they are settled the model question often stops being urgent. It is worth reading before a meeting where someone has already priced the second vendor.

Where a quotation requirement stops working#

There is a limit to how far a quotation requirement can be pushed. It works on claims that some document either states or contradicts, and it does not apply to judgement classes at all — a recommendation that overreaches the data has no contradicting span, because the problem is an inference rather than a conflict. Those findings quote the draft and give source_id: internal, and their evidence is the argument in the explanation field. Roughly 40% of Meridian's findings are in that shape, so the mechanical check covers most of the fabrication risk on factual claims and none of it on judgement calls, which is a division worth knowing when you read the drop log and see it filtering nothing for a week.

The same division explains a reporting hazard in the drop log itself. A quiet week can mean the critic stopped fabricating sources, and it can equally mean the week's traffic ran heavily to judgement-shaped findings that the check never examines. Those two situations produce the same empty log. Telling them apart needs the log to record how many findings were eligible for the source check as well as how many failed it, which is one more counter and turns a number that only falls into a ratio that can be read.

The three pricing defects that survived grounding mark the other boundary, and they are the more important one because they are the boundary a better critic cannot cross. The entitlement matrix was stale, so the critic compared a draft against a document and found agreement, and the agreement was real — the draft matched what the matrix said, and the matrix was wrong. Every mechanism in this chapter is a test of faithfulness to a source, and faithfulness to a wrong source is indistinguishable from faithfulness to a right one from inside the critic's window. Correcting that class is a data-freshness problem in the billing service, and the honest thing a critique loop can do about it is nothing.

Grounding also does nothing about a defect the sources happen not to cover, which is the same boundary approached from the other side. A claim about something no retrieved document mentions has no supporting span and no contradicting span, so the critic can only raise it as unsupported or leave it alone, and neither response establishes whether it is true. The rule that emerges from all three limits is the same rule: this machinery decides whether the draft agrees with what it was given, and every question about whether what it was given is any good lives outside it.

Everything built so far therefore produces findings that are located, checkable and increasingly numerous, and nothing yet says which of them deserve a model call. The average went from 1.8 to 3.1 by adding evidence the critic could legitimately cite, and there is no reason for that number to stop rising as more sources are added — a critic with more documents in front of it can defend more objections, including trivial ones.

That is the shape of the next problem. A finding that quotes a real span from a real document and names a real difference is a well-formed finding whether the difference matters or not, and the checks in this chapter cannot tell the two apart, because both pass the substring test identically. What is missing is a statement of which differences count as defects at all.