What the synthesiser discarded is a policy nobody recorded

Scout is an internal competitive-research assistant, built on the orchestrator–worker arrangement: a parent decides what pieces of research a goal implies, workers run those pieces in isolation and return claims, and a synthesis step turns the claims into the prose a person reads. The previous chapters gave that arrangement a validated plan, represented coverage, explicit merge rules and grounded claims. Every one of those components leaves an artefact behind — a plan, an outcome list, a resolution with a dropped field, a span check with a verdict.

Synthesis leaves prose. Selection, throughout this chapter, is the choice that call makes about which of its input claims to use and which to leave out, and prose does not report it. That choice is a policy: it runs on every request, it changes whenever the prompt or the model changes, and in Scout it had never been written down or measured. The claim this chapter follows is c_31, the CRM note from chapter 5 recording that a competitor's price rise applied to new business only. It survived every mechanism built so far and never reached a reader.

A step with inputs and outputs and no record of what happened between#

Compare synthesis with any other component upstream of it and the asymmetry is structural rather than a matter of logging discipline. The decomposition emits a plan you can validate against the goal's sub-questions. Each worker emits an outcome with a state and a count of sources reached. The merge stage emits a resolution carrying a list of the claims it dropped and the reason. Those resolutions are readable after the fact. In each case the component's decision is expressed in a different medium from its input, so the decision can be compared against the input.

Synthesis breaks that pattern because its output is in the same medium as a paraphrase of its input. A sentence in the answer may restate a claim, combine two, gesture at a third, or come from none of them, and the text carries no marker saying which. Reading the answer tells you what it says. It does not tell you what it declined to say, and what it declined to say is the quantity of interest.

There is an analogy worth using here, and it is an analogy rather than a description of the same machinery. A database query planner chooses how to execute a query — which index to use, which join order to take — and a system that logs only the query and the rows returned lets you see that something was slow while leaving the choice invisible. EXPLAIN changes that by making the planner report the plan. It does not make a single query faster, and it converts a class of problem from folklore into something you can look at. Selection inside a synthesis call is in the pre-EXPLAIN state: inputs known, output known, and the choice between them unrecorded.

The analogy breaks in one place, and the break is the reason this chapter needs more than one mechanism. EXPLAIN reports a plan that the optimiser genuinely executed, produced by a different part of the system from the one being described. A citation marker in a generated answer is produced by the same model call that wrote the sentence, so it is a claim about the call's own behaviour rather than an observation of it. That difference is why an audit pass exists later in this chapter and why it finds sentences with no marker at all.

The first number that came out of instrumenting it was itself informative. Across 200 Scout runs, a median of 9.7 claims reached the synthesis step and a median of 4.1 of them appeared in the answer in any recognisable form. Work the proportion from those two figures: the claims that did not appear are 9.7 minus 4.1, and dividing that difference by the 9.7 that arrived gives roughly 58% of the material the workers produced — material that cost the majority of the run's tokens to obtain — discarded by an unwritten rule. That is not a defect on its own, since a good answer is shorter than its evidence and discarding is most of what synthesis is for, and it does mean the largest single act of judgement in the pipeline was happening unobserved.

Three ways to record what was used, and what each one is worth#

Asking the model to report its own selection is the cheapest option and it is a report rather than a record. Scout tried it first: the synthesis call was changed to return a JSON object with the prose and a used_claim_ids array. Checked against forty answers where a human traced each sentence back to its supporting claim by hand, the declared list agreed with actual usage on 78% of claims, over-declaring on 14% (naming claims whose content is nowhere in the text) and under-declaring on 8% (using a claim's substance without listing it).

Those three figures account for every claim, which is what makes the error rate readable: agreement plus over-declaration plus under-declaration is the whole population, so a 78% agreement is a 22% error rate. A self-report with a 22% error rate is usable as a signal, and the size of that rate is not the disqualifying part. What disqualifies it as the basis of a metric is that the errors are not random. Over-declaration was concentrated in exactly the claims the model had skimmed past — which is to say, in the population the metric exists to measure. An instrument whose error is largest precisely where the signal is cannot be corrected by knowing its average error.

Post-hoc attribution runs in the other direction: take the finished answer, split it into sentences, and for each sentence find the claims that support it, by span overlap first and an entailment call for the rest. This is faithful to what the text actually says and costs about 1,100 tokens per answer, and it inherits the judge's own error rate on the sentences that need a model.

The third option changes the generation contract so that the mapping is structural. The synthesiser is required to emit citation markers inline, each one naming a claim id, and every marker is checked against the claim set before the answer is accepted.

Read the first and third rows against each other rather than down the table. Both ask the model to declare what it used, and both cost almost nothing, and one is exact by construction while the other agrees with hand tracing four times in five. The difference is where the declaration sits. A marker written inside the sentence it belongs to is produced while that sentence is being produced, so there is no separate act of recollection to get wrong. A list assembled at the end is a memory of writing, and a memory of skimming past something is exactly the memory a model does not have.

Scout took the third and kept the second as an audit run on a weekly sample, because markers can be omitted from a sentence that nonetheless uses a claim, and the attribution pass is what measures how often that happens. It ran at 6% of sentences on the weekly audit, which is small enough to leave the metric usable and large enough that reporting selection recall to two decimal places would be false precision.

class Synthesis(BaseModel):
    text: str                       # sentences carry [c_31] style markers
    dropped: list[DropRecord]       # every claim not cited must appear here

class DropRecord(BaseModel):
    claim_id: str
    reason: Literal["redundant", "off_question", "weak_evidence",
                    "space", "unresolved_conflict"]

CITATION = re.compile(r"\[(c_\d+)\]")

def record_selection(result: Synthesis, claims: list[Claim]) -> SelectionTrace:
    available = {c.claim_id for c in claims}
    cited = set(CITATION.findall(result.text))
    dropped = {d.claim_id for d in result.dropped}

    if unknown := cited - available:
        raise InvalidCitation(unknown)          # a marker for a claim that does not exist
    if unaccounted := available - cited - dropped:
        raise UnaccountedClaims(unaccounted)    # silently vanished; re-prompt

    return SelectionTrace(
        run_id=result.run_id,
        available=sorted(available),
        cited=sorted(cited),
        dropped={d.claim_id: d.reason for d in result.dropped},
    )

Both raise lines are set operations and both are conservation laws, which is the same move chapter 2 made on sub-questions applied one stage later. The first checks that nothing was cited that did not exist: cited - available is empty when every marker names a real claim. The second checks that nothing existed without being accounted for: available - cited - dropped removes from the input set everything the answer used and everything the answer explicitly discarded, and whatever is left over is a claim that entered the call and left no trace. The UnaccountedClaims check is what turns the record from documentation into a constraint, because it makes that remainder an error rather than an observation. A claim cannot leave the pipeline without something in the trace saying what became of it.

The reason enum is deliberately small and deliberately not free text, because a free-text reason is another unreviewable prose artefact and a five-value enum can be counted. Scout's distribution across 200 runs was 34% redundant, 22% space, 19% off_question, 14% unresolved_conflict and 11% weak_evidence, and the fourth of those is the row that started the useful argument, since a conflict the merge stage decided to surface was being quietly buried one step later.

Recall against the evidence reviewers rate highest#

A record of what was dropped does not say whether the dropping was right. That requires a reference for what a good selection would have kept, and the cheapest reference is the reviewers who were already reading the outputs.

The labelling protocol matters more than the labels. Reviewers were shown the goal and the claim set, without the answer, and asked to mark each claim must_include, useful or irrelevant. Withholding the answer removes the anchoring effect that makes a reader who has seen a fluent paragraph decide, retrospectively, that what it contains was what mattered. Sixty runs at about ten claims each is 600 labels and took the three reviewers a little under four hours between them, with each run labelled by two of the three; agreement on the must_include boundary came out at Cohen's kappa 0.71, kappa being an agreement score that subtracts the agreement two labellers would have reached by guessing, so that 0 is chance and 1 is perfect, which is high enough to build a metric on and low enough that a run-level difference of one claim means nothing.

Selection recall is the fraction of must_include claims that were cited; selection precision is the fraction of cited claims that reviewers did not call irrelevant. Both are ratios, and both take their denominators from a different place, which is the thing to hold on to before reading the matrix below.

Work both numbers out of the cells. Recall asks about the top row only. Of the 181 claims reviewers marked must_include, 112 were cited, and 112 divided by 181 is 0.62. Precision asks about the left column only, and its denominator is every claim the answer cited: 112 from the top row, 148 from the middle and 74 from the bottom. Of that column, the ones reviewers did not call irrelevant are the 112 and the 148, so precision is those two counts over all three, which comes to 0.78. The two ratios share not one operand, which is why they can move in opposite directions and why neither is meaningful alone.

Reading the matrix as gaps rather than as levels gives the diagnosis. Sixty-nine of the strongest claims never reached a reader, and 74 claims the domain experts considered irrelevant did. Those two cells sit diagonally opposite and are nearly the same size, which says the selection step was not being too strict or too loose. It was making its choices on something other than the axis the reviewers were labelling.

Reading the drop reasons attached to those 69 gave a breakdown nobody would have guessed: 41% carried the reason unresolved_conflict, meaning the synthesiser dropped precisely the claims chapter 5's merge stage had flagged as needing a caveat; 27% were long claims with scope qualifiers, which is the same population by another description, since a claim with a carve-out is harder to fit into a clean sentence; and 19% had reached the final call through a batch summary rather than directly, having already been compressed once by the staged aggregation of chapter 5. That last group connects the aggregation structure to selection loss, and it is not a connection anyone would have made from prose.

With the metric in place, changes became legible. The word budget was the first, since a 600-word target over eleven claims is itself a selection policy, imposed by a number in a config file that nobody had thought of as a quality decision.

The precision column earns its place in the second row, and that row is the only one in the table where the two metrics move apart. Raising the word budget bought recall by letting more claims in, including the irrelevant ones, which is the shape of every metric that can be gamed by including everything. Tracking recall alone would have declared that change a win and shipped a longer, mushier answer. The other two rows move both columns together, which is what a real improvement looks like and why they were kept.

Why the aggregate score sat still through all of it#

Three changes moved selection recall by 26 points and the headline quality score by 0.3, which is the number that explains six weeks of not knowing. The reviewer rubric scored coherence, sourcing, relevance and usefulness on a 10-point composite, and the standard deviation between graders on the same answer was 1.2 points.

The relation between those two numbers and the sample size you need is worth stating as a rule rather than as a result, because it applies to every evaluation in this topic. To resolve a difference between two arms you have to separate the signal from the noise around each measurement, and the noise around a mean shrinks with the square root of the sample. Squaring both sides of that statement turns it into something you can compute with: the sample needed grows with the square of the spread between graders, shrinks with the square of the effect you are trying to see, and carries a constant of order ten in front for conventional confidence and power. Put the two figures in. The spread is 1.2 points and the effect is 0.3, so the spread is four times the effect, and squaring a ratio of four gives a factor in the tens; multiplying by the constant of order ten lands on the order of 250 graded runs per arm — per version being compared, the old prompt and the new one. The weekly sample was 20.

What that licenses is a specific conclusion and not a general one. It says the six weeks of a flat 7.1 were never evidence that nothing had changed, because an instrument that needs 250 runs to see an effect and is given 20 will report the same number whether the effect is there or not. What it leaves open is whether the changes were any good. The identity is about the resolving power of the measurement, and it says nothing at all about the direction of the thing being measured.

The rubric's composition compounds it. A fluent answer with a hole scores well on coherence and sourcing, since everything it does say is coherent and correctly sourced, and loses a fraction of one item for missing something a grader may not know was available. The grader is reading the answer, not the claim set, which is the same blindness the reviewers only escaped by opening a second window. This is why chapter 6's fabricated 34% was equally invisible: a wrong number appearing in one run out of four, costing perhaps a point and a half on the runs where a grader happened to verify it, contributes an expected fraction of a point to a mean with a 1.2-point spread. Dropping the best evidence and keeping an invented statistic are the same unmeasured step failing in opposite directions, and a scalar over the whole output cannot see either, because the failure is in the relationship between the input set and the output rather than in any property of the output alone.

That is the stumble this chapter exists to name. A team watching a stable aggregate score believes it is watching for regressions, and the belief is tempting because the number is real, is computed correctly, and does move when something breaks badly enough. What stayed constant was the 7.1. What varied underneath it was selection recall, across 26 points and two rewrites of the synthesis prompt. A quantity that holds still while the dangerous property moves is not a defence; it is a defence-shaped object, and the way to tell the difference is to ask what the number would have to do for the failure to become visible.

Selection recall is a component metric#

A selection trace with 0.88 recall says the strong claims made it into the text. It says nothing about whether the text explains them, orders them sensibly, or draws a defensible conclusion, and an answer can cite every must_include claim in a flat list and be worse than one that cites eight of ten and reasons about them. Recall is a component metric, useful for the same reason component metrics are always useful and dangerous in the same way, which is why the aggregate score stays even after it stops being the thing you steer by.

The reference set is a labelled opinion with an expiry date. It reflects what three people thought mattered about sixty goals at one point in a competitive cycle, and when Vantage changes strategy, the claims that deserve must_include change with it. Scout re-labels forty runs a quarter, which is a real recurring cost and the reason to keep the set small.

There is also a confusion worth heading off, because it has an appealing shape. Inline citation markers look exactly like a reader-facing sourcing feature, and shipping the markers to users is easy once they exist. Those are different projects. Sourcing is a product decision about reader trust; selection instrumentation is a measurement that needs a labelled reference set and a metric attached to it, and a team that ships the footnotes and skips the labels has a feature and still cannot tell whether a prompt change helped.

Finally, the trace only covers claims that reached synthesis. A must_include claim that no worker ever produced does not appear as a selection failure, because it does not appear at all — that is a decomposition failure from chapter 2 or a coverage failure from chapter 4, and the selection metric will happily report 1.00 recall over a claim set that was missing the point. The metrics stack rather than substituting for one another.

Everything this chapter added rests on one structural change: a claim now leaves the synthesis step by exactly one of two doors, cited or dropped-with-a-reason, and both doors are counted. That makes selection a component with a contract, which is what the decomposition, the workers and the merge stage already were. It also makes the difference between two versions of a prompt into a quantity, which is the precondition for the question that has been deferred since the first design review.

The system now has a validated decomposition, represented coverage, explicit merge rules, grounded claims and a measured selection step, at a cost in tokens and latency that has been growing with each of them. Whether the assembled thing is worth running instead of one agent with a large budget is an empirical question, and answering it honestly turns out to depend on how the runs that produce no answer at all are counted.