Binary criteria

A binary criterion is a question about an answer with a yes or a no and nothing in between — does it name the specific setting, does every claim appear in the provided context, does it refuse when the context is silent. Replacing an overall score with a set of these changes what the harness can report: instead of one number per answer, a verdict per named requirement per answer, and instead of a mean, a pass rate per requirement plus a rate for answers clearing all of them.

The trade is not free and it is easy to discover late. What becomes visible is attribution, definition changes and resolvable disagreement. The one capability that goes away is the ability to say that one acceptable answer is better than another acceptable answer, and a reporting problem follows from that, because a deck that wants a number out of five will get one whether or not it means anything.

Five properties instead of five levels#

The rewrite takes an afternoon. A five-point rubric usually has its properties buried in the anchors, and pulling them out of one written for a support assistant gives four criteria plus one the team had been arguing about in review comments and never wrote down:

C1  answers_question    The answer addresses the user's actual question,
                        not an adjacent one.
C2  grounded            Every factual claim in the answer is supported by
                        the provided context.
C3  cites_source        The answer quotes or names the specific passage it
                        relied on.
C4  no_fabrication      The answer contains no entity, figure or date that
                        does not appear in the provided context.
C5  refuses_when_unsup  If the context does not contain the answer, the
                        response says so rather than guessing.

Each of these is a question with a yes or a no. There is no version of C4 where an answer half-fabricates a date. Run the same 600 stored outputs through five judge calls each and a dashboard tile that used to read 3.7 is replaced by five rows.

The 3.7 contained all of this and reported none of it. Two of the five criteria are doing nearly all of the failing, and they are the two a product team would have prioritised if anyone had shown them the split — citation is a compliance requirement for this assistant and refusing when unsupported is the difference between an assistant and a liability.

The last row is worth deriving rather than reading, because the way it relates to the five above it carries a finding nobody put there deliberately. Add the five failure counts and you get five hundred and forty failures recorded across the run. Only 288 answers fail anything at all. Five hundred and forty failures distributed over 288 answers is close to two failures per failing answer, so the answers that fail tend to fail more than one criterion, and the five criteria are catching heavily overlapping sets of answers rather than five separate populations.

There is a second way to see the same thing, and it is the one to reach for when the counts are not to hand. If the five criteria failed independently — if knowing that an answer missed its citation told you nothing about whether it was grounded — the share passing all five would be the five pass rates multiplied together: 94% of 88% of 61% of 96% of 71%. That product comes to a shade over a third. The harness reports 52%, which is half again as many answers clearing everything as independence would allow, and the only arrangement that produces the excess is failures piling onto the same answers. Hold that number, because the next chapter is about a mechanism that manufactures exactly this overlap whether the answers have it or not, and 52% against a third is the kind of gap that has to be attributed before it is believed.

What becomes visible that a mean hid#

Attribution is the first thing. A question like "did the October retrieval change help" is answerable when C3 goes from 61% to 88% across that release while C1 moves by a point and everything else holds: a named requirement moved, the release that moved it is identifiable, and the size of the movement is a count of examples rather than a position on a scale.

Threshold changes become visible in the same way. Suppose the team decides that C3 requires an exact quoted span rather than a passage title, which is a genuine tightening of the standard. The C3 pass rate drops from 88% to 64% on unchanged outputs, and that drop is legible as a definition change because the criterion has a definition to change.

Under the old scale the same tightening would have moved the mean from 3.7 to about 3.5, and the arithmetic showing why is the argument for the whole migration. Twenty-four points of C3 pass rate, taken over 600 answers, is a hundred and forty-four answers that used to satisfy the citation property and now do not. On a five-point rubric with citation buried in the anchors, each of those drops about one level, so the total of the 600 scores loses about a hundred and forty-four points, and dividing that by 600 costs a little under a quarter of a point on the mean. That movement is indistinguishable from a quality regression of the same size, because the mean has one slot and both causes write into it. The release notes would have recorded nothing at all.

Disagreement between judges changes character too. Under a scale, one judge saying 3 and another saying 4 was an argument with no resolution procedure. Under binary criteria, a disagreement is two verdicts on a named requirement with two rationales attached, and you can read them:

{
  "criterion": "cites_source",
  "judge_a": {"reasoning": "The answer references 'the returns policy' but
    does not quote or name the passage; the context contains section 4.2
    verbatim and the answer does not point at it.", "verdict": "fail"},
  "judge_b": {"reasoning": "The answer names the returns policy, which
    identifies the source region of the context.", "verdict": "pass"}
}

That is a specification bug, not noise. Somebody has to decide whether naming a policy counts as citing it, write the decision into C3, and re-run. Fifteen minutes of work produces a criterion that both judges then agree on, and the same fifteen minutes spent on a 3-versus-4 dispute produces nothing, because there is no sentence you can add to the rubric that resolves it in general. Per-criterion agreement is also a number you can track: Cohen's kappa — an agreement statistic that discounts the agreement two graders would reach by chance — came out at 0.91 on C4 and 0.44 on C5, which tells you exactly which of the five needs rewriting first.

The gap between those two figures has a cause you can read off the criteria themselves, and reading it is more useful than the numbers. C4 asks whether the answer contains an entity, figure or date absent from the provided context. Both judges are looking for the same thing in the same two texts, the thing is either there or it is not, and a kappa of 0.91 says they almost always find the same answer. C5 asks whether the response says so when the context does not contain the answer, and the words "does not contain the answer" hide a threshold nobody wrote down. A context that answers half the question, or answers it for the wrong product tier, is neither containing nor not containing, and each judge resolves that on its own. The format is binary; the question underneath is not. That is what a kappa of 0.44 measures — two graders applying two different implicit thresholds to the same partially covered cases — and it is why the number points at C5 rather than at either judge.

Ranking two answers that both pass#

Take two answers that pass all five criteria. One quotes the relevant passage and adds two sentences of useful context; the other quotes the same passage and stops. Both are 5/5. The instrument has no opinion about which is better, and no amount of care in writing the criteria will give it one, because the criteria were defined as thresholds and both answers are above every threshold.

The loss is structural rather than a matter of the criteria being too coarse, and seeing why saves a quarter spent writing better ones. A criterion sorts the suite's answers into two heaps, pass and fail, and says nothing about the order of the answers inside either heap. Add a second criterion and the two heaps become four, then eight, then thirty-two at five criteria — the partition gets finer, and it stays a partition. Two answers landing in the same cell are, as far as the instrument is concerned, the same answer, and refining the partition cannot separate them because refinement only ever adds new walls between cells. Ordering is a different kind of output from classification. Nothing built out of thresholds produces one.

This matters when the thing being compared is two versions of the system rather than two answers. Prompt A passes all five on 52% of the suite and prompt B on 53%, and one point of 600 examples is 6 examples — well inside what re-running either would produce on its own. A five-point scale would at least have given a mean that moved, and the reason not to trust that mean is the subject of the previous chapter, so the honest position is that neither instrument separates A from B and a third one is needed.

Pairwise comparison is that instrument. The judge sees both answers to the same input and is asked which better satisfies a stated preference, with the presentation order randomised across the suite to cancel position effects — the judge's tendency to favour whichever answer it was shown first. It produces a win rate rather than a pass rate, and 340 wins to 260 across 600 inputs is a separation that a pass rate could not find. Those two counts add to 600, so the win rate is a shade under fifty-seven per cent, which is seven points clear of the even split a pair of indistinguishable prompts would produce. On 600 comparisons the sampling wobble on a rate near a half is about two points, so seven points is several times the noise, against a one-point gap in the all-pass rate that was inside it. What pairwise will not tell you is why, or whether either answer is acceptable at all — two answers can both be ungrounded and one of them still wins.

The reason it can separate what a pass rate cannot is that it asks a different question of each example. A pass rate asks whether an answer clears a bar and throws away everything above the bar, so two prompts that mostly clear it look alike. A comparison asks about the two answers directly, so every example where one is better contributes a vote, including the examples where both were acceptable. The information the criteria discard is exactly the information the comparison collects.

Pairwise has costs of its own that the table understates. It needs a baseline to compare against, so every candidate is measured relative to whatever you froze as the incumbent, and a suite that has been ranked against a two-year-old baseline drifts into measuring the wrong contest. It also scales badly beyond two candidates, since ranking five prompt variants properly means ten pairings per input unless you accept a cheaper tournament structure and the variance that comes with it. Teams reach for pairwise when they want the fine-grained separation and then discover they have bought a comparison instrument with no absolute reading, which is why it belongs alongside a criterion suite rather than in place of one.

The three are not competing for one slot. A release gate wants binary criteria, because a gate needs an absolute standard and a reason it failed. A prompt bake-off wants pairwise, because the question is which of two candidates to ship and both may clear the gate. Running both is normal; the mistake is asking one to do the other's job.

Where binary stops helping#

Binary criteria do not encode severity. C4, a fabricated dosage in a medical answer, and C3, a missing citation on an otherwise correct answer, both contribute exactly one failure to the count, and the all-pass rate treats them identically. That flattening is a modelling choice you can correct at the reporting layer by weighting criteria, or by splitting a criterion into a hard one and a soft one, and it is not something the binary format solves for you.

Nor does making a criterion binary make it well defined. "Is the answer helpful — pass or fail" is as unstable as any five-point scale, because the judge still has to invent the threshold. The gain in the migration came from naming five checkable properties; the pass/fail format is what those properties fit into. A vague criterion with a binary output is a coin flip with a schema.

The all-pass rate has a volatility that follows from its shape, and it is worth knowing before it is put on a trend line rather than after. An answer clears the composite only by clearing every component, so the composite falls whenever any component falls, and a suite in which each of the five criteria slips by a point loses roughly five points of all-pass rate rather than one. The direction that surprises people is the other one: a large repair to a single criterion can move the composite by much less than the criterion moved, because the answers it rescues were failing something else as well. Both behaviours come from the same place, which is that a conjunction of five tests is more sensitive to broad shallow movement than to deep narrow movement. Reporting the five components underneath is what makes the composite readable, since the components are where the movement is attributable and the composite is where it is aggregated.

What the migration buys, then, is not accuracy and not agreement. It is that every number the harness produces has a sentence behind it that somebody can disagree with — a requirement that moved, a definition that tightened, a verdict with a rationale attached to a named property. The mean had none of that, and the ranking power the criteria give up is real and belongs to a different instrument that can be run alongside.

Five criteria also mean five verdicts per answer, and the obvious way to get five verdicts is to ask for all five in one call and read them out of a JSON object. That implementation is where the next failure lives, because the verdicts that come back from one call are not five independent measurements.