Retrieval- and extraction-style scoring

Read these first

• Deterministic and code-based checks

What you will know

• Choose an appropriate scoring method for an extraction task

  Normalize only what the contract permits; case, whitespace, ordering, and aliases can turn a correct answer into a false failure.

• Measure an open-ended generation with an overlap metric and record a concrete case where the score misleads

  A fluent answer may share little wording with the reference, while copied phrasing can still support a false claim.

• Score a structured output field by field rather than as a blob

  Field weights and explicit policies for missing or extra fields reveal partial correctness that record-level pass or fail conceals.

• Compare semantic similarity with a correctness check on paraphrases and factual errors

  Similarity models can accept wrong numbers, negated claims, or substituted entities because the surrounding language remains close in meaning.

• Reproduce a misleadingly high score and trace the resulting decision to the metric choice

  A plausible aggregate can stop investigation even when the evaluated behavior violates the task contract.

Chapters

1. 1

   Normalization and exact match

   Exact match counts every difference in shape as an error, so the standard repair is to normalize both sides before comparing — and the choice of which rewrites are allowed decides what the score means. Lowercasing and trimming a vendor name recover real passes because the consumer's lookup already does both; stripping punctuation and sorting tokens manufacture passes the system does not have, and the same normalizer applied to an amount field turns a hundredfold error into a green build.

   Read chapter
2. 2

   Type-aware equality

   A field with a type should be compared after coercion to that type, using the coercion its consumer performs. When a billing job parses effective_date and charges from it, the right comparator is that job's own parser applied to both sides: '2024-03-01', 'March 1, 2024' and '1 March 2024' pass, 'March 2, 2024' fails, and no list of accepted formats has to be maintained. The same move covers money, quantities, booleans and identifiers — and it inherits every bug the parser has.

   Read chapter
3. 3

   ROUGE and overlap metrics

   A close paraphrase of a reference summary scores 0.31 on ROUGE-L while three sentences lifted verbatim and stitched onto a fabricated root cause score 0.62, which settles what shared wording measures. ROUGE was validated at the system level with multiple references, and production suites invert both conditions. The metric keeps a place as a cheap deterministic drift alarm on every build — a demotion in authority and a promotion in usefulness.

   Read chapter
4. 4

   Semantic similarity scoring

   Embedding cosine places texts about the same thing near each other, which makes it structurally blind to the small edits that flip truth — a dropped negation, a transposed digit, a swapped policy number — while charging correct answers for being organised differently. A gate at 0.85 therefore reports 96% while the escalation queue fills. The chapter also lines up exact match 0.71, token F1 0.86 and cosine 0.97 on one date field and shows they are three different questions rather than a strictness ladder.

   Read chapter
5. 5

   Claim-level verification

   For open text with no parser to appeal to, correctness is not one comparison but many: break the answer into atomic claims and settle each against the source as supported, contradicted, or not found, with a verbatim evidence span attached. An invented citation and a conclusion stitched from three verbatim sentences both fail that check while scoring well on overlap and on similarity — and the check costs four orders of magnitude more per example than a cosine, which is what decides where it runs.

   Read chapter
6. 6

   Record-level and per-field scoring

   Record-level exact match at 41% and mean per-field accuracy at 94% describe the same extractor, and the reconciliation is the diagnosis: twelve independent fields at 0.94 multiply to 47.6%, so landing under that ceiling means the fields are uneven rather than uniformly mediocre. The per-field table tells those two worlds apart, an explicit policy for each way a field goes wrong keeps a missing key distinct from a null, and weights tied to what an error costs — not to how often the field appears — decide where the work goes.

   Read chapter
7. 7

   Scorer validation and canary tests

   A score is a claim a team acts on for months without re-examining, so the property that matters is testable: when the output changes in a way that matters, the number must move. Corrupt the predictions deliberately and watch — a comparator blind to a dropped thousands separator holds steady at 0.93 while a typed one falls by exactly the injected error rate — and publish the metric on shuffled pairs beside the real one, so a dashboard that is a constant with noise on top is legible on day one.

   Read chapter