The limits of LLM evaluation

LLM evaluation is the scoring of what a large language model (LLM) produces. Some of that scoring is done by people reading outputs under a guideline, and some of it is handed to an automated judge — a second model prompted with the same criterion a reviewer would apply and asked to return a score for each output. The economics of the substitution are overwhelming on paper: a judge costs fractions of a cent per item, runs on every build, and never gets tired at 4 p.m.

This chapter is about which parts of a human review are automatable and which are not, for reasons that are structural rather than a matter of judges not being good enough yet. The running case is a product that extracts clauses from commercial contracts, where every output is read by a paralegal and a proposal circulated to automate four tracks of review at once: schema conformance, register — meaning the level of formality and the conventions of the genre — whether every extracted clause carries a citation to a location in the source contract, and whether the legal characterisations in the summary are correct. Three of the four are reasonable candidates. The fourth is where the substitution removes the only thing standing between a confident wrong answer and a signed contract.

Where independence holds, and where it collapses#

The property that decides is whether the judge's errors are independent of the generator's. An evaluation is worth something when the checker fails in different places from the thing being checked; two independent checks with 20% miss rates each let 4% through, while two perfectly correlated checks with 20% miss rates each let 20% through, because the second one fails exactly where the first did.

That pair of figures is the entire argument of the chapter and takes two lines to derive. For a defect to escape both checks, both have to miss it. If the two checks are independent, the chance of that is one miss rate multiplied by the other, and multiplying is allowed precisely because independence is the assumption that knowing the first one missed tells you nothing about the second — a fifth of a fifth is a twenty-fifth, which is the 4%. If instead the two checks fail on exactly the same items, the multiplication is not available, because the second check missing is not a separate event: it is the first event, described again. Every defect the first check misses is a defect the second check misses too, so the escape rate is the miss rate itself, unchanged at 20%. Adding a second checker bought a fivefold improvement in one case and nothing whatever in the other, and nothing about the quality of either checker distinguishes the two. What distinguishes them is whether their errors land in the same places.

Form is where independence holds most easily. Whether a JSON payload has twelve keys does not depend on anything the generator believes about contract law, so a checker gets it right or wrong for reasons entirely unrelated to why the generator got it wrong. Truth is where independence collapses. If the generator asserts that a limitation-of-liability clause caps consequential damages, and a judge from the same model family reads that assertion, the judge's assessment is drawn from the same distribution that produced the claim. A statement that is wrong in a way the training data makes plausible reads as correct to both of them.

Follow that last sentence one step further down, because it is the mechanism and it is usually left as a slogan. Ask what the generator was doing when it produced the false claim. It was continuing text in the way its training data made most likely, which is to say it emitted the claim because the claim looked plausible against a particular distribution of legal writing. Now ask what the judge is doing when it assesses that claim. It is estimating how well the sentence fits the same kind of distribution, because that is the only faculty it has for a question with no string to match against. The two operations are running against the same underlying object. So the generator's mistake is not a mistake the judge might independently avoid; it is the judge's own criterion, arrived at from the other direction. The place where the generator is most likely to be confidently wrong — a claim that sounds exactly like the true ones and is not one — is precisely the place where the judge is most likely to approve it, which is the correlation stated as a mechanism rather than as a rate.

Two rows in that table carry a second finding, which is that the tracks safest to automate are also the ones that should never have been a judge. Schema conformance and citation presence are decidable by code, and code is faster, cheaper, deterministic and auditable. Handing them to a model buys nondeterminism at 300ms and $0.002 a call for a check that json.loads performs in microseconds.

REQUIRED = {"indemnity", "limitation_of_liability", "termination", "governing_law",
            "confidentiality", "assignment", "notices", "payment_terms",
            "warranty", "ip_ownership", "force_majeure", "dispute_resolution"}

def structural_checks(extraction: dict, contract_text: str) -> list[str]:
    failures = []
    missing = REQUIRED - extraction.keys()
    if missing:
        failures.append(f"missing_slots:{sorted(missing)}")
    for slot, clause in extraction.items():
        span = clause.get("verbatim_span", "")
        if span and span not in contract_text:
            failures.append(f"span_not_in_source:{slot}")
        if clause.get("char_start") is None:
            failures.append(f"no_citation:{slot}")
    return failures

Nine lines of that kind removed 61% of the items the paralegals had been rejecting, at zero marginal cost per contract and with a failure message specific enough to route straight back to the extraction service. The judge should be reserved for what is left, which is the material where a rule cannot be written.

What a paralegal's review is actually made of#

"Replace the paralegal review" treats the review as one task. Watching a paralegal work through a contract shows six distinct operations with very different automation profiles.

The fourth and sixth rows are judgement calls a well-validated judge might reach 80% of the way on, and the honest description is that it would be a useful pre-filter with a human on the residue. The fifth row is the one that cannot be handed over at any level of judge quality, because an absence has no output to evaluate. When the extractor never emits an indemnity slot for a contract that contains one, a judge reading the output sees eleven well-formed clauses and no evidence of a problem. Detecting the omission requires reading the source document independently of the extraction, which is the paralegal's job and is roughly the cost of doing the extraction again.

Absence failures are the general case of the hardest problem in automated evaluation. A judge scores what is in front of it, so any failure whose signature is missing content — a dropped clause, an unasked clarifying question, a caveat the answer should have carried, a refusal that should have happened — is invisible to it unless the judge is separately given the ground truth to compare against. If you have that ground truth, you did not need the judge.

The impossibility there is worth stating as an argument rather than as an observation, because it is the one limit in this chapter that no improvement in models can lift. A judge is a function whose input is the output being judged. Two contracts that produce the same extraction — one which contained an indemnity clause and one which did not — present the judge with identical input, since the missing clause is missing from the artefact the judge reads. A function cannot return two different answers for one input. So no judge, at any capability, distinguishes those two cases, and the failure of judges here is not a shortfall in skill but a consequence of what they were given. Widening the input is the only available move: hand the judge the source contract as well as the extraction, and the two cases stop being identical. What that change costs is the whole point of the fifth row of the table above. Reading the source contract carefully enough to notice a clause the extractor failed to mention is not a cheaper task than extracting it; it is the same task, done again.

Why a 5% audit reports clean#

The fallback in a proposal like this is the usual one: automate everything and have the paralegals audit a random 5% sample. Put the numbers through it.

The product processes 3,000 contracts a month. Historical data says an indemnity clause is present but missed by the extractor in about 0.8% of contracts, so 24 a month. Assume the judge catches half of those, which is generous given that a missed clause produces no output for it to inspect. Twelve escapes a month, spread through 3,000 contracts, is an escape rate of 0.4%. A 5% audit is 150 contracts, and the chance that a uniform random 150 contains at least one of the twelve is 1 − (1 − 0.004)^150 = 45%.

Take that expression apart, because the shape of it recurs every time somebody proposes a sampling rate. It is easier to compute the chance of finding nothing and subtract, which is what the leading 1 − is doing. A single contract drawn at random is an escape with probability 0.4% and is not one the rest of the time, so the chance one draw comes back clean is one minus 0.004. Draw a hundred and fifty and multiply that by itself a hundred and fifty times, which is what the exponent is — and multiplying is allowed here because each draw is being treated as independent of the others, an approximation that is very close at this sample size relative to the population. The result is a little over half, so the chance of finding at least one escape is a little under half.

The intuition to take away is the product of the two numbers rather than either of them. A hundred and fifty draws at four in a thousand comes to about six-tenths of an expected finding, and when the expected number of findings is below one, an audit that reports zero is the modal outcome rather than a result. That is the sense in which the audit reports clean by construction: it was never sized to see the thing it is being cited about.

More than half of all months, the random audit reports zero findings while twelve contracts go out with a missing indemnity clause. The audit is not measuring the failure rate, it is measuring nothing with a confidence interval wide enough to contain any conclusion, and its output is a green tick in a compliance report. Reaching 95% confidence of catching a single instance requires auditing a quarter of all traffic, which is five times the review budget the proposal was trying to eliminate.

The last row is the design that works, and it comes from abandoning uniform sampling. If the escapes concentrate — long contracts, particular counterparty templates, contracts where the extractor's own confidence was low, contracts where the judge and a cheap structural heuristic disagreed — then auditing that stratum puts the paralegal's hours where the failures are. On this product, contracts over 40 pages plus those using two known third-party templates were 8% of volume and historically contained 60% of the missed clauses. Auditing all 240 of them concentrates roughly seven expected escapes into the sample and finds at least one with better than 99% probability, at 40 paralegal hours instead of 125.

Compare the two designs on the quantity that decides them, which is the expected number of escapes landing in the sample. The uniform 150 catches whatever share of the twelve monthly escapes its share of the volume implies: a twentieth of the traffic gets about a twentieth of the escapes, which is the six-tenths worked out above. The targeted 240 is not sampling the stratum at all — it audits every contract in it — so it catches every escape the stratum contains, and the stratum contains three fifths of them, which is a little over seven. The comparison that matters is against the uniform plan which reaches the same confidence, which is the 750-contract row at 125 hours: the targeted design buys that confidence for 40. The lever is not the sample size. It is that sampling and enumeration are different operations, and inside a stratum small enough to enumerate, sampling gives away most of the coverage for a saving the stratum was too small to be worth.

The stratification is only as good as the historical data behind it, and it has an obvious failure mode: escapes in strata you did not identify are now audited at 0%. That is why the design keeps a small uniform slice — say 50 contracts a month — whose purpose is to detect new failure modes rather than to estimate a rate. It will rarely find anything, and the month it finds something in a stratum you were not watching is the month it pays for itself.

Where a human is not optional#

Some of this is a cost calculation and some of it is not negotiable regardless of how good the judge gets. Four categories keep a mandatory human in the loop.

Regulatory sign-off is the clearest, because the requirement is on the process rather than the accuracy. Where a named qualified person must attest to an output, the judge can prepare, summarise, pre-filter and flag, and someone still signs. Unrecoverable outcomes are the second: a wrong clause extraction that reaches a countersigned agreement cannot be rolled back with a hotfix, so the expected cost of a miss is not the miss rate times an incident, it is the miss rate times a contract dispute. Novel harm categories are the third, because a judge is validated against a reference set of failures that have already been seen, and its behaviour on a failure mode nobody has written a criterion for is undefined. Absence detection is the fourth, for the reason above.

None of this argues that humans are better evaluators in general. They are demonstrably worse at the form tracks — a paralegal checking 200 extractions for schema conformance at the end of a day will miss malformed payloads that a parser catches every time, and the miss rate rises with the length of the queue. The correct assignment of work puts deterministic checks in code, form and style judgements on a validated judge, and human hours on the failures where a model's error correlates with the generator's or where there is no output to inspect. That assignment saves more human time than blanket automation does, because it stops spending reviewer attention on things a regex already settled.

The question to ask of any track proposed for automation is therefore not how well the judge scores on it but what varies between the judge and the thing being judged. Where the answer is "nothing, it is a parse", the track belongs in code. Where it is "surface style", a validated judge is fine. Where it is "the same priors that produced the output", the judge is a second opinion from the same source — and where the failure is an absence, there is no output to hold an opinion about. Which leaves the 94% unexamined. It is the number the proposal rested on, and as reported it cannot support any conclusion at all. The measurement itself was carefully run. What undermines it is how the judge was built against those 500 items, and a second figure the 94% was never placed beside. Chapter 6, on validating a judge against held-out adjudicated labels, takes both apart.