Mid · Core path · 7 chapters
Why traditional testing fails
No exact-match assertions, output distributions, flaky by design
Exact-match tests create false failures and false confidence; production quality needs repeatable measurements over representative samples.
Read these first
What you will know
Build a test for generated prose that checks a property or structure instead of exact wording
A valid answer can differ in phrasing while still satisfying requirements such as containing required fields, citations, or constraints.
Show that the same input produces different output, and say why that isn't a bug to fix
Run one prompt five times and diff the results. That variance is the thing your test strategy has to survive.
Replace an exact-string assertion with a batch of checks over properties, structure, or scores
A single evaluator result can be noisy or biased, so useful signals come from repeated cases and thresholds rather than one brittle verdict.
Reproduce a false-confidence result from a passing manual spot-check
A handful of favorable examples can hide systematic failures on rare inputs, adversarial wording, or longer contexts.
Measure a controlled regression in a generated-output test suite and report its uncertainty
A model may look worse because of sampling noise or a changed case mix; compare distributions over the same sample before treating the change as real.
Chapters
- 1
Golden files and the acceptable set
A golden file records one draw from a model, not the model's answer, so the assertion built on it measures how often the sampler reproduces one particular string. This chapter measures the set of acceptable outputs for a single ticket, shows why a synonym and a dropped order id produce the same uninformative red, and marks the line between the deterministic half of a model feature, where exact comparison still works, and the sampled half, where it does not.
- 2
Removable flakiness and sampling variance
Two intermittently failing tests can sit next to each other on a dashboard and need opposite treatment. This chapter separates flakiness with a removable root cause — a shared path, a clock, an ordering dependence — from variance that is a property of a sampled generator, prices what a three-attempt retry does to the claim an assertion makes, and gives the replay experiment that tells the two apart in ten minutes.
- 3
Property assertions on model output
What actually replaces assertEqual: write the requirements down where a machine can read them, parse the output into a typed object, and assert the things every acceptable answer must have — citations present, every cited id among the ids retrieved for that query, forbidden phrasing absent. Then aggregate across the whole set of cases and gate on the pass rate, because demanding that every case pass on every run fails a genuinely excellent system four builds in five.
- 4
Bounding a failure rate
Twelve hand-written questions that all pass are consistent with a true failure rate anywhere up to 22 percent, and one person's twelve questions also miss the inputs no colleague would think to type. This chapter works the arithmetic that separates two distinct defects in a demo-sized check — too few draws to bound the rate, and a sample drawn from the wrong distribution — and shows why adding the escalated cases from an incident fixes neither.
- 5
Run-to-run noise in a pass rate
A pass rate moves between two runs of an identical prompt over identical cases. This chapter estimates the standard error of a single 200-case run near 0.85, works out how often a two-point release threshold fires against a system that did not change, and gives the drill to run before reporting a drop: repeat both prompts over the same cases several times and look at whether the two spreads overlap.
- 6
Paired comparison and confidence intervals
Knowing that two points on 200 cases sits inside the noise does not yet say what to ship. This chapter turns a measured spread into a decision: cross-tabulate the two prompts over the same pinned cases so only the discordant ones carry information, put a bootstrap interval on the paired difference, price three gate designs per release, and pin the things that move a rate without any change in quality — case-set edits, harness errors scored as failures, a floating model alias.
- 7
Reference bias in a model comparison
Forty replies captured from the model currently in production guarantee that model a perfect score before anyone runs anything, so a candidate scoring 11/40 against them is mostly being measured on phrasing distance. This chapter re-runs the incumbent against its own recordings, re-grades both models on requirements neither one authored, reports the difference with the interval it deserves, and names the three other comparisons that carry the same asymmetry.
External resources
- Challenges in Testing Large Language Model Based Software: A Faceted Taxonomyarxiv.org
Names precisely which testing assumptions break once outputs are distributions.
- There is no such thing as a flaky testjonkeane.com
This substantive engineering explainer shows why unreliable tests undermine confidence and details concrete causes such as excessive integration boundaries, poor code structure, and randomness.
- A pragmatic guide to LLM evals for devsPragmatic Engineer
Backend-engineer framing of what replaces assertEquals in an LLM codebase.
- Why Your LLM App Tests Are Lying to Youanup.io
Why a single pass or fail run is noise, and what to measure instead.
- LLM evaluation: a beginner's guideevidentlyai.com
Shows why LLM testing needs distribution-based evaluation rather than exact-match assertions for flaky outputs.
- Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya ShankarLenny's Podcast
Sets up why shipping LLM features without evals is guessing.
- Welcome to the LLM evaluation courseEvidently AI
Shows why LLM testing requires distributional metrics, not exact-match assertions, for inherently variable outputs.