Building a golden dataset

Read these first

• Why traditional testing fails

What you will know

• Build an eval set from real production traffic rather than invented examples

  Redact sensitive data and preserve the context needed to reproduce failures; otherwise the set is either unsafe or too sanitized to be useful.

• Label examples by hand and say how long it took

  Record annotation time and ambiguity decisions so future refreshes can be budgeted and inconsistent judgments can be found.

• Justify the size of the set — and explain why ~100 good ones beat 10,000 generated ones

  The useful quantity is coverage of distinct failure modes and decision boundaries, not raw row count or apparent statistical precision.

• Cover the failure modes, not just the happy path

  Stratify sampling around rare intents, ambiguous inputs, long contexts, known incidents, and adversarial cases so aggregate scores cannot hide dangerous gaps.

• Define a refresh trigger and cadence, then run a refresh

  New product behavior, traffic mix, policies, and incidents should force additions before the benchmark becomes a record of an older system.

Chapters

1. 1

   Model-authored eval sets

   Five thousand generated support questions with generated reference answers measure behaviour on clean, single-intent, well-punctuated input, graded against a model's own idea of a good answer. Real traffic brings typos, missing order numbers, mixed languages and three questions in one message. This chapter takes both halves of a generated row apart, and names the jobs synthetic cases genuinely do better than sampled ones.

   Read chapter
2. 2

   Sizing a golden dataset

   Extra rows narrow the interval only when they are independent draws from real traffic; near-duplicates buy cost and a false sense of power. This chapter works the design-effect arithmetic on a clustered generated set, reframes size as coverage of distinct failure modes, shows why paired comparison does the job more rows were meant to do, and names the two cases where the answer genuinely is a bigger set.

   Read chapter
3. 3

   Stratified sampling for an eval set

   A uniform sample of 200 conversations scoring 0.91 can contain zero instances of a data-exposure bug that hits 0.3% of traffic. This chapter defines slices by intent, customer segment, input shape and known incident pattern, sizes each so it can say something on its own, deliberately oversamples the severe ones, and keeps population weights separate so the single aggregate number is reweighted rather than averaged.

   Read chapter
4. 4

   Frozen context and ground truth

   A row in the set is everything needed to re-run the conversation and get the same verdict: the input as the customer sent it, the retrieval context and tool fixtures as they stood at the time, and the ground truth beside them. This chapter separates the three kinds of ground truth — pass/fail label, reference answer, rubric — because the labelling budget cannot be estimated until that choice is made, and explains what the labelling metadata is for.

   Read chapter
5. 5

   Pseudonymising an eval case

   Replacing every name, email and order id with [REDACTED] satisfies compliance and makes half the cases unanswerable. This chapter covers substituting realistic, format-valid surrogates consistently across the conversation, the retrieved documents and the tool fixtures, why keyed hashing fails on exactly the cases worth keeping, what the design does and does not buy you legally, and the residue even a good pass leaves behind.

   Read chapter
6. 6

   Adjudicating labeller disagreements

   Agreement on 130 of 150 is not the interesting number; the 20 are. This chapter covers running a blind labelling pass, measuring agreement in a form that removes the base rate, adjudicating the disagreements into written rules rather than verdicts, re-labelling the earlier cases each new rule changes, and versioning the set so a sharper definition of a pass is not read as a regression.

   Read chapter
7. 7

   Refreshing a golden dataset

   A record 0.93 on a set with no subscription cases describes a product that shipped three months ago. This chapter sets refresh triggers on launches, traffic shifts, policy changes and incidents, shows why a wholesale rebuild destroys every historical comparison, and gives the additive refresh that preserves it: new strata alongside the old cases, a versioned set, a re-baselined previous release, retired cases, and a named owner.

   Read chapter