Re-scoring sampled production runs is what convicts the eval

Re-scoring sampled production runs is the measurement that decides between two explanations an eval set cannot tell apart on its own. An eval set here is a fixed collection of tasks with expected outcomes that the agent is run against on demand; a success definition is the set of assertions that read the world after a run and decide whether it passed; and component scores split a run across the stages that produced it — which tool was selected, whether retrieval returned the record the task needed, whether the final message matched what actually happened. Two explanations for a dashboard that reports success in the nineties while users complain are still standing after that split, and the component table looks identical under both. If the success definition is too generous, selection, retrieval and synthesis all score in the nineties because each stage did its job and the verdict at the end was charitable. If the eval set is full of easy inputs, the same three stages score in the nineties because the decisions in front of them genuinely were easy.

The measurement that can see the difference points somewhere the eval set does not reach. Production is executing the same policy against inputs nobody curated, and the trace store from topic 10.8 — the span-structured record of every run the agent has made, kept for a fixed window — has been recording every one of those runs. Scoring a sample of them with exactly the assertions from chapter 1, where task success was defined as a list of checks over the end state that two raters could apply without conferring, produces a second number. The relationship between the two numbers, together with the shape of the inputs behind each, is what settles the argument.

Component scores read the same under both explanations#

Component scores are rates over decisions rather than over runs: of all the tool-selection decisions the agent made, the fraction that were right; of all the retrievals, the fraction that returned what the task needed; of all the final messages, the fraction consistent with the end state the run produced. The previous chapter, which splits an end-to-end drop across those three stages to find where it came from, built them and used them correctly. When a set fell after a prompt change and the selection number lost most of the drop while retrieval and synthesis held, the split named the stage, and it did so in a way an aggregate success rate could not.

Reaching for them again here is tempting for three good reasons. They are a real decomposition rather than a proxy, they localised a regression correctly once already, and re-running them costs a scorer pass over runs you have. None of that helps, because of what the numbers do under the two explanations on the table. A generous definition leaves them in the nineties, an unrepresentative input distribution leaves them in the nineties, and running the component analysis again produces the same reassuring numbers and no new information. That is the signature of an instrument that cannot see the difference you are asking about: the quantity stays pinned while the property you are trying to distinguish varies from one extreme to the other.

That is the general form worth carrying out of this chapter. A measurement is evidence about a question only if the two possible answers would have produced different readings of it. When they would not, it carries no information about which answer is true, however accurate it is and however often it is repeated, since repetition only tightens an estimate of a quantity nobody was disputing. The test before running a diagnostic again is whether it could have come out differently, rather than whether it is trustworthy.

Making production runs scorable at all#

The instrument that could come out differently draws from a population the eval set has no access to, and it produces two comparisons rather than one.

The assertions in chapter 1 read the end state — the calendar, the mailbox — so they can run against production provided three things hold, and each of them is a design decision made earlier rather than something you can arrange on the day.

The end state has to be readable after the fact, which means the agent's writes carry a run_id that the scorer can query by, exactly as world.calendar.events_created_by(run_id=...) assumes. The trace retention window has to be long enough that the run and its end state are both still available; fourteen days is comfortable, and a seven-day window makes it awkward to investigate anything reported on a Friday. And the redaction policy set in topic 10.8, where what a trace is allowed to retain is fixed before the first payload reaches it, constrains which assertions can execute: if message bodies are stripped from traces, an assertion that checks whether the reply mentions the agreed time cannot run on production data, while assertions over structural metadata — event count, attendee set, start time, invite status — survive redaction intact. That asymmetry is a good reason to have written the success definition against structure in the first place.

Calling those three design decisions understates them, and the sharper statement changes when they have to be argued for. Each is a decision whose absence makes this chapter's instrument impossible to build later rather than merely expensive. A write that went out without a run identifier cannot be retrofitted with one, since the association between the write and the run existed only at the moment of writing. A trace that has aged out of a seven-day window is gone rather than slow to retrieve. A body redacted on ingest was never stored, so no later change of policy recovers it. Underneath all three sits the distinction between what is forced and what was chosen. Given assertions that read the end state, being able to find that end state afterwards is forced, while carrying a run_id on every write is one particular way of satisfying it that somebody picked. Nothing forced the success definition to be written against structure either — a definition can be written against prose, and many are — but a structural one is portable onto redacted production traces and a prose one is not.

The harder gap is expected values. The eval set carries task.expected because a person wrote it; a production thread carries no such thing. Three sources cover most of it. Some expectations are derivable from the input, since the requested duration and the attendee list are usually stated in the thread. Some come from cheap human labelling of the sample, which for 150 runs is a couple of hours. And some come from what the user did next, which is the most useful signal in the set: a calendar event that a human edited or deleted within thirty minutes of the agent creating it is a strong indicator that the agent got it wrong, and it costs nothing to compute.

Scoring production runs also touches real customer data, and the access question should be settled before the first query rather than discovered during a review. The scorer reads calendar events and mailbox metadata for accounts the agent already acts on, which is usually within the scope the product was granted; a human labelling 150 threads to establish expected values is a wider ask, and it is the part that needs an explicit basis, an access log and a small population of reviewers. Where that is not available, the derived signals carry more of the load — repair actions, user follow-up messages, cancellations — and the sample size goes up to compensate for the weaker labels.

def sample_production_runs(store, days=14, n=150, strata=("segment", "attendee_bucket")):
    pop = store.runs(task_type="schedule_meeting", since=days_ago(days))
    return stratified_sample(pop, by=strata, n=n, seed=20250311)

def rescore(runs, world, labels) -> dict:
    rows = []
    for run in runs:
        expected = labels.expected_for(run) or derive_expected_from_thread(run)
        verdict = score_schedule_meeting(Task(run_id=run.id, expected=expected), world)
        repaired = world.calendar.human_edited_within(run_id=run.id, minutes=30)
        rows.append({"run_id": run.id, "score": verdict.score, "repaired": repaired,
                     "messages": run.features["thread_messages"],
                     "attendees": run.features["attendee_count"],
                     "cross_tz": run.features["distinct_timezones"] > 1})
    return rows

The width of the interval around a sampled rate#

A sample of 150 gives a 95% confidence interval of roughly ±7.3 points on a proportion near 0.7, meaning the true rate across all production runs sits inside that band on nineteen samples out of twenty, which is precise enough to detect the kind of gap that generates a support queue and not precise enough to argue about two points. Going to 400 runs narrows it to about ±4.5 points, and that is the right size for a monitor you run weekly rather than a one-off investigation.

Both of those half-widths come out of one expression, and deriving it is what lets you pick a sample size the paragraph above never mentions. Write the measured pass rate as a proportion and the number of runs in the sample as a count. Each run is a single pass-or-fail draw, and the spread of one such draw — its variance — is the proportion multiplied by one minus the proportion, which is largest at a half and shrinks towards either end. Averaging independent draws divides that spread by the number of them, so the variance of the sampled rate is the proportion times one minus the proportion, all divided by the count. Taking the square root converts a variance back into a spread measured in the same units as the rate itself, giving one standard error. A 95% interval is that standard error multiplied by a fixed factor a little under two, which is the factor that puts nineteen samples in twenty inside the band.

Evaluate it on the chapter's own figures. Near 0.7, the proportion times one minus the proportion is seven tenths times three tenths, which is twenty-one hundredths. Divided by 150 that is fourteen ten-thousandths, whose square root is a shade under four hundredths — one standard error. Multiplying by the just-under-two factor gives a bit over seven hundredths of the rate, which is the ±7.3 points already stated, written as points rather than as a fraction. Redo the last two steps for 400 runs and only the division changes: twenty-one hundredths over that larger count is a little over five ten-thousandths, its square root is a shade under two and a third hundredths, and the same factor turns that into ±4.5 points.

The general consequence is the part to keep. The count sits under a square root and nowhere else, so the interval narrows with the square root of the sample size: narrowing it by some factor costs that factor squared in extra runs. The two stated half-widths satisfy this as a check — the sample size grows by rather less than a factor of three between them, the interval narrows by about a factor of one and a half, and one and a half squared is rather less than three. It also settles sizes nobody quoted. Halving the investigation's interval means quadrupling its sample, so a sample of six hundred buys an interval of about three and a half points, and the halving after that costs two thousand four hundred runs. That is exactly why one size is right for an investigation and another for a weekly monitor: an investigation needs only to see a gap that is tens of points wide, while a monitor is watching for movement of a few points and has to pay the square for it.

The two comparisons, and the four verdicts#

Re-scoring produces one number to compare against the eval's 92.1%. The features produce a second comparison, over the shape of the inputs. Together they place the situation in one of four cells.

The bottom-right and bottom-left cells are the two explanations that were tied, and the top row exists because a third possibility has to be ruled out rather than assumed away.

Ruling it out is available only because the table is two binary comparisons crossed with each other, which makes its four cells exhaustive by construction: the score either matches the eval's or falls below it, the input shapes either match or differ, and every real situation has to land in one of the four. That is what licenses treating the top row as a rule-out rather than an afterthought, since a possibility the grid contains has to be excluded by evidence rather than by not being thought of. The residual is that both axes are thresholded judgements over continuous quantities. "Matches" and "differs" are verdicts somebody has to set a cut-off for, so a case sitting near either boundary is assigned by where the cut-off was put rather than by the data, and exhaustiveness says nothing about whether a borderline case landed in the right cell.

On the scheduling agent, the sample scored 68.2% against the eval's 92.1%, a gap of 23.9 points that is far outside the interval. The feature comparison was equally stark.

Nothing in that table is read as a level. A median of eleven messages is not high or low in itself, and the finding is carried entirely by the distance between the two columns on each row: threads roughly twice as long, attendee counts more than twice as large, cross-timezone runs going from a rounding error to nearly half the traffic. The sharpest row is the reschedule row, where the eval column is zero. A gap between two non-zero figures says the set under-weights something; a zero says the set contains no instance of a category that is a fifth of production, so no amount of running the eval more often can produce information about it.

The last row is a dash rather than a number, and the dash is a fact rather than a missing measurement. Whether a human repaired the event within thirty minutes is derived from what the user did next, and a fixture has no next: an eval task ends when the assertions run, with nobody on the other side to accept or undo the booking. That is a general point about where expected values come from. Production supplies a class of signal that a curated set cannot contain at any price, because the signal is somebody's later behaviour rather than a property of the input, and it is precisely the class the eval most needs help from.

One more relation makes the comparison usable, and it is arithmetic rather than a second opinion. Split any sample into groups and the overall rate is the mean of the group rates weighted by group size — each group contributes its own rate multiplied by its share of the sample, and the shares sum to one by construction. Nothing is assumed there beyond the definition of an average. So a rate computed on part of the sample is not a competing measurement of the same thing; it is one term of the number you already have, and the other term is fixed once you know the first. That is what turns conditioning on a subset into an argument: the subsets account for the aggregate rather than disputing it.

Rebuilding the eval set to match the traffic it stands for#

Knowing the set is missing 22% reschedules and 41% cross-timezone threads makes the repair specific. Turning that into a set which will not do this again is mechanical from there. Sample production runs stratified to match the production distribution, label expected values for them, and add them until the eval set's feature histograms are within a few points of production on each axis that matters. That took the scheduling set from 300 to 480 tasks, and the reported success rate fell from 92.1% to 74.3% on the first run of the new set — a drop that corresponds to no change whatsoever in the agent, and which is the number the team should have been looking at all along.

Matching the production distribution exactly is not always what you want, and the exception is worth stating. An eval set weighted strictly by traffic will contain 78% easy tasks, which wastes runtime and buries the signal from the cases that fail — a two-point movement in the reschedule stratum is invisible inside a set where reschedules are 22% of rows. The weighted-mean relation above is what makes that concrete. Hold every other stratum fixed and move the reschedule stratum's rate by two points: the aggregate moves by 22% of two points, which is under half a point, while a re-run of a set of a few hundred tasks carries an interval of about four points either side by the square-root law. The movement is buried an order of magnitude inside the noise, and the burial is arithmetic rather than bad luck — the weight that makes a traffic-matched set faithful is the same weight that shrinks anything happening inside a minority stratum.

That is a genuine tension rather than a mistake with a correct side. Matching production exactly maximises the fidelity of one number, the estimate of what users experience, and minimises the resolution of another, the ability to see a stratum move. Over-sampling the hard strata and recording the weights separately gives both readings: an unweighted score for detecting change in the difficult cases, where a two-point movement inside the stratum stays a two-point movement, and a traffic-weighted score for estimating what users experience. The weights live with the set, and the two numbers are reported side by side rather than one being derived silently from the other.

Two practices keep it honest afterwards. The first is versioning: a set whose composition changes cannot be compared across releases, so each version gets an identifier recorded alongside every reported score, and a re-baselining run on the previous release accompanies any change to composition. The second is a rolling sample, since input distributions move — a new customer segment, a new integration, a seasonal shift in how people schedule — and a set that matched production in March will drift by September. Re-running the comparison monthly on 150 sampled runs takes about an hour of compute and labelling time and catches the drift while it is still small.

Where the eval still cannot follow#

Re-scoring production is not a substitute for the eval set. It cannot be run before a release, since it needs the release to have run; it cannot cover cases that have not yet occurred; and it cannot inject the faults from chapter 3, where recoverability is measured by breaking a tool on purpose and watching what the agent does, because you do not get to break a customer's calendar to see what happens. The two instruments answer different questions, with the eval acting as a gate and the production sample acting as an audit of the gate.

The audit also inherits a blind spot from its own labels. Deriving expectations from the thread works when the thread is explicit, and roughly one production thread in six is not — the participants had a conversation elsewhere, or the requester genuinely did not care which afternoon it was. Those runs land in a bucket where no assertion can be evaluated fairly, and the honest handling is to report them as unscoreable with a count rather than to guess, because a scorer that guesses on 15% of its sample can produce a gap of that size out of nothing. That failure mode is worth stating in the same terms as the component scores: a guessing scorer returns a number whose value depends on how the guesses were made rather than on how the agent behaved, so it stops being evidence about the agent at all.

What you have now is a gate that fails for stated reasons, a set whose composition is defensible against the traffic it stands in for, and a monthly check that tells you when either has stopped being true. Every measurement in this topic reports a number about a set of tasks, and no measurement can vouch for the set it was computed over, because that is a question about where the tasks came from rather than about how carefully they were scored. Answering it takes a population nobody curated, which is why re-scoring production runs is the last piece rather than an optional extra.

Wiring it into the release process is the ordinary part: the eval on every pull request touching the agent, the fault-injection subset nightly because it is slower, the pass^k run before a release — the same task attempted repeatedly, counted as passing only if every attempt succeeds — and the production re-scoring on a monthly cadence with its own owner. The first release where the gate blocks something everybody wanted to ship is the one that establishes whether the numbers are load-bearing, and that argument is much easier to win with a budget that was written down in February.