92% on the eval with users complaining points at the eval

The dashboard says 92.1% task success, and it has said something within a point of that for six weeks. Over the same period the support queue picked up thirty-four tickets about the scheduling agent, most of them variations on "it booked something odd and I had to fix it by hand". The model ID is pinned and unchanged, the prompt hash in the trace metadata matches the one from six weeks ago, and the tool set has had no additions. Two measurements of the same system disagree, and one of them is wrong.

Task success here means what the eval harness reports: each task in a fixed set runs once, the end state the agent leaves behind is checked against a list of assertions written for that task, and the fraction of tasks whose verdict came back a pass is the headline. Every clause in that sentence is a place a number can go wrong while the agent stays as it was. The first thing said in the meeting is almost always that the eval set is not representative and the team needs more examples. That is very likely true, and it is close to useless as a next step, because it names no test, offers no way to know when enough examples have been added, and puts the answer three weeks out. It also skips three explanations that can be checked against data already on disk, which between them account for most cases where an eval number and a user population disagree.

Triaging the tickets against the eval's own categories#

Before any of them, there is a cheaper instrument sitting unread: the thirty-four tickets. Complaints arrive as prose and get triaged as a queue, so nobody usually classifies them against the eval's own categories, and doing so takes about an hour and points at which explanation to check first.

The counts in that table add up to thirty-four, which is the whole queue. Every ticket went into exactly one row, and the last row exists so that a ticket fitting none of the patterns still has somewhere to go. The categories therefore partition the queue: exhaustive, non-overlapping. Without that identity, "eleven tickets" and "thirty-four tickets" are two unrelated counts and dividing one by the other means nothing. With it, every count is a share of one fixed population, and the rows can be ranked against each other because they compete for the same total.

Nineteen of thirty-four land on assertions the definition treats as secondary or does not check at all, which is a strong prior — a belief about where the fault lies, held before any evidence from the harness arrives — before a single number is recomputed. The triage is not evidence on its own, since users complain about what annoys them rather than about what is most common, and eleven tickets out of a population of forty thousand runs says nothing about rates. What it does is order the three checks below, and the order matters when each takes an afternoon.

That last limitation is the residual of the same identity rather than a caveat sitting beside it. The partition is exhaustive over complaints, and the denominator it supplies is therefore the number of people who chose to write in. Complaints are generated by what annoys somebody enough to open a ticket, which is a different process from what the agent does most often. The shares transfer perfectly within the queue and to no rate over runs at all: a row holding half the queue may correspond to a failure mode occurring in one run in a thousand. The table orders the investigation. It does not size the problem.

Recomputing the number you already have, three ways#

Each of the three checks reuses runs the harness has already executed. None requires new labelling, and all three finish inside an afternoon.

The first check is the fastest and the most often decisive. The definition from chapter 1 — a task's outcome written as a list of conditions on the end state the agent leaves behind — splits assertions into required and secondary and reports pass, partial and fail separately, and a task that satisfies every required assertion while missing a secondary one lands in partial. That is a third category rather than a shade of the first. The moment those three get folded into one headline number — usually because a dashboard wanted a single line — partial tends to get counted as success. The fold is a decision somebody made and could have made the other way; nothing in the definition forces it. What follows from it is arithmetic: every task in partial moves into the numerator, so the headline rises by exactly the share of tasks sitting in partial, without a single assertion having changed its verdict.

A related check costs five minutes and is worth running at the same time: count how often each assertion has ever failed. An assertion that has not failed once in six weeks is either trivially true or is not actually being evaluated, and both happen. On this set, invites_delivered had never failed. The tempting reading of a row like that is the flattering one — a check that never fires looks like a component that never breaks. It is wrong, and it is undetectable from the dashboard, because a check that never fires and a check that is not wired to anything produce identical evidence: a green mark, on every row, for as long as the record goes back.

Here it was the second. The fixture mail client — the stand-in the eval uses so that runs do not send mail to real people — returned a delivered status unconditionally, and the assertion had been passing on a constant since the day it was written. The quantity it read never varied, while the property it was written to guard, whether an invite reached anybody, varied freely underneath. That is the general shape of a defence that is not one: a quantity that stays fixed while the dangerous thing moves, so that watching it feels like surveillance and works like a blindfold. The headline is the same shape one level up — 92.1% held within a point for six weeks while what users got out of the agent changed enough to generate thirty-four tickets.

Assertions that pass on a constant are common enough to be worth a standing guard rather than an occasional audit. The cheap version is a mutation check: for each assertion, run the set once with a fixture deliberately configured to violate it, and confirm the assertion fails. An assertion that passes when the world has been rigged against it is not measuring the world. That generalises past mail fixtures and past this one set — the only way to learn whether a check is coupled to reality is to change reality in the direction the check claims to detect, and confirm the check notices.

The second check reuses the trajectory work directly — scoring the path a run took, span by span, rather than only the state it finished in. Outcome scoring held at 92.1% while the amber rate — runs breaching a soft budget threshold — was 19%, the unsafe-action rate was 6%, and the faulted subset had nine unnoticed_success classifications. Users experience all three: a run that takes fifty-six seconds instead of nine, a run that leaves a file nobody expected, a run that books on an assumption after an empty availability response. None of them subtract from a success rate computed on the end state alone.

The number was also measured once#

Before accepting any of the three explanations, it is worth asking whether 92.1% is a stable property of the system. Each task in the set ran once, at a nonzero temperature, and the reported figure is a single sample of a distribution.

The tau-bench work from Sierra made this concrete with a metric they call pass^k: run every task k times and count a task as passing only if all k attempts succeed. A user does not get an average of five attempts; they get one, and if they are a repeat user they get many independent ones, so consistency is closer to what they perceive than a one-shot rate. Re-running the scheduling set five times cost about $45 in total and reported 92.1% pass@1 against 74.8% pass^5. The seventeen-point spread means roughly one task in six succeeds sometimes and fails sometimes, and a user who hits one of those tasks weekly will encounter a failure about once a fortnight while the dashboard continues to read 92%.

Those two figures together say something about the shape of the set before any individual task is examined. Suppose every task were the same coin: an independent draw that comes up a pass with the reported one-shot probability, with no memory between attempts. The probability of five draws all coming up passes is then the probability of one draw multiplied by itself five times, which is that rate raised to the fifth power. Multiplying a number near nine tenths by itself five times lands well below the observed 74.8%, so the identical-coin model predicts a worse result than the harness measured.

A set can only beat that prediction by not being made of identical coins. Most of the tasks must be close to deterministic, carrying their full weight into pass^5 as well as pass@1, while a minority are genuinely unstable and contribute to pass@1 while dropping out of pass^5. The deterministic majority contributes equally to both, which leaves the whole spread coming out of the unstable minority — a place to look, derived from two aggregate numbers with no per-task inspection at all.

Dropping the temperature does not remove the spread, for the reasons the replay work in topic 10.8 — re-running a recorded trace against an environment that has moved underneath it — established: provider-side variation and, far more, divergence in what the tools return between runs. It also trades one problem for another, since a deterministic policy that fails a task fails it every time, and some of the 17% are tasks where the agent's willingness to try a different approach is what makes it succeed at all. The useful reading of pass^k is as a measure of how much of the reported success rate is repeatable, and the tasks with the largest spread between pass@1 and pass^5 are a short, high-value list to investigate — on this set, eleven tasks accounted for two-thirds of the gap, and nine of the eleven were multi-attendee threads where the availability search sometimes found a slot and sometimes gave up. Eleven tasks carrying two-thirds of the spread is the identical-coin argument confirmed by measurement.

Variance of that size also has a mundane consequence for gating. A release-over-release move of two points on a 300-task set run once is not distinguishable from noise, since the standard error — the movement you should expect from the luck of which tasks are in the set, with nothing about the agent having changed — on a proportion near 0.9 with n = 300 is about 1.7 points, and the run-to-run instability above adds to that.

That figure is worth deriving rather than accepting, because the same few lines settle every future version of the question. Write p for the proportion the set reports and n for the number of tasks in it. Each task contributes a pass or a fail, and the standard error of such a proportion is the square root of p times one minus p, divided by n. Substituting the set's own figures: p is near 0.9, one minus p is near one tenth, and their product is about nine hundredths. Dividing that by n, which is 300, gives about three ten-thousandths, and the square root of three ten-thousandths is a little over seventeen thousandths — the 1.7 points quoted above, read on a percentage scale. A release-over-release movement smaller than that is not distinguishable from which tasks happened to be drawn into the set.

The general consequence sits in the shape of the formula rather than in the answer. The number of tasks enters under a square root, so the standard error falls with the square root of the set size and not with the set size itself. Halving it therefore requires quadrupling the set: four times the tasks, four times the runtime, four times the bill. Better tooling does not change that exchange rate. Reporting pass^3 alongside pass@1, at three times the cost, buys a number that moves for reasons other than chance.

An end-to-end rate cannot say which stage moved#

Suppose the same set drops from 88% to 71% after a prompt change. Seventeen points have gone somewhere, and the aggregate rate says nothing about where, which is the argument from topic 3.14 — that a pipeline's end-to-end score cannot attribute a regression to any stage inside it — arriving in an agent-shaped setting.

The adaptation that matters here is that an agent's stages are recurrent, and what follows from it is forced by the loop rather than chosen by whoever wrote the scorer. The selection stage — the model choosing which tool to call and with what arguments — does not run once per task. It runs at every iteration, so the count of selection decisions inside one run equals the number of iterations that run took. A rate computed over runs would have to collapse all of those decisions into a single verdict, which gives a forty-step run and a six-step run equal weight while the first carries many times as much evidence about selection. So agent component scores are rates over decisions rather than over runs.

What remains a choice is which denominator to publish beside each rate. The function below reports decisions scored per component, which makes each row honest about how much evidence it rests on and deliberately non-comparable in count against the others. Per-run averages would have produced rows that lined up neatly and hid that difference.

def component_scores(runs, labels) -> dict:
    sel_ok = sel_n = 0
    ret_ok = ret_n = 0
    syn_ok = syn_n = 0
    for run in runs:
        for span in run.spans:
            if span.kind == "model_call" and span.produced_tool_call:
                lab = labels.action_for(span.state_digest)      # valid-action set, chapter 4
                if lab:
                    sel_n += 1
                    sel_ok += score_action(span.tool_call, lab)["args_ok"]
            if span.kind == "tool_call" and span.name in RETRIEVAL_TOOLS:
                ret_n += 1
                ret_ok += span.result_rows > 0 and span.contains(labels.gold_for(span))
        if run.reached_final_message:
            syn_n += 1
            syn_ok += final_message_consistent_with_end_state(run)   # judge, calibrated
    return {"selection": sel_ok / sel_n,
            "retrieval": ret_ok / ret_n,
            "synthesis": syn_ok / syn_n,
            "end_to_end": sum(r.verdict.score == "pass" for r in runs) / len(runs)}

Running that before and after the change localises the drop immediately.

Nothing in that table teaches by its levels. A retrieval score of 91.0% is neither good nor bad on its own, since it depends on how hard the retrieval calls in this set happen to be, and the same holds for every other cell. The finding lives in the gaps between the before and after columns, read row by row. Selection lost sixteen points while the other two held, which points the investigation at the prompt's tool descriptions rather than at the availability API or the answer template. The retrieval row moved four tenths of a point across more than a thousand scored decisions and the synthesis row six tenths across three hundred, both well inside the standard error the derivation above gives for proportions of those sizes. One gap that dwarfs its own noise and two that sit inside theirs is what localisation looks like.

Without the component split, the seventeen-point end-to-end drop is compatible with a retrieval regression, a synthesis regression, a selection regression, or two of them moving in opposite directions and partly cancelling — and the last case is the one that makes a flat end-to-end number actively misleading. Work it through. A prompt change rewrites the tool descriptions and also lengthens the instructions. Selection improves, because the descriptions are clearer. Retrieval degrades, because the longer prompt pushes query construction into a worse shape. If the two effects are close in size the end-to-end rate comes back level, and level is exactly what it looks like when nothing has changed at all. The aggregate cannot separate "no stage moved" from "two stages moved and met in the middle", and a reviewer reading it reports the first — a positive claim the number does not support. An uninformative number stays silent. This one answers, and answers wrongly.

The synthesis row is worth a note on its own, because it is the one component here whose correct output is not mechanically checkable. Chapter 1 deliberately removed everything unobservable from the success definition; a final-message consistency check reintroduces a judge — a model scoring another model's output against a rubric — assessing whether the message the agent sent matches the end state it actually produced. That judge needs the same calibration discipline any judge needs, and its number should be read as a component signal rather than as part of the gate.

A generous definition and a set of the wrong shape leave the same component scores#

The three explanations at the top of this chapter are not equally addressed by the work above. Re-scoring under a strict definition tests the first directly. Trajectory classifiers test the second directly. Component scores localise a regression across stages, which is a different question again and a valuable one.

None of them can distinguish a generous success definition from an eval set whose inputs are the wrong shape, and the reason is structural: both leave the component numbers looking healthy. A set full of easy two-attendee same-timezone threads produces high selection, retrieval and synthesis scores because the decisions in it genuinely are easy, and a definition that counts partial completion as success also produces high scores on stages that individually did their job. In both cases the components look fine and users are unhappy, and running the component analysis a second time will produce the same reassuring table.

The deeper reason is that all three instruments draw their evidence from the same source, which is the eval set: re-scoring changes how the stored runs are read, trajectory classifiers change which part of them is read, and component scores change the granularity. None of them changes what was run, so none can report on a task the set does not contain, and a set of the wrong shape is defined by what it leaves out. An instrument cannot detect an absence in its own input.

Separating the two needs an instrument pointed at something outside the eval set entirely, which is a sample of what the agent is actually doing for real users, scored by the same assertions the eval uses. That is the subject of the next chapter, which compares the eval's input distribution against production's, and it is why the third row of the table above defers rather than resolving.

Everything before that is one move applied at three depths: take the number that has stopped moving and ask what it is a function of. The headline is a function of the fold from three verdict categories into two, of the assertions making up each verdict, of the end state those assertions read, and of the tasks that were in the set at all. Each is a place where a quantity can hold steady while the thing underneath it changes, and each has a check that runs against stored data in an afternoon. The cheapest of them carries the least judgement — rig a fixture against each assertion, confirm it notices, repeat on a schedule. Consistency in a measurement is evidence of consistency in the system only once the measurement has been shown capable of moving.