Agent evaluation

Read these first

• Agent observability
• Component vs end-to-end evaluation

What you will know

• Define task success for an agent, precisely

  Use observable assertions, tolerances, and explicit handling for partial completion so independent raters produce the same score.

• Measure trajectory quality, not just the final answer

  Score unnecessary steps, unsafe actions, and recoverability alongside outcome correctness; a plausible result can conceal a brittle path.

• Measure tool-selection accuracy separately

  Compare chosen tools and arguments with a labeled expected action or valid-action set, including cases where the correct choice is to use no tool.

• Track cost per task as a first-class metric

  Break spend down by model calls, tokens, tool invocations, retries, and p95 or tail tasks rather than hiding expensive outliers in an average.

• Show an agent that succeeds expensively and say whether that passes

  Set a task-level budget and policy for trading extra work against reliability, then test how the decision changes at different volume and latency targets.

Chapters

1. 1

   Task success has to be checkable by two raters who never speak

   "The meeting got scheduled and the user is happy" is not a definition. Replace it with observable assertions on the end state, a tolerance where one is legitimate, and an explicit rule for partial completion — then test the definition against the run that booked the right attendees at the wrong time. When two independent scorers disagree, the definition is the defect rather than the scorers.

   Read chapter
2. 2

   A correct answer reached by a lucky path will repeat the path

   Outcome-only scoring cannot see the run that took forty tool calls where six would do, or the one that wrote a file it had no reason to write. Scoring the trajectory adds unnecessary steps, unsafe actions and recoverability alongside correctness, and each of the three needs a mechanism rather than an intention: a reference path or a per-task step budget to count steps against, a declared list of actions that count as unsafe, and the fault injection the next chapter builds. The path is what production repeats thousands of times a day while the answer is only what you inspected once.

   Read chapter
3. 3

   Recovery is measurable only if you break something on purpose

   Every other metric in this topic reads an unmodified run, which is exactly why recovery cannot be one of them: nothing failed. Put fault-injection fixtures into the eval set — a tool that times out, an API returning a 500, a search that comes back empty — with a stated policy for which failures are injected into which tasks, and score three distinct outcomes rather than one: the agent reached the goal by a different path, it gave up cleanly, or it succeeded despite the fault without ever noticing it. Separating those is what turns recoverability from a word in the rubric into a column in the report.

   Read chapter
4. 4

   Tool-selection accuracy is meaningless without the no-tool cases

   Labelling one correct tool per input produces a metric that rewards always calling something. The ground truth needs inputs where the right action is to call no tool at all, and it has to admit more than one valid action where more than one exists. Even then the number stays silent about arguments: a correct tool invoked with the wrong parameters still scores as a hit.

   Read chapter
5. 5

   The mean cost per task is where the expensive runs hide

   One task at fifty times the median disappears into an average taken across a few hundred. Report the distribution instead, broken out by model calls, tokens, tool invocations and retries, with p95 and the tail named explicitly. This chapter covers the breakdown that lets you say where the money went rather than only how much of it there was.

   Read chapter
6. 6

   A run that succeeds expensively passes only against a budget

   Forty steps and $2.10 against a median of six steps and $0.30, with the right answer at the end. Whether that passes is not a judgement made afterwards; it depends on a task-level budget declared before the eval ran and on a stated policy for trading extra work against reliability. Then answer the same question again at a hundred tasks a day and at a hundred thousand.

   Read chapter
7. 7

   92% on the eval with users complaining points at the eval

   Nothing changed in the model, so the explanations worth checking today are about measurement: a success definition that scores partial completion generously, trajectories nobody scored, and a distribution the set does not contain. Component scores are part of the answer, since an end-to-end number cannot say whether selection, retrieval or synthesis moved when it falls from 88% to 71%.

   Read chapter
8. 8

   Re-scoring sampled production runs is what convicts the eval

   Component scores locate a regression among selection, retrieval and synthesis, and they cannot tell a generous partial-completion rule from an eval set whose input distribution is wrong, because both leave the component numbers looking the same. The instrument that separates them is a sample of real production runs, scored with the same end-state assertions written in the first chapter, compared against the eval set on both the success rate and the shape of the inputs. A gap in the score convicts the definition; a gap in the distribution convicts the set.

   Read chapter