Instrumenting a tool surface change

An arm, in this chapter's sense, is one complete configuration of the system — surface, prompt, model, task set — run end to end over every task, differing from the arm it is compared against in a single respect. The word matters because tool surface changes are cheap to make and every one of them is accompanied by a temptation to also fix the description while you are in there. Two changes in one diff produce one number, and one number cannot be attributed to either of them. Isolating one from the other is not a statistical refinement to be added later; it is the difference between a result and an anecdote.

This is the least glamorous chapter in the topic and it is the one everything else rests on. Every claim in the chapters that follow — that absorbing a cursor is worth nineteen points, that a merge halves a cache miss rate, that a hundred and forty tools cost seventeen points of task success — is a difference between two arms that differed in one respect. The instrumentation below is what makes those differences readable, and gathering it before a change is the half that cannot be recovered afterwards.

Four arms and one task set#

The fix is unglamorous. Freeze a task set, run every candidate against it, and change exactly one thing per arm.

Arm C is the one that answers the question a surface redesign claims to answer, and it is the one that most often goes unrun. Running all four is not four times the work in engineer-hours; it is one harness invoked four times with different configuration, and the difference between C and A is the number the redesign is allowed to claim.

The frozen task set is the part that people resist, because it goes stale and because everyone knows it leaks. Both are true, and the discipline is to version it rather than replace it. Keep tasks-v1.jsonl forever, add tasks-v2.jsonl when you need harder cases, and report every arm against both. A new task set is a new instrument, and the first thing you do with a new instrument is calibrate it against the old one by running the baseline through it. If arm A scores 61% on v1 and 79% on v2, you have learned that v2 is easier by eighteen points and you can stop wondering whether your redesign is real.

Each task in the set is a user request plus a checkable outcome, not a transcript to match. Matching transcripts punishes any solution that takes a different path, which is exactly what a surface redesign is supposed to do.

{"id": "t_041",
 "request": "Email Acme their most recent unpaid invoice.",
 "fixtures": {"customer": "c_8f2a", "open_invoices": 27},
 "assert": {"tool": "send_email",
            "args": {"to": "ap@acme.example",
                     "attachment_url": {"contains": "in_4c19"}}},
 "budget": {"max_calls": 12}}

Four numbers of which two interact

Running all four arms produces a set of results that does not decompose as cleanly as the table implies. On one billing agent's frozen set, with three seeds each — three complete passes with a different sampling seed, since two passes of one arm do not return the same number — the figures were 61% for A, 74% for B, 80% for C and 85% for D. The prompt rewrite on its own is worth thirteen points and the surface redesign on its own is worth nineteen, so if the two effects were independent the shipped configuration would land near 93%, and it came in eight points below that.

Each of those two claims is a subtraction against the same reference, and the subtraction is only legitimate because of how the arms were built. Arm B differs from arm A in the prompt and in nothing else, so whatever separates their scores is the prompt and cannot be anything else. Arm C differs from arm A in the surface and in nothing else, so the gap between it and the baseline is the surface. Adding the two gaps back onto the baseline is what the word independent is doing in that sentence: it assumes the prompt is worth the same thirteen points whether or not the surface has already been fixed. Sixty-one plus thirteen plus nineteen lands in the low nineties. The shipped arm came back in the mid-eighties, and the eight-point difference between the prediction and the measurement is not noise in the measurement — it is the assumption that was wrong.

Sub-additivity of that size deserves a look rather than a shrug, because it usually means the two changes were solving an overlapping share of the same problem. Here they were. Most of the 1,100-token prompt was a worked example showing how to chain the nine old tools — carry the customer id forward, keep requesting pages until has_more goes false — and that instruction has nothing to act on once the four-tool surface ships, because there is no id to carry and no page to request. Roughly eight of the prompt's thirteen points had been compensating for the surface, and when the surface stopped needing compensation those points had nothing left to buy.

The belief worth dislodging here is that a shipped configuration's score is the sum of the changes inside it, which is how a pull request comes to be described as prompt plus surface, thirty-two points. Two changes that repair overlapping shares of the same failures cannot both be credited with the overlap, and there is no way to tell from the shipped arm alone how large the overlap is, because the shipped arm is a single number and the overlap is a relationship between three. That is the argument for running arm B and arm C even when nobody intends to ship either of them. They are not candidates. They are the only way to price what the candidate is made of.

That reading suggests a fifth arm, and it is the one that shipped: the new surface with the prompt trimmed to 420 tokens, the chaining example deleted and the escalation section kept. It scored 84%, one point below arm D and inside the spread across the three seeds, while sending 680 fewer prompt tokens on every turn of every request.

Counting four things, not one#

End-to-end task success is the number that goes in the summary, and on its own it tells you almost nothing about where a surface is failing. The instrumentation needs three more counters, recorded per run and aggregated per tool.

@dataclass
class RunRecord:
    task_id: str
    arm: str
    success: bool              # the assertion held
    calls: list[ToolCall]      # every call, in order
    invalid_calls: int         # rejected by schema validation before execution
    error_calls: int           # executed, returned an error result
    retries: int               # same tool, same args, within one run
    input_tokens: int
    wall_ms: int

def invalid_rate(runs, tool=None):
    calls = [c for r in runs for c in r.calls
             if tool is None or c.name == tool]
    return sum(c.invalid for c in calls) / max(len(calls), 1)

The separation between invalid_calls and error_calls carries most of the diagnostic weight. An invalid call never reached your backend: the arguments failed schema validation, so the model produced something the contract does not permit. An error call was well-formed and the backend refused it, which is a different problem — usually a real-world condition like a missing record or a permission the user does not have. Surface design moves the first number directly. It moves the second only by changing which operations exist at all.

Retries are worth counting separately from calls because they are the signature of an unclear failure result. A model that issues the identical call three times in a row is a model that read the first result as transient when it was permanent, or as permanent when it was transient, and neither of those is a prompt problem.

Reading the breakdown on a real surface#

Take a customer-support agent that recurs through this topic: search_kb, lookup_article, get_article(id), create_ticket, escalate_ticket (opens a P1 and pages on-call), close_ticket, list_open_tickets, get_ticket(id). Eight tools, task success 64%, and about 11% of all calls failing validation. That aggregate is where most teams stop and start arguing about the prompt. Broken down per tool, over 200 runs of the frozen task set, it stops being an argument.

The aggregate and the rows are the same data, and doing the sum once is what makes that visible. Add the calls column and the surface issued a little over twelve hundred calls across the two hundred runs; add the invalid column and a hundred and twenty-eight of those never reached a backend. Dividing the second total by the first is where the roughly-one-call-in-nine figure comes from, and the division is the step that destroys the information. It weights every tool by how often that tool is called, so a tool failing validation on nearly a third of its attempts is averaged against several failing on about one attempt in twenty, and what comes out is a number describing no tool in the surface.

Two tools carry the losses. lookup_article fails validation on nearly a third of its calls and is retried more often than it succeeds cleanly, and search_kb is called nearly twice per run in which it appears. The 11% aggregate — a hundred and twenty-eight invalid calls out of twelve hundred and one — was an average over one badly broken tool and seven healthier ones, and the average was the reason nobody could see it.

The per-pair breakdown is the second half of the diagnosis, and it is the one that names the fix. For every run, record which tool the model called first for each user intent, and cross-tabulate against the intent the task file declares.

A model splitting 47/37 on one intent and 21/31 on another is not choosing; the two tools are interchangeable from where it is standing, and the split is close to what you would get from a coin. get_article shows the contrast — 57 of 64 on its own intent — because it takes an argument nothing else takes, and the argument routes the call without any wording having to. Chapter 5 is about what to do with a pair that looks like the top two rows of that table. What matters here is that the table exists before anyone proposes a change, because the same table run after the change is what tells you whether the change worked.

The counters that are allowed to get worse#

A redesign can raise task success while making the system worse along an axis the success number cannot see, which is why the same RunRecord has to carry the counters that would show it. The ones worth gating are the ones where a regression is expensive and quiet: operations the agent performs that cannot be undone, runs that land on a human's desk anyway, and the load the tools place on the services behind them.

Two of those rows moved the wrong way in the arm that everyone was happy with. Upstream reads nearly tripled, because a tool that assembles its own result set fetches pages the model used to stop asking for, and that was harmless against this backend while it would not be against one where a single search page costs 200 ms of database time. The count of irreversible actions is the more serious of the two, since a surface that makes a workflow easier makes every operation inside it easier, including the ones nobody wanted made easier, and eight extra per hundred runs is a question a human has to answer by reading transcripts rather than a rate you can threshold.

Both of those rows are quoted against a denominator, and the denominator is where a gate quietly stops meaning what it says. Irreversible actions are counted per hundred runs rather than per run, so the gate is a rate; a surface that raises task success also raises the number of runs that get far enough to perform an action at all, which means some of the increase is the redesign working rather than the redesign being reckless. Reading the row honestly means dividing by completed runs and not by attempted ones, and those two divisions stop agreeing once completion has moved nineteen points. Upstream reads carry the same problem from the other side: the figure is per run, and a run in arm C is shorter than a run in arm A, so a per-run count that nearly tripled understates how much heavier each individual call became.

The limits of an offline harness#

An offline harness measures a fixed distribution of requests, and production is not fixed. The frozen task set was written by people who already had a model of what users ask, so it under-samples the long tail — the malformed requests, the ones with three intents in a sentence, the ones where the user is wrong about their own account. An arm that gains six points offline can gain two in production, or none, and the honest way to run this is to treat the harness as a filter that stops bad changes from shipping rather than as a predictor of the gain.

Sample size deserves more caution than it usually gets. With 50 tasks, an observed 88% carries a 95% confidence interval of roughly 76% to 95%, which overlaps comfortably with an observed 78% on the same set. Two arms that differ by ten points on 50 tasks have not been distinguished. Getting the interval down to about ±5 points at these rates takes on the order of 200 tasks, and if your tasks are expensive to check, paired comparison helps: run both arms on identical tasks and count only the tasks where they disagree, which removes the variance contributed by tasks that are trivially easy or impossible for both.

Where that interval comes from is worth having rather than taking on trust, because it is the calculation that decides how many tasks somebody has to write. Each task either passes or fails, so the number of passes behaves like a run of coin flips whose bias is the arm's true success rate, and the spread of such a count — expressed as a rate rather than a count — shrinks with the square root of the number of tasks rather than with the number of tasks. Quadrupling the task set therefore halves the interval, and merely doubling it buys under a third. That square root is the whole reason fifty tasks cannot separate two arms and two hundred can: the step between them is a factor of four in labelling work and a factor of two in precision. It is also why adding a fifth arm is cheap and adding a digit of precision is not.

Non-determinism is the third caveat. The same arm on the same task set will not produce the same number twice, so a single run per arm is a sample of one. Three seeds per arm, reporting the mean and the spread, is usually enough to tell a real four-point difference from sampling noise, and it costs three times the API budget of being wrong.

Finally, none of these counters explains why a tool is failing validation. They localise the damage to a tool or a pair, and the reasoning that turns "29% of lookup_article calls are invalid" into a design change is human.

What the instrumentation buys is the right to say a number out loud. Four arms over one frozen task set separate the surface from the prompt; four counters separate a malformed call from a refused one; a per-tool table separates a broken tool from a broken surface; a per-pair table separates a routing problem from an argument problem. None of it is expensive, and all of it has to exist before the first change, because a baseline you did not take is a baseline you cannot take.

The next chapter takes on the first and largest of the design decisions that follow, the one that determines what the whole surface will look like before any individual tool is designed: whether the model gets the API you already have, or one built for it.