Per-tool and per-pair selection accuracy

Chapter 4 answered the question of why a catalogue got worse after the fact. This chapter is about the run you design before adding tools, which is a different exercise, because the design that happens by default — add the tools on a branch, run the eval, look at the number — produces a figure that cannot answer the question no matter which way it comes out.

Two things have to be right for it to answer anything. The configuration has to change in exactly one respect between rungs, which is harder than it sounds because at least four things drift while nobody is watching. And the output has to be read at the granularity where a decision lives, which is the individual tool and the individual confused pair rather than the catalogue.

What has to be held still, and what has to move#

The measurement is one fixed set of labelled requests replayed against catalogues of increasing size, with the size as the only thing that changes. Each of the four drifts below is large enough on its own to swamp the effect you are trying to see.

The model is the first. "The latest version of the model" is not a fixed quantity, and a provider alias that resolves to a new snapshot mid-experiment will move selection accuracy by several points on its own. Pin the exact version identifier in the run config and record it in the results, so that a comparison between two runs a month apart can be rejected on sight if the identifier differs.

The prompt is the second, and it drifts because the prompt is usually where someone writes the routing guidance. Adding twelve tools tempts a developer to add two sentences about when to use open_dispute rather than refund_order, and those two sentences may be an excellent idea, but including them in the same run confounds the catalogue change with a prompt change. Ship the prompt change as a separate rung of its own, a rung being one step in the ladder of configurations the run climbs — one catalogue, one prompt, one model, scored on its own before the next thing is added.

The third is tool ordering inside the request. Tools are serialised into the request in some order, and models are measurably sensitive to position: the same thirty-tool catalogue, permuted three ways, gave 88.4%, 89.1% and 90.6% on the same 400 requests in this run. That 2.2-point spread is the noise floor of the whole exercise. Rotate the order across at least three seeds — a seed being the number that fixes one particular shuffle of the tool list, recorded so the same shuffle can be reproduced on a later run — report the mean, and treat any difference smaller than the seed spread as nothing at all.

Reporting the mean is what connects those three permutation figures to the ladder below, and the connection is worth checking rather than assuming. Averaging eighty-eight point four, eighty-nine point one and ninety point six lands within a rounding of the eighty-nine that the thirty-tool rung reports, so the two tables are the same run seen at two levels of detail rather than two runs that happen to agree. That also fixes what the spread means. The gap between the highest and lowest permutation is the amount the same configuration can move without anything changing, so it is the floor under every comparison in the chapter, and a difference that does not clear it has not been measured.

The fourth is the labelled set, which cannot stay entirely fixed and has to change in a controlled way. Twelve new tools bring capabilities that the old 400 requests never exercised, so a set that only contains old requests measures regression on existing behaviour and is blind to whether the new tools work. It also contains requests whose correct answer genuinely changes: nine of the 400 were labelled refund_order for situations that a real disputes process would route to open_dispute, and leaving them labelled as before scores the model wrong for being right. Adjudicate those nine before the run, write down the decision, and keep the two slices separate in the output — 400 existing-capability requests and 140 written for the new tools, never averaged into one figure.

LADDER = [
    ("base",        BASE_18),
    ("+disputes",   BASE_18 + DISPUTES_4),
    ("+subs",       BASE_18 + DISPUTES_4 + SUBS_4),
    ("+giftcards",  BASE_18 + DISPUTES_4 + SUBS_4 + GIFT_4),
]

def run_rung(name, catalogue, requests, model_id, seeds=(11, 12, 13)):
    rows = []
    for seed in seeds:
        tools = permute(catalogue, seed)          # position control
        for req in requests:
            chosen = agent.select_tool(
                req.messages, tools=tools,
                model=model_id, temperature=0,    # pinned, not "latest"
            )
            rows.append({
                "rung": name, "seed": seed, "n_tools": len(catalogue),
                "request_id": req.id, "slice": req.slice,
                "expected": req.label, "chosen": chosen,
                "correct": chosen == req.label,
                "pair": None if chosen == req.label
                        else (req.label, chosen),
            })
    return rows

The pair column is the one that pays for the whole harness. Recording only correct gives you an average and forces a second run to find out what the average is made of; recording the ordered pair of expected and chosen gives you the confusion structure for free, from the same rows.

Read it per tool and per pair#

The ladder came back with a headline that looks like a modest, acceptable cost.

Four points of regression on existing behaviour in exchange for twelve capabilities that themselves route at 90%. Read as a trade, that is a good quarter's work, and a team that stops here ships. The per-tool view of the same rows describes a different product.

Percentages on denominators that small are exact fractions in disguise, and converting them back is what stops the table being over-read. Twenty-four requests sit behind refund_order, so each one is worth a little over four points on that row and nothing else is possible: ninety-five point eight is twenty-three of the twenty-four, and forty-five point eight is eleven of them. The fifty-point drop is therefore twelve requests changing hands, which is both smaller than it sounds and more alarming than it sounds — smaller because twelve is not many observations, and more alarming because half of a capability's labelled requests now go somewhere else. The same conversion applies to the row below it, and the two together are what the sentence after this table is counting.

Sixteen requests moved from correct to incorrect across the whole set, and seventeen of those losses sit in two tools, offset by one request elsewhere that flipped the other way inside the seed noise. refund_order, which handles the single highest-volume intent the agent receives, has gone from a capability that works to a coin flip. In production terms that is not a four-point degradation of the agent; it is one feature breaking, with an average laid over the top of it.

Reading per tool does cost something in statistical resolution, and the cost is worth stating plainly. With 24 requests behind refund_order, a single-run figure of 45.8% carries a 95% interval of roughly ±20 points, meaning the band the true rate would fall inside on nineteen reruns out of twenty, which is wide enough that a five-point wobble on a tool is unreadable. Three seeds and 72 effective trials narrow it to about ±12, and a drop from 95.8% to 45.8% survives that comfortably, which is the general shape of the trade: per-tool reporting is blind to small movements and sharp on collapses, while the aggregate is sharp on small movements and blind to collapses. Since a collapse is the thing that ends a feature, the per-tool view is the one to build the criterion on, and the way to buy resolution where you need it is to over-sample the four or five highest-volume tools in the labelled set rather than to spread requests evenly.

The reason three seeds narrow the band by so much less than a factor of three is the one piece of statistics worth carrying out of this chapter. An interval of that kind shrinks with the square root of the number of trials, not with the number of trials, so tripling the trials divides the width by the square root of three — a little over one and a half — which turns roughly twenty points into roughly twelve and no further. Getting to half the original width would need four times the trials, and to a quarter of it, sixteen times. Seeds are a cheap way to buy the first factor and a hopeless way to buy the third, which is the arithmetic behind putting more requests behind the tools you care about instead of running the same set more often.

The pair column says which two tools to look at.

Eleven of the sixteen new failures are one pair. The remedy that follows from that row is a description rewrite, a merge, or a routing rule between two named tools, which is an afternoon; the remedy that follows from "89%, down four points" is a decision about whether to ship twelve features, which is a quarter. Both readings come from the same 1,620 rows.

That row count is the arithmetic that makes the claim literal rather than rhetorical. The labelled set is the four hundred existing-capability requests plus the hundred and forty written for the new tools, which is five hundred and forty, and each is scored once per seed across three seeds. Five hundred and forty times three is the sixteen hundred and twenty. Nothing was run twice and nothing extra was collected: the aggregate, the per-tool view and the pair table are three GROUP BY clauses over one set of rows, which is why the expensive part of this exercise is deciding what to record per row and not deciding what to compute afterwards.

Write the acceptance criterion before you run it#

The instrument only helps if the threshold was fixed in advance, because a number that arrives before a criterion will find a criterion that accommodates it. Eighty-nine percent looks like a pass when 93% was the baseline and twelve capabilities were the price; it looks like a failure when someone has already written down that no tool may fall below 85%.

A criterion worth committing to has three parts and refuses the aggregate as an answer to any of them. No individual tool with meaningful volume may drop more than a stated margin — five points is a defensible starting figure and anything that large is already user-visible on a high-traffic intent. No single confused pair may account for more than a stated share of total failures, because a pair above that share is a surface problem masquerading as a capacity problem. And the per-tool figures are weighted by production call volume when you roll them up, since a tool that serves 30% of traffic and a tool that serves 0.2% contribute equally to an unweighted mean and unequally to everything a user experiences.

That last point cuts against the eval set as it is usually built. Labelled sets are assembled for coverage, so they carry roughly equal numbers of requests per tool, which is the right shape for detecting a regression anywhere and the wrong shape for estimating the damage. The 24 refund_order requests are 6% of the set and refunds are 31% of live conversations, so a volume-weighted read of the same rows puts the effective regression around 17 points rather than four.

Working that reweighting through by hand is what makes it convincing, and it takes one multiplication per tool. An unweighted mean gives refund_order the share it holds in the labelled set, which is six in a hundred. Production gives it thirty-one in a hundred, roughly five times as much. Its fifty-point drop therefore contributes about six-hundredths of fifty to the unweighted figure and about thirty-one hundredths of fifty to the weighted one, which is a bit over fifteen points from that tool alone — most of the seventeen, with the rest coming from the other collapsed capability. The unweighted mean was never wrong about anything; it was answering how much of a coverage-balanced set the model gets right, and nobody asked that.

Selection accuracy is not task success#

A chapter of measurement design should be explicit about the distance between the two. Every number above scores which tool the model chose against a human label, and says nothing about whether the arguments were right — a perfectly routed refund_order carrying the invented reason_code from chapters 1 through 3 counts as a success here. It also says nothing about multi-step tasks, where the correct behaviour is a sequence and the model can pick every tool correctly in the wrong order, or stop one call short. Single-call selection accuracy is the tractable slice of routing quality, and the fact that it is measurable is a reason to measure it, not evidence that it is the whole of the thing.

The set also ages. These 540 requests describe how customers wrote to this agent in one quarter, and the traffic mix shifts with seasons, promotions and whatever the mobile team changed in the entry point. A labelled set that has not been resampled from live traffic in a year is measuring a population that no longer exists, and it will keep returning stable, reassuring numbers while doing it. Resampling is also the only thing that keeps the volume weights honest, since those weights come from production and drift for the same reasons.

With the pair table in hand the mitigations become comparable, and the cheapest of them is a naming convention. Prefixing a catalogue with crm., billing. and support. costs nothing and ships in an hour, and it is the first thing most teams reach for once a catalogue crosses thirty tools. Whether it moves anything is decided by the same table: a scheme that separates categories can only help with failures that cross a category boundary, and the pairs above are not obviously of that kind.