Compressing a tool description

A pull request lands with a title that reads like good housekeeping: trim export_report description, 180 → 45 words. The old text carried three worked examples and a paragraph on when the tool applies. The new one is two sentences. The author has attached eval results, and they are the reason the PR looks safe: 500 labelled requests replayed against both versions, selection accuracy 94% either way. Given chapter 5's arithmetic, cutting 135 words out of a block that is re-sent on every turn is worth about $2,400 a year on this traffic, and the eval says it costs nothing.

That dollar figure is a year of a per-turn saving, and it inherits every condition the previous chapter attached to one: this traffic, this turn count, and caching configured the way it currently is. It is also the entire benefit on the table. Nothing else improves when a description gets shorter — not accuracy, not latency worth measuring, not anybody's time — so the whole decision rests on whether the cut costs something, which is precisely what the attached eval was supposed to answer.

The eval is not lying. It is answering a question that has almost nothing to do with why the description exists, and the reason is visible in the arithmetic of the 94% itself.

What an aggregate accuracy number is made of#

A labelled sample drawn in traffic proportions is mostly easy traffic, because production is mostly easy traffic. Break the same 500 requests into slices and the flat number resolves into something with structure:

The aggregate is a weighted average of that column, and reconstructing it is what the table is for. Multiply each slice's accuracy by the number of cases in it — four hundred and twenty at ninety-seven percent, fifty at ninety-two, fifteen at seventy, fifteen at sixty-two — then add the four products and divide by five hundred. The result rounds to the ninety-four percent the pull request quoted. Do the same on the other arm, where only the third slice differs, and it rounds to ninety-four as well. Neither figure is wrong. Neither is a summary of anything a reader of the pull request cares about, because the weights in that average are shares of traffic and shares of traffic have nothing to do with which slices a description arbitrates.

The top row is 84% of the sample and it is decided before the description is read. A request that says download last quarter's revenue as a CSV matches one tool and no other; the name alone routes it, and the description could be four words or four hundred without moving the number. That row is most of the 94%, which means most of the 94% is a measurement of how unambiguous your traffic is rather than of how good your descriptions are. Two descriptions can be identical on it while being materially different objects.

The third row is the one the 180 words were written for, and the sample contains fifteen requests from it. Six percentage points on fifteen requests is 0.9 requests. The difference between A and B, in this eval, is that one request out of five hundred went somewhere else. There is no experimental design under which that is a signal, and there is no threshold you could have set that would have caught it.

The statistics make the failure precise rather than merely intuitive. At n = 500 and p ≈ 0.945, the standard error on each arm — how far the measured accuracy would wander if you drew the sample again from the same traffic — is about 1.0 point, so the 95% confidence interval on the difference between the two arms is roughly ±2.8 points of overall accuracy. Now ask what a slice worth 3% of traffic can contribute to overall accuracy at most: three points, if it goes from perfect to zero. The entire confusable pair could collapse completely and the aggregate would move less than its own noise floor. An eval built this way is not insensitive to the slice; it is structurally incapable of seeing it, and running it on 5,000 requests instead of 500 improves the aggregate's precision while leaving the slice at 150 requests and the design unchanged.

Those two numbers come out of one formula, and writing it down is worth the paragraph because the sample-size decision later in this chapter uses the same one. The spread of a proportion measured on n independent trials is the square root of the rate, times one minus the rate, divided by n. At a rate near ninety-four and a half percent on five hundred cases that works out at about a point. Comparing two arms needs the spread of a difference, which is larger than either arm's own spread by a factor of the square root of two, and turning a spread into a ninety-five percent interval multiplies by a little under two more. That is the two point eight.

Set that against what the contested slice can actually contribute. Three percent of traffic caps its contribution to the aggregate at three points, and only if it collapses from perfect to nothing; the movement that was really observed, six points inside the slice, contributes under a fifth of a point. The noise on the comparison is more than a dozen times that. Sampling ten times as many requests narrows the noise by about a factor of three and leaves the contribution exactly where it was, because the slice is still three percent of whatever you draw. The gap closes, in other words, but at a rate and a cost that make the sample size the wrong thing to change.

An eval that can see the thing you are testing#

The fix is to stop sampling in traffic proportions and start sampling by slice — stratified sampling, in the statistical vocabulary, where you fix the sample size of each subgroup yourself instead of letting the population decide it — with the contested slices deliberately oversampled to whatever size gives you power on them. Requests where two tools genuinely compete are the ones the description exists to arbitrate, so they get the sample budget; requests that route on the name get a small regression set to catch outright breakage and nothing more.

from collections import defaultdict
from dataclasses import dataclass
from math import sqrt


@dataclass(frozen=True)
class Case:
    request: str
    expected: str | None  # None means: no tool should be called
    slice: str


def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """95% interval that behaves near 0 and 1, unlike the normal approximation."""
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return (centre - half, centre + half)


def evaluate(cases: list[Case], route) -> None:
    hits: dict[str, int] = defaultdict(int)
    totals: dict[str, int] = defaultdict(int)
    for case in cases:
        totals[case.slice] += 1
        if route(case.request) == case.expected:
            hits[case.slice] += 1

    for name in sorted(totals):
        n, k = totals[name], hits[name]
        lo, hi = wilson(k, n)
        print(f"{name:<34} {k:>4}/{n:<5} {k / n:6.1%}  [{lo:5.1%}, {hi:5.1%}]")

The expected field being nullable is doing more work than it looks. A case whose correct answer is no tool at all is the other thing chapter 4's negative criteria are for, and an eval that only scores which-tool-was-picked will happily give full marks to a description that has taught the model to reach for a tool on every question. That slice belongs in the harness from the start, because it is the one that silently disappears when someone compresses a description by deleting the sentences that begin with do not.

Sizing the contested slice is a power calculation rather than a guess, power being the probability that a difference of a given size, if it is really there, comes out of your measurement as something you can distinguish from noise. At an accuracy around 70%, detecting a six-point difference between two arms at 95% confidence needs roughly a thousand cases per arm: the standard error on a difference of two proportions at p ≈ 0.7 and n = 1,000 is about 2.0 points, so six points clears it comfortably, while at n = 300 the same standard error is 3.7 points and a six-point gap is inside the noise. On the agent above, a thousand contested cases took two people a day and a half to label, and it was the only measurement in the whole exercise that distinguished the two candidates.

Both of those standard errors come out of the same square root as before, with the rate now near seven in ten rather than nineteen in twenty, and two things follow that are easy to get backwards. A lower accuracy carries a larger spread at the same sample size, because the spread peaks when a rate sits near a half and shrinks towards either extreme — so the contested slice, which is the hardest one, is also the one needing the most cases. And detecting a six-point difference takes a standard error comfortably below six points rather than merely below it, since a difference has to clear roughly twice its standard error before anyone should act on it. That is why a thousand cases is the figure and three hundred is not. Three hundred does not fail by a little.

Auditing the 135 words that were removed#

The eval tells you whether a cut hurt. Reading the description tells you which cut hurt, and it is worth doing before the eval rather than after, because it turns a binary result into a diff you can act on. Every clause in a description is doing one of a small number of jobs, and they are not equally load-bearing.

The PR had cut 135 words, and 96 of them came from the first three rows, which is why the author was right that the description was bloated. The other 39 were the handoff sentence naming schedule_report and two of the three examples. The handoff is the whole of the six-point gap: without it, export_report and schedule_report are two accurate self-descriptions with an undefined boundary, which is exactly the state chapter 3 was written to get out of, reintroduced by a compression pass that nobody thought of as a behavioural change.

Ninety-six and thirty-nine is the hundred and thirty-five, and that split is the whole finding. Roughly seven words in ten of what came out were restatement, global policy in the wrong file, or prose duplicating a schema the model reads anyway — so the author's instinct that the description was bloated was correct. The other three in ten were the two things the eval could not see. A compression pass that had stopped after the first three rows of that table would have removed ninety-six words, captured most of the money, and changed nothing else at all; the revised version removed a hundred and nine by also dropping two of the three examples, which is more than the ninety-six and still keeps the sentence that was doing the work.

That gives the criterion cleanly. Cutting a description from 180 words to 60 is safe when nothing you removed was what separated this tool from its nearest neighbour, or from not calling anything at all. Everything else in a description is either decoration or duplicated from a place the model reads anyway, and decoration is worth removing on chapter 5's arithmetic alone. The revised PR on this agent landed at 71 words — the two discriminating sentences, one precondition, one example — scored 70% on the contested slice against A's 70%, and saved 109 words.

Halving a description is therefore not the operation to hold an opinion about. Two cuts of identical size can differ completely in effect, and the word count is the one property of a cut that carries no information at all about which of the two was made. It is still the property every review sees first, because it is the one visible in a diff header.

Why a global cap is the wrong rule#

Someone will eventually propose that every description be capped at one sentence, and the proposal is attractive because it is checkable in CI and needs no judgement. It fails on the same fact the audit exposes: the length a description needs is a property of its neighbourhood, not a constant. A tool with no near neighbour, an obvious argument shape and no precondition genuinely needs one sentence, and giving it three is waste. A tool sitting in a contested band with a sibling it must hand off to and an id it must not invent needs four sentences, and capping it at one deletes exactly the content that a nearest-neighbour audit would have flagged as load-bearing. A uniform cap spends the same budget where it is not needed and starves the case that pays for the whole exercise.

The budget worth enforcing is on the block rather than the sentence. Cap the total token cost of the tool definitions, publish the per-tool breakdown from chapter 5's measurement, and let the allocation be as uneven as the neighbourhood requires. That version has the property the sentence cap lacks: when the budget is exceeded, the conversation it forces is about which tools are competing and whether they should both exist, rather than about which adjectives to delete.

Padding, and the shelf life of a slice#

It is not an argument that long descriptions are safe. Padding costs more than tokens, because a discriminating sentence buried in four sentences of restatement is competing for attention with text that says nothing, and on the same eval a version of A with the three examples expanded into a worked walkthrough scored 66% on the contested slice — four points below A, at 60% more tokens. Compression is a real improvement when what it removes is decoration; the claim here is only that you cannot tell which case you are in from an aggregate.

It is also not a claim that per-slice evals are cheap or permanent. A thousand labelled contested cases is real work, and the slices themselves are not stable: every tool you add creates new pairs, and a slice that was uncontested last quarter becomes contested the moment something lands next to it. Teams that build this harness once and never revisit its stratification end up with a precise measurement of last year's ambiguities.

A description is not general-purpose text and cannot be judged by a general-purpose number. It exists to arbitrate a small, identifiable set of contested requests, and the only measurement that speaks to it is one whose sample is chosen from that set. Everything else in the string is decoration or duplication, and an audit that names each clause's job tells you which is which before you spend a day and a half labelling anything.

The audit also has a failure mode of its own, which is the subject of the last chapter. Sometimes you classify every clause honestly, find that none of them are decoration, and are left with a description whose load-bearing content runs to a paragraph of exclusions against four different neighbours. Nothing in this chapter tells you what to do then, because at that point the problem has stopped being the prose.