Independently labelled evaluation sets

The step under discussion in this topic is a classifier: a piece of software that reads one piece of text and returns one label from a short fixed list — for a support desk, whether an incoming ticket belongs in the billing queue, the technical queue or the refunds queue. Its sibling job is extraction, which reads the same sort of text and pulls named fields out of it, so that a supplier's invoice email becomes an invoice number, an amount and a date in known positions rather than a paragraph. The fixed list of labels a classifier chooses from is usually called the label set, or the taxonomy. Both jobs have been done for decades by keyword rules and by small models fitted on hand-labelled examples, and both can now be done by writing the label definitions into a prompt and asking a large language model (LLM) to answer with exactly one of them. The LLM version spread quickly because it needs no training data and no training run: a working classifier exists an afternoon after somebody decides to build one, which is why these steps now sit all over ordinary backend systems.

Whichever way the job is done, somebody eventually has to say how often it is right, and the instrument for that is an evaluation set, also called a test set: a fixed collection of inputs with the correct label written next to each one, held apart from the system and rerun unchanged every time the system changes. Accuracy is the share of those inputs the system labels the way the set says it should. Where the labels in that set came from is what the number turns on, and it is the one property of an evaluation set that a dashboard cannot show you.

Labels derived from predictions cannot disagree with them#

Most first evaluation sets are built the same way. Somebody samples a few hundred inputs from the previous quarter, runs the system over them, and goes through the output row by row marking each label correct when it looks reasonable. That takes an afternoon and produces a spreadsheet with an input, a predicted label and a green tick. It feels like a test set because it has the shape of one: fixed contents, a score, a reproducible run.

What it measures is the agreement between the model and a human reviewing the model's answer. The labels were derived from the predictions, so no row in the set can contain a prediction that disagrees with its label unless the reviewer actively overruled the model. That reframes the reviewer's task. Asked which queue does this ticket belong in, a person has to hold the policy in their head and make a call. Asked is technical a reasonable label for this ticket, they only have to find a story in which it is, and for a ticket that says "I was charged twice and the app crashes when I open billing", such a story is always available. The second question is much easier to answer yes to, and human reviewers answer it yes at rates in the high nineties even when the underlying decision is genuinely contested.

The arithmetic of that construction is worth doing once, because it explains why the number behaves the way it does. Every row's label was copied from that row's prediction, except on the rows the reviewer actively overruled. Score the system against the set on the day the set is built, and it is right on every row except the overruled ones. The accuracy is therefore one minus the overrule rate, exactly, and no property of the classifier enters into that expression. A reviewer who overrules one row in thirty produces a set that reports agreement in the high nineties whether the classifier is excellent or barely functional, because the classifier is not the quantity being measured.

That also fixes what the later runs measure, which is the part that survives even after somebody notices the first problem. The set is built once and then held fixed, so every subsequent prompt change is scored against labels that record what the first version of the system thought. A change that moves a prediction on some row registers as an error, including when the new prediction is the correct one. A change that leaves every prediction alone registers as no change at all. The quantity being tracked after day one is churn relative to version one, not correctness relative to policy — so the number can move, and it moves for reasons that have nothing to do with whether the system got better.

Two defences get raised at this point and both are worth naming, because each is tempting for a real reason. The first is the discipline around the set rather than the set itself: the evaluation is version-controlled, it is rerun on every prompt change, nobody edits it in place, so a regression could not slip through unnoticed. Every part of that is correct practice, which is exactly why it convinces — but administering an instrument rigorously does not repair what the instrument reads. A rigorous procedure around a quantity equal to one minus the reviewer's overrule rate reports that quantity rigorously. The second defence is that the reviewer did overrule sometimes, so the labels are not simply copies of the predictions. That is true, and it is the wrong shape of true: the overrules are the only source of error in the set, so the reviewer's appetite for disagreement sets the floor of the metric. The classifier's behaviour does not decide how low the score is allowed to go. The reviewer's mood does.

Building a set the model can lose to#

The fix is ordering. Labels get fixed before any prediction exists, by people who cannot see a prediction, working from a written policy that resolves the cases where two labels are both defensible. The policy is the part teams skip and the part that determines whether the exercise means anything. "I was charged twice and the app crashes" needs a rule — primary intent wins, or billing wins over technical when both appear, or the ticket is dual-labelled and the classifier is scored on either. Whichever rule you pick, it has to be written down and applied the same way to every arm of every comparison you run afterwards.

Two annotators on the same 500 tickets, with disagreements adjudicated by a third pass, gives you something else worth having: an agreement rate. If two people following your written policy assign different labels to 12% of tickets, then 88% is roughly the ceiling for any classifier on that task, and a model reporting 94% is either better than your policy or being scored against labels that were themselves coin flips. Cohen's kappa, which scores that agreement after subtracting the agreement two annotators would reach by guessing at your class frequencies, sits around 0.75–0.85 on a normal, workable three-class support taxonomy. Below about 0.6 the problem is the label definitions rather than the classifier, and no amount of prompt work will show up as a measurable gain.

The subtraction inside kappa is the whole idea, and it is easy to read past. Two annotators on this taxonomy agree a great deal by accident, because most tickets are billing. Take the traffic mix this chapter uses below — billing at 62% of tickets, technical at 34%, refunds at 4% — and imagine both annotators ignoring the ticket entirely and drawing a label at random in those proportions. They both say billing on a little under two-fifths of tickets, both say technical on a bit over a tenth, and both say refunds on a small fraction of one in a hundred. Add those three and two annotators who read nothing agree about half the time. So a raw agreement rate of eighty-eight in a hundred is not eighty-eight parts signal; it is roughly half floor, and the rest earned. Kappa reports the earned part as a share of the room there was to earn: agreement above the floor, divided by the distance from the floor to perfect. Eighty-eight against a floor of about half leaves a little under two-fifths earned out of the half that was available, which is a little over three-quarters — and that is where a workable support taxonomy sits. Skew the traffic harder and the floor rises, which is why the same raw agreement rate means less on a taxonomy with one dominant class than on a balanced one.

The ceiling claim needs a condition attached to it, because as stated it is slightly too strong. Eighty-eight in a hundred is the ceiling on agreement with one annotator's unadjudicated labels: on the twelve tickets in a hundred where two people following the same written policy disagreed, one of the two recorded labels is arbitrary, and a classifier can only match whichever one happened to be written down by luck. It is not a ceiling on agreement with the adjudicated set, where a third pass has settled each disputed ticket and a classifier that picks the settled answer is simply right. What the disagreement rate genuinely bounds is how much of a headline number is a measurement of the policy rather than of the system, which is the reason to report it alongside the score rather than in a footnote.

Independent labelling costs real time. Five hundred tickets at roughly ninety seconds each, doubled for two annotators, is about twenty-five person-hours including adjudication. That is the actual price of having a number, and it is worth naming plainly because the alternative that everyone reaches for — letting the model propose and a human confirm — costs an afternoon and produces a number that cannot go down.

That figure is worth seeing assembled rather than taken on trust, because it is the one number in this chapter a manager will push back on. Ninety seconds a ticket across five hundred tickets is forty-five thousand seconds, which is twelve and a half hours: one annotator's pass. A second annotator working the same five hundred tickets independently doubles it to twenty-five. The third pass touches only the tickets the two disagreed on, so at a twelve-in-a-hundred disagreement rate it is sixty tickets and adds something over an hour on top. Twenty-five is therefore the right figure to quote and the wrong one to budget to the hour — the labelling dominates, and adjudication rides just above it.

Aggregate accuracy hides the class you care about#

The second failure is arithmetic rather than epistemic. Take a real-shaped traffic mix for a three-queue support desk over a 500-ticket evaluation set:

Now run two systems over it. The first is a thirty-line rule set built from keyword matching. The second is an LLM classifier with a short few-shot prompt. Their aggregate scores land three points apart, at 91.0% and 94.0%, which is the kind of gap that makes a cost-conscious manager ask why you are paying for the model at all. Broken out by class — precision being the share of the tickets sent to a queue that belonged there, recall the share of a class's tickets that reached the right queue — the two systems are not doing the same job:

The two tables are the same measurement at two resolutions, and it is worth seeing how the second collapses into the first. Recall is the share of a class's tickets that reached the right queue, so recall multiplied by the number of that class present in the set is the count of tickets the system got right in that class. Billing recall of 0.96 over three hundred and ten tickets is a shade under three hundred correct. Technical recall of 0.90 over a hundred and seventy is a hundred and fifty-three. Refund recall of 0.10 over twenty is two. Add the three counts, divide by five hundred, and that is the aggregate — which is another way of saying the aggregate is the per-class recalls weighted by how much traffic each class carries. The refund column enters that sum multiplied by four in a hundred, so a class that is a twenty-fifth of the traffic cannot move the headline by more than about four points however badly it is served. That is not a flaw in the arithmetic. It is the arithmetic working, on a question it was never asked to answer.

Reconstructed that way the two systems land within about half a point of the 91.0% and 94.0% quoted above, in opposite directions, because the per-class figures in the table are rounded to two places before they are multiplied and the rounding errors do not cancel. That is a property of how the report is printed rather than a disagreement between the two tables, and it is a good argument for computing an aggregate from the raw counts and not from a table somebody has already rounded for reading.

The rule set finds two refund requests out of twenty. The other eighteen land in billing, because a refund request is written in billing vocabulary — it mentions a charge, an order number and an amount — and the keyword rules have no way to distinguish "you charged me twice, please fix the invoice" from "I want my money back for order 4471". Those eighteen tickets are 3.6% of traffic, so missing every one of them costs 3.6 accuracy points, and the rule set still lands at 91% because it is nearly perfect on the 96% of traffic that is easy.

Eighteen tickets out of five hundred is three and six-tenths in every hundred, which is where the 3.6% comes from. The step from there to accuracy points is the one worth being explicit about, because it only holds under a condition that the next section removes. Accuracy counts rows and weights every row the same. Eighteen rows moving from right to wrong takes eighteen off the numerator and leaves the denominator untouched, so the score falls by eighteen five-hundredths — the same fraction, now read as a share of the whole set. A share of traffic converts one-for-one into accuracy points precisely because both are shares of the same five hundred rows. Stratify the evaluation set so that its class mix no longer matches production, and the conversion stops working; that is why the weighting step appears again below. "Nearly perfect on the easy 96%" is also looser than the table supports — the rule set is dropping a few billing and technical tickets too, and the recall column says so. What survives the tightening is that the refund column is the only one that is catastrophic, and that the aggregate files catastrophic and ordinary under the same number.

The precision column carries the other half of the story, and it reads back into counts by the same route. Precision is the share of the tickets sent to a queue that belonged there, so a refund precision of 0.67 sitting next to two correct refunds means the rule set sent three tickets to the refund queue in total and got two of them right. It is not flooding refunds with billing complaints. It is barely routing anything there at all. The LLM's 0.81 against seventeen correct puts its refund queue at around twenty-one tickets, so it is pulling in roughly four that do not belong — a different failure, with a different fix, and the aggregate score shows neither of them.

Aggregate accuracy is the wrong summary for a routing problem for exactly that reason. The metric weights each ticket equally, and the business does not. A misrouted billing question waits an extra hour in the wrong queue. A misrouted refund request waits days, and a company's own escalation data will usually show that the chargeback rate on refund requests answered after 72 hours is several times the rate on ones answered same-day. Two systems can be three points apart on a metric that gives no weight at all to the difference that generates the cost.

Reporting per-class precision and recall is what makes that visible. Recall for refund requests answers "of the refund requests that arrived, how many reached the refund queue", which is the question the refunds team is actually asking. Precision answers "of the tickets we sent to refunds, how many belonged there", which is the question the refunds team asks after you over-correct and start dumping every billing complaint on them. You need both because they trade against each other, and a system tuned to catch every refund request by routing anything mentioning money is not an improvement.

from sklearn.metrics import classification_report, confusion_matrix

labels = ["billing", "technical", "refund"]

print(classification_report(
    y_true, y_pred,
    labels=labels,
    zero_division=0,
    digits=3,
))
print(confusion_matrix(y_true, y_pred, labels=labels))

The second call prints the confusion matrix, which is the grid of true class against predicted class with one cell per pair, and it is where you read which specific substitution the system is making rather than how many it makes. The labels= argument matters more than it looks. Without it, a class the system never predicts and never encounters silently drops out of the report, and a taxonomy that has quietly stopped producing one of its labels is exactly the failure you want the report to surface rather than hide.

Sampling error at twenty examples#

There is a sampling problem hiding underneath the table above. Refund recall of 0.85 came from seventeen correct out of twenty, and twenty is a small number. The 95% Wilson interval, which is the form of confidence interval that stays sensible when the counts are small, runs on 17/20 from about 64% to 95%, which means the honest statement is that refund recall is somewhere between two-thirds and nearly perfect. A prompt change that moves the count from 17 to 19 has not demonstrated anything; it has moved one and a half tickets.

Twenty is small in a way that is easier to feel than to state. Every ticket in a twenty-example class is worth five points of recall by itself, so the measurement can only ever land on multiples of five, and one annotator changing their mind about one ticket moves the reported figure as far as a serious engineering change would. The confidence interval is the formal version of that observation. It does not ask how far the true rate is from the observed one; it asks which true rates could plausibly have produced seventeen right out of twenty. A process that is right two-thirds of the time produces seventeen out of twenty often enough that the data cannot rule it out, and so does one that is right nineteen times in twenty. Both ends stay in the interval because both remain live explanations of what was seen.

The reason to reach for the Wilson form rather than the textbook one is the same smallness. The textbook interval is the observed proportion plus and minus a multiple of its standard error, symmetric by construction, and on 17/20 that symmetry pushes the upper end past certainty — an interval whose top implies more correct tickets than the class contains. Wilson's form is built the other way round, by asking which true rates are consistent with the observation rather than by drawing a symmetric band around it, so it cannot leave the range a proportion can occupy and it leans away from whichever end of the scale the observation sits near. At 0.85 that produces a long tail downward and a short one upward, which is exactly the shape of the interval quoted above.

The fix is to stratify. Build the evaluation set so that each class has enough examples to measure — 150 to 200 per class puts the interval on a 0.85 recall at roughly ±5 points — and then, when you want an aggregate figure, reweight the per-class results by the true traffic shares rather than by the counts in your set. Sampling 200 refund requests out of a quarter's traffic is easy even when they are 4% of it, because a quarter contains a lot of tickets. What you must not do is sample 500 tickets uniformly, find twenty refunds, and then treat the resulting recall as a number worth defending in a design review.

That 150-to-200 band is the interval width run backwards, and the exponent in it is the reason the advice sounds expensive. The half-width of one of these intervals falls with the square root of the number of examples, because what is being estimated is an average over independent draws and the noise in such an average shrinks by the square root of how many draws went into it. Halving an interval therefore costs four times the data, and cutting it to a third costs nine times. Twenty examples multiplied by nine is a hundred and eighty, which is where the middle of that band comes from, and a third of the roughly thirty-point span on 17/20 is the ±5 points quoted. The relationship is unkind in the direction people hope: the second hundred examples buys much less than the first hundred did, and there is no sample size at which a rare class becomes cheap to measure.

The tempting alternative is to keep sampling uniformly and simply sample more, and it is tempting because it sounds like more rigour rather than less — one sampling rule, no per-class bookkeeping, no argument about weights. Run the arithmetic on it. Refunds arrive at four in a hundred, so reaching a hundred and eighty of them by uniform sampling means drawing about four and a half thousand tickets, and every one of those has to be labelled by two annotators at ninety seconds each whether it is a refund or not. Stratifying reaches the same hundred and eighty refunds by changing one clause in a database query. The uniform sample is not more rigorous; it is the same rigour bought at nine times the price, and it spends nearly all of that money re-measuring billing recall to a precision nobody asked for.

class_shares = {"billing": 0.62, "technical": 0.34, "refund": 0.04}

# recall_by_class measured on a stratified set, ~180 examples per class
weighted_accuracy = sum(
    class_shares[c] * recall_by_class[c] for c in class_shares
)

That weighted figure is comparable to a production accuracy number while being built from per-class measurements that each have enough support to be stable. It also makes the sensitivity obvious: change the refund share from 4% to 9% after a product change, and the same per-class numbers produce a different headline.

Correct labels are not representative ones#

A correctly labelled, stratified evaluation set fixes the provenance of the labels and nothing else. If every ticket in it came from one week in December, it encodes December's product incidents and December's promotional refunds, and a classifier tuned against it will look worse in March for reasons that have nothing to do with the classifier. Representativeness is a separate property from label correctness, and it decays: a set built in January is a progressively worse description of April's traffic, which is why the sampling window and the traffic mix belong in the set's own documentation.

Per-class metrics also do not encode the asymmetric cost of error types. Precision and recall treat a refund-routed-to-technical and a technical-routed-to-refund as one miss each, and your business does not. If you want the metric to reflect that, you need an explicit cost matrix, and most teams are better served by reporting the confusion matrix in full and letting the reader see which cell is large.

None of this is about output format. Constraining a model to emit exactly one of three enum values, which is the schema-constrained generation problem, guarantees you get a syntactically valid label back on every call. It says nothing about whether the label is right. A classifier that always returns a well-formed "billing" is perfectly schema-conformant and useless, and evaluations that check parse rates rather than label correctness are measuring the wrong layer.

Once labels are independent and the set is stratified, the score has a property it did not have before: the model can lose. Everything else in this topic depends on that property, because every comparison worth running — against a rule set, against a trained classifier, against a cheaper model — is a comparison on one set under one label policy, and a set that cannot produce a losing score cannot produce a ranking either.

The next chapter runs something deliberately unsophisticated against exactly this set, under exactly this label policy, and takes seriously the objection that always comes back when you propose it.