Evaluating a block deletion

Deleting a block from a prompt is the one selection decision that looks free. The tokens come off every request, the block is usually old, and the obvious experiment — run the eval with it and without it — is cheap enough to do in an afternoon. The trap is in what that experiment reports: a single aggregate over a case set assembled to look like production, compared against itself.

That comparison is structurally blind to the reason most blocks exist. Nobody adds worked examples for the cases the model already gets right, so a block's effect is concentrated in a population that the aggregate is mostly not made of. The figures below come from a ticket-triage classifier with twelve classes, carrying six few-shot examples at 2,200 tokens on every request, scored on a frozen 400-case eval.

Where the 0.91 came from#

Macro-F1 averages the per-class F1 scores with equal weight, where a class's F1 is the harmonic mean of its precision — the share of tickets the model put in that class which belong there — and its recall, the share of tickets that belong there which the model actually put there. Averaging the twelve classes with equal weight means a class with nine cases in the eval set counts exactly as much as a class with ninety-six. That property is normally the reason to use it — it stops a dominant class from hiding everything else. It also means one class can fall a long way and be offset by eleven classes rising a little, and "a little" here is under half a point of F1 each.

The two macro figures are not close. They are identical to four decimal places, because the eleven gains sum to exactly +0.36 and the one loss is exactly −0.36. Nothing engineered that; it is what happens when a small effect spread over eleven classes meets a large effect concentrated in one, and it is common enough that seeing equal aggregates should make you suspicious rather than reassured.

That cancellation is worth adding up rather than accepting, because the addition is what shows how ordinary it is. Read the delta column: four classes gain 0.03, five gain 0.04, and two gain 0.02. In hundredths that is four threes, five fours and two twos, which come to thirty-six hundredths — the +0.36 quoted above. The chargeback class loses 0.36. Macro-F1 is the sum of the twelve per-class scores divided by twelve, and the divisor is the same in both arms, so two columns whose totals agree must produce means that agree. The identical figure is not a coincidence about this data. It is what a sum of zero looks like after it has been divided by a constant.

The size of the individual gains is what makes the cancellation easy to reach. Eleven classes each moving by two to four hundredths is barely visible in any one row, and eleven of anything small is not small. One class had to move by more than a third of its F1 to be offset by it, which sounds like an extreme requirement and is exactly the requirement a rare, awkward class can meet: with nine cases in the set, a class's F1 moves in coarse steps, so it does not have the option of losing a hundredth.

On the class that collapsed, the underlying counts are: recall 7 of 9 with the block, 3 of 9 without it; precision 0.92 with, 0.88 without. Two of the four newly-missed chargeback tickets were labelled billing.refund_request and two billing.invoice_query, which is exactly the confusion the few-shot examples were added to resolve — and that is the second half of the argument. The category that gets an example written for it is by construction a rare, awkward one, so a deletion evaluated on an aggregate is being evaluated on precisely the population the block was never about.

Those counts are where the 0.84 and the 0.48 in the table come from, and it is worth seeing the whole path from tickets to score. Recall is the share of tickets that belong to the class which the model actually put there, so seven of nine is recall of about 0.78 and three of nine is about 0.33. F1 is the harmonic mean of precision and recall — the mean that a single low term drags down, which is the reason it is used here — so with precision hardly moving, 0.92 against 0.88, the F1 tracks the recall collapse and falls from 0.84 to 0.48. Four is the difference between seven and three, which is why exactly four tickets have to be accounted for, and both of the classes they went to are billing classes that also appear among the eleven that improved. The same six examples were separating chargebacks from refunds and invoices, and removing them let those two classes absorb the tickets that had been held apart from them.

The gains are real too, and worth naming rather than dismissing. Six examples all drawn from billing scenarios pull the model's priors toward billing; removing them lets the eleven other classes settle, which is where the +0.02 to +0.04 comes from. The block was buying one class and taxing eleven.

The paired test does not save you either#

The reflex once someone sees the per-class table is to reach for a significance test on overall correctness. It is worth running the numbers, because the test agrees with the naive reading.

from statsmodels.stats.contingency_tables import mcnemar

# per-case correctness on the same 400 cases, same model, same settings
both_right      = 355
with_only       = 14   # right with the block, wrong without it
without_only    = 12   # wrong with the block, right without it
both_wrong      = 19

table = [[both_right, with_only], [without_only, both_wrong]]
print(mcnemar(table, exact=True).pvalue)   # 0.845

Accuracy is 0.923 with the block and 0.918 without it, twenty-six cases changed their label, and McNemar's exact test on the discordant pairs returns p = 0.845. There is no detectable difference in overall correctness, and there was never going to be: the test pools every class into one binary, so four chargeback tickets flipping the wrong way sit in the same bucket as eleven classes' worth of small improvements flipping the right way. A more rigorous aggregate is still an aggregate.

The four counts in that table are the whole eval set sorted into the only four things a case can do across two arms, and they add to 400, which is the first thing to check. A case correct in the with-block arm is either in both_right or in with_only, so those two together are the with-block correct count, and dividing by 400 gives the 0.923. The same addition on both_right and without_only gives 0.918. The two accuracies differ by half a point because the two off-diagonal cells differ by two cases, and those two cells are everything the test has to work with: both_right and both_wrong are cases where the arms agreed, and a case where both arms did the same thing carries no information about which arm is better. That is what "discordant pairs" means, and it is why the test is right to ignore the cases where nothing moved.

McNemar is the reflex here because it is the correct test for exactly this design, which is what makes it dangerous. It is paired, it does not assume the two arms are independent samples, and it is the textbook answer to "did this change help, on the same cases". All of that is true and none of it touches the problem, because a test answers the question its statistic encodes, and this statistic encodes one bit per case. Fourteen cases moved one way and twelve the other; on a fair coin a split that even is unremarkable, and p = 0.845 says so accurately. What the statistic cannot represent is that the twelve are spread across eleven classes while the four that matter are concentrated in one, because by the time a case has become a bit, its class is gone. Rigour improves the answer to the question you asked. It does not change the question.

Blinding is a separate axis and is also not the answer to this. Blinding the reviewer to which arm produced which output controls for the reviewer preferring the output they expect to be better; it does nothing about a metric that averages over the population where the difference lives. You want both — blinded scoring so the labels are trustworthy, per-case comparison so the aggregation does not erase what the labels found.

The comparison worth running#

What the evidence has to look like is a joined, per-case diff between the two arms on a set that was frozen before either arm ran, with the model, settings and judge identical. The script is short; what makes it useful is that its primary output is not a number.

import pandas as pd

with_block    = pd.read_json("eval_with_block.jsonl",    lines=True)
without_block = pd.read_json("eval_without_block.jsonl", lines=True)

df = with_block.merge(
    without_block, on="case_id", suffixes=("_with", "_without")
)
df["correct_with"]    = df.predicted_with    == df.gold
df["correct_without"] = df.predicted_without == df.gold
df["flipped"]         = df.correct_with != df.correct_without

# 1. Which classes moved, and by how much, weighted by nothing.
per_class = df.groupby("gold").agg(
    n=("case_id", "size"),
    recall_with=("correct_with", "mean"),
    recall_without=("correct_without", "mean"),
)
per_class["delta"] = per_class.recall_without - per_class.recall_with

# 2. The cases the deletion broke, printed in full for a human to read.
regressions = df[df.correct_with & ~df.correct_without]
regressions[["case_id", "gold", "predicted_without", "text"]].to_csv(
    "regressions.csv", index=False
)

print(per_class.sort_values("delta").head())
print(f"{len(regressions)} regressions, {df.flipped.sum()} total flips")

The regressions.csv is the deliverable. Fourteen rows is a fifteen-minute read, and reading them is what turns "a rare class lost recall" into "four chargeback disputes were routed to the refunds queue, which has no 20-day regulatory clock on it." A per-class table tells you where to look. Only the cases tell you what it costs.

Fourteen is not a new measurement. The regressions filter keeps cases that were correct with the block and wrong without it, which is the same population as the with_only cell of the contingency table above, and the printed count of flips is that cell plus its opposite. The same two numbers that produced an uninformative p-value produce a fourteen-row file, and the file is the useful artefact. That is the practical form of the whole argument: the test and the listing are computed from identical inputs, and one of them discards the case text on the way to a scalar while the other does not.

Nine cases is not enough to decide on#

Here is the part that a clean story would leave out. The chargeback class has nine cases in the frozen set. Seven correct versus three correct on nine trials is, by Fisher's exact test, p ≈ 0.15 — not significant by any conventional threshold. The per-class table found a signal that the aggregate hid, and the signal is itself too small to confirm.

The reason nine cases cannot settle it is worth having as intuition rather than as a p-value. With nine trials the only recall values available are the tenths of a nine-step ladder, so a single ticket moving is already a large change in the measured rate, and a gap of four tickets is only two tickets' worth of coincidence away from being a gap of eight or of nothing. Fisher's exact test is asking precisely that question — how often would a split this lopsided appear by chance if the block made no difference — and roughly one time in seven is its answer. One time in seven is common enough to happen in an eval you run every week.

The wrong response is to conclude that the drop is noise and merge. The right one is to notice that the eval set was never designed to measure this class, and to fix that before deciding: pull 60 more chargeback tickets from the last two quarters of production traffic, label them, add them to the frozen set as a named slice, and re-run both arms. Sixty cases distinguishes 78% recall from 33% recall decisively. It also gives you something durable, because the slice stays in the eval set and every future change to this prompt gets measured against it.

The general form of the rule is that your eval set has to contain enough of each population you care about to detect the size of regression you care about in that population, and a set assembled by sampling production traffic never does, because production traffic is by definition mostly the common cases.

The enlarged slice makes the drop unambiguous — recall 0.78 with the block against 0.33 without, on 69 cases — and it also shows that the same masking happens one level further down. Splitting those 69 by the channel the ticket arrived on gives two populations that behave nothing alike.

Email tickets carry quoted threads, forwarded bank correspondence and signature blocks, so the sentence that identifies a chargeback sits several hundred tokens into a noisy document; the few-shot examples were doing most of their work there, and removing them costs getting on for three times as much on email as on the web form. The class-level −0.45 is the average of two effects that would lead to different decisions: if 90% of chargeback volume arrived through the web form, the block would be close to unnecessary, and the class-level number would still read −0.45.

The word "average" in that sentence is doing precise work and the table lets you check it. The class-level recall is not the mean of 0.80 and 0.75; it is those two weighted by how many cases each channel contributed. Forty-one cases at 0.80 and twenty-eight at 0.75 come to about fifty-four tickets classified correctly out of sixty-nine, which is the 0.78 in the bottom row. Repeat it on the without-block column — forty-one at 0.20 and twenty-eight at 0.52 — and the correct count is about twenty-three of sixty-nine, which is the 0.33. The class-level figure is therefore a fact about the traffic mix as much as about the block, and that is what makes the hypothetical in the paragraph above real rather than rhetorical. Change the mix and the same two channel effects produce a different class-level number with nothing about the model or the prompt having changed at all.

Four arms, not two#

Suppose the enlarged slice confirms the drop. The conclusion is still not necessarily "keep the block" — 2,200 tokens to buy 45 points of recall on 0.4% of traffic is a bad trade if there is a cheaper way to buy the same recall, and there usually is. Two examples of the chargeback case specifically, at 700 tokens, recovered recall to 6 of 9 in a follow-up arm. A single line in the system prompt naming the distinguishing signal — the ticket mentions the card issuer rather than the merchant — recovered 5 of 9 for 30 tokens.

The two middle rows beat both endpoints on the aggregate, which is the ordinary result once you stop treating the choice as binary and is invisible to an experiment that only runs the block and its absence.

That an interior arm wins is not luck, and reading the table's two right-hand columns against each other shows why it was predictable. The eleven-class mean recall rises down every row: 0.913, 0.941, 0.946, 0.949, improving steadily as the billing-flavoured prefix gets smaller and stops pulling the model's priors towards billing. Chargeback recall falls down every row, from seven of nine to three of nine. Macro-F1 is a single number built from both movements, so it is the sum of a quantity that increases and a quantity that decreases as the prefix shrinks — and a sum of that shape has its maximum somewhere in the middle unless one of the two terms happens to dominate everywhere. The binary experiment sampled the two ends and reported, correctly, that they are equal. Equal endpoints on a curve with an interior maximum is the one result that guarantees something better lies between them. The trade also stops being about tokens at that point: 700 tokens against 30 for one further chargeback case is a question about how much a missed dispute costs, and the regressions file says what that is — a ticket routed to a queue with no regulatory clock. Whichever row wins, the losing rows are worth writing into the decision record, because the next person to look at this prompt will otherwise re-run the same four arms.

Slices have to be chosen before the results#

The recursion from class to channel has no natural stopping point, and chasing it indefinitely is how a two-day experiment becomes a quarter. Every partition is another average, and any average can be hiding a population that behaves the opposite way, so a reviewer who wants a deletion stopped can always ask for one more cut.

The workable version is to decide in advance which partitions you must be able to see — for this classifier, the label and the arrival channel, because both change what happens to the ticket downstream — and to register them as named slices on the frozen set with a minimum case count each. Every subsequent change to this prompt then reports against the same partitions, and no one has to argue about which cut is legitimate after seeing the results, which is when that argument is least likely to be settled honestly.

Pre-registration also fixes the sampling problem in the only place it can be fixed. A slice with a minimum case count is a standing instruction to go and find cases when production does not supply them, which is the opposite of what sampling proportionally does. The 69 chargeback cases exist because someone decided the class had to be visible; they do not exist because the traffic produced them.

Equal averages, then, are a question rather than an answer. The per-case diff is what turns them into evidence, the named slice is what makes the diff big enough to trust, and the four-arm comparison is what stops the decision being a choice between the whole block and nothing. All three are cheap next to the deletion they are gating, and none of them is what an eval reports by default.

All of this treats the 2,200 tokens as a fixed cost worth an experiment to remove. Before running that experiment on a block, it is worth knowing what the block is actually being charged. The few-shot examples sit inside a stable system prompt that is identical on every request, and identical prefixes are not billed the way per-request tokens are. A block that looks like 2,200 tokens on a token counter can be costing a small fraction of that, or — if you move it in the wrong direction — considerably more than it appears to.