Per-slice evaluation

A slice is a named predicate over your eval cases: English single-issue tickets, Brazilian Portuguese, messages under twelve words with an image attached. Scoring per slice costs one extra loop over results you already have, and it converts a single accuracy figure into an error analysis — a statement about which kinds of input fail and by how much, rather than about how many failures there are in total.

Every remaining decision in this topic is a decision about a subset of inputs. Which four examples to keep, whether one mislabelled demonstration has done damage, whether a retrieval-based selector is safe to ship: an aggregate score cannot express any of them, and that is not a defect in the number. A single figure standing for four hundred cases is doing what a summary does, which is to discard the differences between the things it summarises — and those differences are the entire content of every question in that list.

The comparison worth holding onto is service latency, and it is an analogy rather than an identity. Nobody runs a production API on a single mean response time, because the mean is a weighted average dominated by the cheap endpoints and it goes green while the checkout path times out. Everyone breaks latency down by endpoint and reads the tail, and the reason is exactly the reason to break accuracy down by slice: the aggregate is an average over a population whose members fail for different reasons and matter to the business by different amounts.

Where the analogy breaks is worth naming before it is leaned on. A latency percentile is one population read at a point in its own distribution — the slowest one request in twenty and the median describe the same requests, sorted. A slice score is a different population altogether, since the caption-only cases and the Portuguese cases are disjoint sets of tickets rather than two readings of one set. So a latency tail moves when the same traffic gets slower, a slice score moves when one particular kind of input starts failing, and only the second of those hands you a group of inputs to go and look at. Both are the part a mean is blind to, for different reasons.

Defining the groups before you need them#

The naming is the part that does the work. Some slices come from properties you already have in the data — language, channel, customer tier, the gold label (the answer a human recorded as correct, which is what the eval scores against), whether the ticket has an attachment, message length above or below some threshold. Others come from reading a sample of failures and writing down the cause of each one in a few words, which is a manual exercise on thirty or forty cases and produces the categories that actually predict failure in your system. Slices defined the first way are cheap and often uninformative; slices defined the second way are where the signal is, and the loop between them is the point. You read errors to discover a group, you turn the group into a predicate, and from then on the predicate is scored automatically on every run.

The mechanics are small enough that there is no reason to defer them:

SLICES = {
    "en_single_issue":  lambda c: c.lang == "en" and c.issue_count == 1,
    "en_multi_issue":   lambda c: c.lang == "en" and c.issue_count > 1,
    "pt_br":            lambda c: c.lang == "pt-BR",
    "caption_only":     lambda c: len(c.body.split()) <= 12 and c.has_image,
    "forwarded_thread": lambda c: c.body.count("-----Original") >= 1,
}

def report(results):
    print(f"{'slice':18} {'n':>4} {'acc':>6} {'errors':>7}")
    print(f"{'ALL':18} {len(results):>4} {acc(results):>6.1f} "
          f"{sum(1 for r in results if not r.ok):>7}")
    for name, pred in SLICES.items():
        rows = [r for r in results if pred(r.case)]
        print(f"{name:18} {len(rows):>4} {acc(rows):>6.1f} "
              f"{sum(1 for r in rows if not r.ok):>7}")

Run against a triage eval that prints accuracy: 89.0 with forty-four cases wrong out of four hundred, it returns this:

slice                 n    acc  errors
ALL                 400   89.0      44
en_single_issue     210   97.1       6
en_multi_issue       54   72.2      15
pt_br                48   79.2      10
caption_only         36   69.4      11
forwarded_thread     30   93.3       2

Every figure in that report is one division, and two of them are worth doing by hand before anything is concluded from the rest. An accuracy over a set is the count of cases the system got right divided by the count of cases in the set. Four hundred cases with forty-four wrong leaves three hundred and fifty-six right, and three hundred and fifty-six over four hundred is the eighty-nine at the top. The same division on the first slice starts from two hundred and ten cases with six wrong, which leaves two hundred and four right, and two hundred and four over two hundred and ten is the ninety-seven point one. The error count and the percentage are the same fact written twice, which is why the report prints both: the count is the thing you can act on, and the percentage is the thing you can set against a slice of a different size.

Nothing in that report is new information — it is the same run — and it changes what the team argues about. The service does not have an 89% problem. It has three specific problems, each with a name, each affecting a group of inputs that a product manager can recognise, and each with an error count that says how much of the total it accounts for: fifteen failures on tickets mentioning two separate problems, ten in Brazilian Portuguese, eleven on one-line messages that are just a caption on a screenshot, and the remaining eight scattered across everything else. The 210 clean English single-issue tickets, which are 53% of the set and most of what anyone looks at when they spot-check outputs, contribute 6 of the 44 errors.

The accounting behind that sentence is worth doing rather than reading past. Fifteen multi-issue failures, ten Portuguese failures and eleven caption-only failures come to thirty-six; forty-four less thirty-six leaves eight, and those eight are exactly the six in the English single-issue slice plus the two in forwarded threads. Two hundred and ten cases out of four hundred is a little over half the set, which is the fifty-three per cent above. Put the two together and the shape of the problem states itself: the slice holding half the set holds roughly one error in seven, and three slices holding about a third of the set between them hold four fifths of the errors.

Two constraints make the report trustworthy. Slices must be defined before the run rather than discovered in it, because a group carved out of a run's failures will always look bad in that run and will regress to the mean in the next one. And each slice needs enough cases to have a meaningful score of its own; a slice of twelve cases moves 8.3 points every time one case flips, so it can only ever tell you about catastrophes. Slices may freely overlap — a Portuguese multi-issue ticket belongs to both — and the columns will not sum to the total, which is fine and is not a bug to fix.

The first of those constraints has a mechanism under it, and "regress to the mean" is the name of the effect rather than the cause. A group assembled by reading a run's failures is selected for containing failures; that is the criterion it was built on. So its score in that run is not an estimate of how the system handles that kind of input, it is the definition of the group read back out, and a definition cannot be wrong about itself. The next run scores the same predicate over the same cases with a fresh set of sampled draws, and no draw in it knows which cases were chosen or why, so that run returns an estimate for the first time — a higher one, since the criterion is no longer doing any work. A predicate written down before any run has no such history behind it, which is why its first score means the same thing its tenth does.

The second constraint is arithmetic on the slice's size. One case out of twelve is one twelfth, and one twelfth expressed as a percentage is the eight point three figure above, so a twelve-case slice has no value it can report between that number and the next one up. The measurement is not wrong; the instrument's resolution is simply coarser than most of the changes you will want it to detect. The same division sets the floor for every row of the report: a thirty-case slice moves by a little over three points whenever one case flips, and the two-hundred-and-ten-case slice at the top moves by under half a point, which is why the n column is worth reading before any of the scores beside it.

The manual pass that produces the predicates is worth describing precisely, because teams tend to imagine it as a large project and then never do it. Take the failures from one run, cap the sample at forty, and for each one write a single line saying what went wrong — the model's label, the gold label, and a guess at the cause in a handful of words. Twenty minutes of that on the run above produces notes of the form multi-issue, billing+delivery, chose delivery, then pt-BR, "cobrança" read as delivery, then caption only, 7 words, guessed product_quality, and forwarded thread, labelled the original not the reply. Sort those lines and the clusters are visible without any analysis, because the same phrase keeps recurring; the ones that recur four or more times become slices, and the singletons go in a file to look at next month.

Two things make the pass more useful than it sounds. Writing the cause down forces a hypothesis about mechanism, which is what distinguishes a slice that predicts failure from a slice that merely describes the data. caption_only qualifies because the reason short inputs are hard is checkable rather than asserted: the model is scoring a continuation conditioned on the text in front of it, a six-word caption supplies almost none of that text, and what is left to condition on is the instructions and the examples. The label then comes out of whatever prior those two supply rather than out of the ticket, which is a claim you can test by changing the examples and watching that slice alone. tickets received on Tuesday is a partition with nothing behind it, and the way to tell is that no such sentence can be written for it. And doing the pass by hand, on real inputs, is the only step in this whole topic where you look at what your users actually send, which routinely turns up things no metric would have surfaced: a customer whose integration wraps every message in a quoted email header, or a template from the mobile app that puts the useful text after 200 tokens of device metadata.

Reading a change through the slices#

The value of the instrument shows up on the next change, not on the first report. A change is proposed, the eval is re-run, and the aggregate moves. What you want from the report is the same six rows, before and after, with the error counts alongside the percentages, because on a small slice the count is the honest unit and the percentage is a derived figure that overstates its own precision.

The arithmetic connecting those rows to the top line is worth stating once, because every later argument in this topic leans on it. An accuracy figure over a set is the number of correct cases divided by the number of cases. Cut the set into groups that do not overlap and that between them hold every case, and the number of correct cases is the sum of the correct cases in each group, because a sum may be taken group by group before it is taken as a whole. Divide through by the size of the set and each group's contribution comes out as its own accuracy multiplied by the share of the set it holds. The aggregate is not merely summarised by the slices; it is assembled out of them, and the weight on each one is its share.

The slices in a real report overlap and do not quite cover the set, so that identity is an approximation rather than an equality. A Portuguese multi-issue ticket is counted in two rows, and a case matching no predicate is counted in none. The approximation stays close enough to work with while the overlaps are small and the uncovered cases are not where the movement is, and it stops being usable when either of those fails, which is worth checking before quoting it at anyone.

Two consequences follow from the identity, and between them they are the reason the mean and the complaint queue disagree. A slice's power to move the aggregate is its share of the eval set, so a large slice moving a little outweighs a small slice moving a lot. A slice's power to generate escalations is its share of live traffic multiplied by what being wrong on it costs, which is a different quantity, and neither of those two factors is present in the eval set at all. A change that trades a small slice for a large one is arithmetically an improvement and can be commercially a loss, and no rearrangement of the aggregate will tell you which of the two you have.

Read the error counts and the percentages together, and read both for direction before magnitude. An aggregate gain is an average, so it is arithmetically ordinary for it to contain a slice moving the other way, and the question is never whether the mean improved — you already know it did, or the change would not be in front of you. The question is which rows moved down, whether any of them was already the worst row, and whether the movement has a mechanism behind it or is the wobble a thirty-case slice produces on its own. Those three questions are answerable from the table; none of them is answerable from the number at the top of it.

That gives a usable rule for what you need to see per slice before shipping a selection change: every slice's score, alongside the aggregate; no slice regressing beyond its own noise floor, with the floor stated rather than assumed; the slices most likely to break for a known mechanism sized large enough to detect a regression you would care about; and a named owner for the decision when a change genuinely helps most inputs and hurts some. Not every regression is a blocker, and a team that treats every slice as a veto will never ship anything, so the useful distinction is between guardrail slices, where any regression stops the change, and informational slices, where a regression is recorded and weighed. Which slices are guardrails is a product decision, and making it explicitly beats making it accidentally by choosing what to look at.

Slices are a reporting change, not a fix#

Cutting the set into groups makes nothing better on its own. The forty-four errors are the same forty-four errors after slicing as before. What has changed is that they are now attached to hypotheses you can act on, which matters because the actions available in this topic are all targeted — spend the example budget here, check this slice before shipping, build a fallback for that input shape.

Slicing also does not rescue a small eval set. If your held-out set is 120 cases, cutting it five ways leaves slices of twenty-odd cases whose scores swing by five points on a single flip — the same one-case-over-slice-size division as before, with a smaller number underneath it — and the report will manufacture regressions and improvements every night. Cutting a set into groups creates no information the set did not already hold, so slicing a small one finely returns more numbers of less worth, each moving further on less evidence. The answer there is more labelled data before more slices, and it is usually cheaper than people expect, because slices can be enlarged selectively — you do not need another 400 uniformly sampled tickets, you need another 200 caption-only ones, which a query against production traffic will find in a few minutes.

There is a statistical trap in the other direction too. Watching twelve slices on every nightly run means twelve chances per run for something to look like a regression by accident, and a team that investigates every red cell will spend all of its time chasing noise. The way out is not fewer slices; it is stating each slice's noise floor next to its score, so a 3-point move on a slice with a 6-point floor renders as unremarkable rather than as an alarm.

The arithmetic under that is ordinary and worth spelling out, because the instinct on being shown it is to cut the number of slices. Each slice is a separate opportunity for variation that nobody caused — the sampler's draws, and which cases happened to land in the eval set — to produce a move that reads as a signal. Opportunities add: if one slice throws a misleading-looking move on one night in twenty, twelve slices throw one every couple of nights, and nothing has gone wrong on any of those nights. Reporting twelve numbers means reading twelve numbers, and a reader with no floor under each of them will find the largest and treat it as the finding.

Cutting to four slices does reduce the false alarms. It reduces them by discarding the resolution the report exists to supply, which is the wrong end of the trade to give up. Stating the floor costs one extra column and leaves every slice in place: a three-point move on a slice whose own one-flipped-case resolution is six points has said nothing, and the column says so without anybody having to remember how large the slice was.

Finally, per-slice scoring is not a fairness or bias audit, even when the slices are demographic. Equal accuracy across language slices says nothing about whether the errors have equal consequences, and a slice-level metric that looks even can hide a systematic direction in the errors — every Portuguese misroute going to the same wrong queue, for instance. That pattern, where the interesting quantity is not how many errors a slice has but where they all point, is the subject of a later chapter on what a single mislabelled demonstration does.

What the instrument buys, then, is leads. One number tells you that something is wrong and nothing about what; six numbers with names on them tell you that multi-issue tickets, Portuguese and screenshot captions account for 36 of your 44 failures, which is a list of three things to work on and a way to check afterwards whether the work landed. The cost is a dictionary of predicates and one extra loop, and the discipline is writing the predicates down before the run rather than after it.

The report now says which inputs fail and by how much. The example block, meanwhile, is still whatever someone pasted in when the prompt was written, and there is no reason to expect a set assembled that way to have anything to do with the three groups the report just named.