Choosing which failures to read

A nominated pool — the candidates a complaint, an escalation, a tool error or a low judge score flagged as possibly bad — holds more conversations than anyone will read. Reading fifty out of four thousand makes the read a sample of a sample, and every step that reduces the four thousand to the fifty is a chance to import somebody's theory about what matters. Sorting by recency reads whatever happened last week. Sorting by judge score reads the failures the judge is most confident about, which are the ones it was written to catch. Filtering to the conversations with most turns reads the hardest cases and calls them typical.

Those three are usually listed as bad habits, and underneath they are one mechanism. Each gives some conversations a chance of being read equal to one and every other conversation a chance of zero, and each decides which is which using a property — how recent, how confidently flagged, how long — that is itself related to the failures about to be counted. A count from such a draw is an accurate count over the conversations that property selected. Nothing downstream can convert it into a count over the pool, because converting requires knowing the chance each case had of being drawn and a sort order never records one. Any ordering that is not random inside a defined group answers a question you did not ask.

This chapter is about the draw and about the discipline of reading what it gave you. Two figures hold throughout: a pool of about four thousand nominated candidates from one month, and a budget of fifty traces at four to five minutes each.

Stratify on what changes the answer#

A stratum is one of the groups the pool is cut into before any drawing happens, each carrying a quota of the fifty fixed in advance, and stratified sampling means drawing at random inside each group rather than at random across the pool as a whole. A simple random draw is not wrong on average: repeated many times, a draw of fifty gives each intent its share of the pool. That is precisely the guarantee which does not help here, because the draw happens once and the whole analysis is built on the one sample that came out.

What matters for a single draw is how far it can land from the pool's composition, and for a small group that distance is large in relative terms. Promotions and pricing is a bit over four per cent of a pool of about four thousand, so an unstratified fifty holds about two promotions cases on average, and a draw returning zero or one of them is entirely ordinary. Fixing the quota in advance removes that variation for the dimension you fixed on, since the number of promotions cases drawn stops being a random quantity at all. The randomness is pushed inside the group, where it does the job you actually want it doing: deciding which promotions cases you read rather than how many.

The two dimensions worth stratifying on for a routine read are intent and time. Intent, because a product with six intents has six different failure profiles and an unstratified draw gives you the failures of whichever intent happens to be highest volume. Time, because deployments, promotions, seasonality and upstream data changes all make a given week unrepresentative, and a month of pool spread over four weeks lets you see whether a mode is constant or arrived on a date.

Time earns its place for a second reason that only shows up afterwards. Restricting the pool to one week is a filter, and a filter destroys the comparison: a mode produced entirely by that week's promotion looks identical in the counts to a mode running steadily all month. A quota in every week keeps all four in the sample, so a mode concentrated in one appears as concentration rather than as prevalence. That distinction decides what gets worked on, since a mode which arrived on a Tuesday usually has a change behind it.

Proportional allocation gives each stratum a share of the fifty equal to its share of the pool, which sounds right and quietly guarantees that small intents contribute nothing. If refunds are 4% of the pool, proportional allocation gives them two cases, which cannot distinguish a mode from an accident. A floor of three to five per stratum, taken out of the largest strata, buys enough signal in the tails to notice a mode at the cost of some precision on the head. The consequence is that the sample is no longer self-weighting, meaning its intent proportions no longer match the pool's, so counts from it need reweighting before they become rates — which is where chapter 5 starts, dividing each stratum's share of the pool by its share of the sample to get a weight per case.

The fourth column is arithmetic rather than output, and it is worth doing one row of it by hand. Order status is forty-one per cent of the pool, and forty-one per cent of fifty is twenty and a half, which rounds to the twenty-one shown. Promotions and pricing is a shade over four per cent, and four per cent of fifty is two. Add the fourth column down and it comes to fifty-two rather than fifty, because rounding each stratum independently does not preserve a total, and the overshoot has to be taken back out of somewhere before anybody reads anything.

A floor deliberately distorts the sample, and a deliberate distortion needs a justification rather than an assurance. The justification is that this particular distortion is known and invertible. Every case inside a stratum was drawn uniformly at random, so every promotions case had the same chance of being drawn, and that chance is the stratum's quota divided by the number of promotions candidates in the pool. Knowing that chance is exactly what lets a count be converted back into a statement about the pool.

Doing the conversion on this table takes one division per stratum. Promotions holds four of the fifty read against a shade over four per cent of the pool, so it is represented at roughly twice its true weight, and dividing its share of the pool by its share of the sample gives a factor of about a half — each promotions case counts as about half a case when rates are computed. Order status, cut from twenty-one to seventeen, holds about a third of the sample against forty-one per cent of the pool, so each of its cases counts for about a fifth more than one.

Set that against the recency sort, because the comparison is the whole argument for choosing a sample this way. Both draws distort the pool. The stratified draw distorts it by a factor that can be written down before the reading starts, since the design is what fixed the factor. The recency sort distorts it by an amount nobody can compute afterwards, because the chance a conversation had of being read depended on where it fell relative to a cut-off, and nothing records the composition of the part never reached. An unknown distortion cannot be undone. A designed one can, and that is what buys the right to over-sample a small stratum at all.

The floor is not free, and its price is visible in the same table. The four cases taken out of order status are precision spent: every mode counted inside order status is now measured on fewer traces, and the interval around each of those counts widens accordingly. What that buys is the ability to see a promotions mode at all rather than to see it accurately. A stratum at two cases cannot separate a mode from an accident under any amount of care, so the floor decides which question gets answered badly, in favour of answering the small-stratum question badly rather than not at all.

Within each intent, split the allocation across the four weeks and draw uniformly at random inside each cell, a cell being one intent in one week and the smallest group this design controls. Order status at seventeen across four weeks comes out as five in the first week and four in each of the others; promotions at four comes out as one a week. Where the leftover case goes is arbitrary and harmless provided the rule is written down rather than decided afresh each run. Thirty lines of code does the whole thing, and writing it down beats doing it by hand because next quarter you want the same procedure rather than a similar one.

import random
from collections import defaultdict

def stratified_sample(pool, n=50, floor=4, seed=20260731):
    """pool: list of dicts with 'id', 'intent', 'week'. Returns n conversation ids."""
    rng = random.Random(seed)
    by_intent = defaultdict(list)
    for c in pool:
        by_intent[c["intent"]].append(c)

    total = len(pool)
    quota = {k: max(floor, round(n * len(v) / total)) for k, v in by_intent.items()}

    # take the overshoot back out of the largest strata, largest first
    while sum(quota.values()) > n:
        biggest = max(quota, key=lambda k: quota[k])
        quota[biggest] -= 1

    drawn = []
    for intent, cases in by_intent.items():
        by_week = defaultdict(list)
        for c in cases:
            by_week[c["week"]].append(c)
        weeks = sorted(by_week)
        per_week = [quota[intent] // len(weeks)] * len(weeks)
        for i in range(quota[intent] % len(weeks)):
            per_week[i] += 1
        for w, k in zip(weeks, per_week):
            drawn += rng.sample(by_week[w], min(k, len(by_week[w])))

    rng.shuffle(drawn)          # read in random order, not intent order
    return [c["id"] for c in drawn]

Two things about that function repay printing quota before anything is drawn. The floor applies to every key in by_intent, including the residual "everything else" group, where it buys nothing: a bucket defined by not being any of the named intents is not a mode and cannot become one by being read more often. The overshoot loop is the second. It removes one case from whichever stratum is largest at that moment, so where one stratum leads clearly, every unit of overshoot comes out of that single stratum until it stops leading — a different allocation from the one in the table above, which spreads the reduction across the large strata. Both are defensible, and the choice between them should be made rather than inherited from a loop.

The shuffle at the end is not decoration. Reading all seventeen order-status cases in a row builds a mental template that then gets applied to the returns cases, and the labels drift toward whatever you saw first. That drift has a direction rather than being noise: matching a case to a category that already exists is cheaper than deciding it needs a new one, so an early template absorbs later cases instead of being contradicted by them. Interleaving the intents costs nothing and keeps each case being judged on its own.

Fixing the seed matters for a different reason. It makes the sample reproducible, so a colleague can run the same function with the same seed, get the same fifty conversation ids, and code a share of them without having sat through your reading. The two codings can then be compared, which is the check on whether the definitions travel at all (chapter 4, where a second reader applying the written codebook to the same traces is what makes a mode countable rather than merely named). Without the seed the two people are coding different cases, and any disagreement between them cannot be told apart from the difference between two samples.

After a release, the strata are different#

A read triggered by a regression is a different problem from characterising a system, and the difference is worth being precise about because the natural approach is wrong in a specific way. The routine draw above exists to estimate: its counts get turned into rates over real traffic, which is why its quotas are fixed against the pool and why its distortion has to be invertible. A post-release read exists to explain one movement in one number. Nothing it produces gets projected onto a month of conversations, so its strata can be chosen for how informative each group is rather than for how large it is, and none of the design weights above apply to it.

Take the case from chapter 1, where a results grid showed a 200-case set falling from 0.86 to 0.79 after a release, so 28 failing before and 42 failing now. Fourteen per cent of two hundred is twenty-eight and twenty-one per cent is forty-two, and the gap between those two counts is the 14 that the dashboard reports. The obvious move is to read the 14 new failures. There are no 14 new failures. Comparing the two runs case by case typically shows something like 22 cases that went pass to fail and 8 that went fail to pass, netting the 14, and the eight that started passing are as informative as the twenty-two, because a fix that trades one behaviour for another shows up there first.

The fourteen is a net, and a net is what survives two counts moving in opposite directions. Thirty cases changed state to produce it, so more than half the movement cancelled inside the subtraction and never reached the pass rate. That is a general property of an aggregate rather than a quirk of this release: it reports a difference and discards the gross movement behind it, and the gross movement is where the reading has to happen.

Worse, some of those thirty flips are not the release at all. A generator sampling at temperature 0.7 flips cases in both directions when you rerun the identical build; on a set like this, 3–6% of cases flipping each way is ordinary, which is 6 to 12 cases per direction from noise alone. Set that beside the twenty-two and the eight, and the eight sits inside the range that an unchanged system would produce on its own. Rerunning the old build twice before comparing anything is an hour of compute that tells you how much of the movement is real.

Better still, run each build three times and work with each case's failure frequency — 0/3, 1/3, 2/3, 3/3 — so that "it flipped" becomes a magnitude rather than a coin toss. The reason that helps is multiplicative. A case flips by chance with some small probability on any single run, and for a case to read 0/3 on the old build and 3/3 on the new one, chance has to fall the same way six separate times over. That small probability multiplied by itself six times leaves almost nothing, so a case in that stratum is very unlikely to be noise. A case reading 1/3 against 2/3 needs chance to cooperate far less, which is why it belongs in a different stratum and gets read at a different rate.

With two hours to spend, you get roughly 25 traces at four to five minutes each. Two hours at five minutes a trace is a little under twenty-five and at four minutes a little over. Spend them across four strata rather than one.

Both numeric columns are worth adding up. The available column sums to the full two hundred, so the five strata partition the set rather than sampling from it, and the read column sums to the twenty-five the budget allows. The frequency strata come from the three-run data and do not line up one for one with the pass-to-fail counts above: a case at one in three before and two in three after can show as either flip, or as no flip, depending which runs a single comparison used.

That allocation is deliberate about the third row. A case that failed before the release for a retrieval reason and now fails for a completely different reason still shows as "failed, failed" in every comparison view, and no automated signal will surface it. The only way it gets found is by someone reading a handful of the stably failing cases and noticing that the cause has changed.

What this allocation cannot do is produce a rate. Ten of the fourteen sharply-risen cases against two of the hundred and forty-two stable passes over-represents the first group by a factor nobody is going to invert, since the design was chosen for what each group explains rather than for what it weighs. A number that has to come out of a post-release read — how much of the drop one mode accounts for — comes from re-coding a properly drawn sample rather than from the counts in this table.

Finish the fifty#

Twenty-five cases into a read, a pattern will appear. Eight of the twenty-five will share an obvious cause, the one-line prompt change that fixes them will be visible, and stopping to ship it will feel like the responsible use of the remaining two hours. Finish the fifty first.

The belief that makes stopping feel safe is that reading more traces is what makes an estimate better, so stopping early spends only a little of something there was plenty of. Sample size and sample design fix different errors, and neither one substitutes for the other. Reading more narrows the interval around a count, and it narrows it slowly, because the width shrinks with the square root of the number of cases and halving it therefore costs four times the reading. Reading more does nothing at all to a frame drawn from the wrong four days, since that error lives in what the sample is a sample of, and it is the same size at fifty cases as at five hundred.

The argument for finishing is about counts rather than about discipline. At case twenty-five you know that a mode appears eight times in twenty-five, which is consistent with anything from a fifth to a half of the full sample; the remaining twenty-five cases are what turn that into a number comparable against the other modes. Since the decision this is heading toward is which mode to fix, and that decision is a comparison between modes, a count you stopped collecting is a comparison you cannot make. The mode that eventually matters most may not have appeared yet — a mode at 6% of failures has a reasonable chance of showing up zero times in the first twenty-five cases and three times in the full fifty. At that rate the expected count in twenty-five cases sits between one and two, and the chance of seeing none of it at all is around one in five, which is an ordinary outcome rather than an unlucky one.

Stopping at twenty-five carries a second cost that has nothing to do with width. The decision to stop was triggered by the mode looking large, which makes the moment of stopping correlated with the quantity being measured. Of all the moments the read could have been abandoned, the one chosen is a moment when the mode happened to be running ahead of its true rate. The count carried into the meeting is inflated by the same thing that made it seem worth carrying, and no interval computed afterwards has any way of knowing that.

There is a second effect that is easy to miss. Once you have named a mode and become attached to it, ambiguous cases start getting coded into it, and the step in between is worth spelling out, because it is a mechanism rather than a lapse of character. An ambiguous case is one that two definitions both fit. Which of the two gets applied is settled by which one comes to mind, and a mode named an hour ago and hunted for ever since comes to mind first. Cases twenty-six to fifty read after you have decided the answer are therefore coded differently from the same cases read blind, the direction of the bias is always toward confirming what you already found, and the count of that mode rises for a reason that has nothing to do with the traffic. Reading all fifty before committing to any interpretation is partly about the counts and partly about not contaminating the second half.

The practical accommodation is a second file: keep candidate-fixes.md open beside the notes and write the one-line change into it the moment you think of it, with the case ids that suggested it. It will still be there in ninety minutes, it will be better informed, and writing it down is usually enough to stop it occupying the next ten cases. What the note cannot do is un-name a mode, which is why the durable protection is the independent coding the fixed seed makes possible: a colleague working from the written definitions, who was not there when the theory formed, disagrees with you hardest at exactly the ambiguous cases the theory pulled.

The same rule applies to the cases you would rather skip. Marginal ones — where the assistant's answer is defensible, or the user's question was genuinely ambiguous, or the failure is only a failure under a strict reading of the rubric — are the most tempting to drop, and dropping them systematically inflates every mode you keep. The arithmetic is simple and it catches people anyway. Dropping a case takes it out of the denominator while every mode you kept keeps its count, so every share rises, and the dropped cases are not a random subset of the fifty, because what made them droppable was fitting no mode cleanly.

If a case is genuinely unreadable, because the trace is truncated or the conversation is in a language nobody on the team reads, replace it with another draw from the same stratum and record how many you replaced. The same stratum, because the quota is what every design weight is computed from, and a promotions case quietly replaced by an order-status one changes the sample share of both. The record matters for the same reason: a stratum where most draws turned out unreadable was not sampled at the rate the table claims, and that surfaces later as a wrong weight rather than as a missing case.

When the reading should wait#

None of this argues against reverting. A revert that takes eleven minutes and returns the system to a known state is cheap and reversible, while reading fifty traces takes four hours, so in most releases the two are not competing for the same slot at all: revert, then read.

The cases where you should revert before understanding anything share a shape. The failure produces side effects that cannot be undone — money moved, emails sent, records written to a system of record, orders cancelled. Or it touches a regulated or safety-critical path, where exposure at any rate is treated as a constraint rather than a cost. Or the blast radius is large and growing, because the change is on the revenue path or in front of all traffic rather than 5% of it. Or the revert itself is cheap and low-risk, which pushes the calculation on its own, since the option to look more carefully afterwards then costs almost nothing.

Those four share one property that makes them decisive rather than merely serious. In each, the cost of the delay keeps accruing for as long as the reading takes, and in the first it accrues into a form money cannot buy back: a refund can be reversed, an email that has arrived cannot be recalled. Reading first spends a fixed four hours to buy an explanation, and that trade holds only while the meter running against it is bounded.

The cases where reading first is the better call are the mirror image: the change is behind a flag or at low exposure, the failures are recoverable, reverting would also revert a fix that is holding something else together, or reverting is genuinely expensive because it entails a data migration going backwards. What makes the two-hour argument reasonable is narrower than reading beating reverting: reverting without reading leaves you with a system at 0.86 and no idea what to do next, and the same regression then ships again in a fortnight.

The draw is the cheapest place in this whole practice to destroy the result, and the most expensive place to discover you did. Fifty traces read carefully out of a badly chosen fifty produce a confident, well-documented, reproducible description of the wrong thing, and nothing downstream — not the coding, not the rate conversion, not the prioritisation — can detect it, because every one of those steps takes the sample as given. None of them even holds the information that detection would need: a codebook sees notes, a rate conversion sees counts and a denominator, and none of them ever sees the conversations that were not drawn. Twenty minutes of allocation and a committed seed is the entire defence.

At the end of the read you have fifty traces and fifty or so notes, written in the vocabulary of whatever each conversation happened to be about. Turning that into something with counts requires deciding what the categories are, and the first attempt at that is nearly always a set of labels nobody else can apply the same way.