Measuring whether a mode was fixed

A fix aimed at one failure mode gets checked, almost everywhere, by rerunning the eval set and reading the pass rate — the share of cases in that set a grader marks as passing. That check is unable to answer the question being asked of it, for a reason that is arithmetic rather than methodological. A mode running at a fraction of a percent of conversations occupies about one case in a set of two hundred drawn to look like production, and a case in which the mode cannot occur cannot be flipped by a fix to it. Eliminating the mode entirely therefore moves the aggregate by half a point at most. Everything above that half point is something else, and a release usually contains several something-elses.

Two measurements can carry a claim about one mode where the aggregate cannot: a re-code of the same cases against the same codebook, reported as a rate per mode, and a pair of constructed slices, one where the mode's precondition holds in every case and one where the behaviour the fix suppresses is the correct behaviour. Both rest on the property that made the mode countable in the first place, which is that its definition was written as a rule about conditions present in the trace (chapter 4, where a mode name becomes something a second reader can apply without knowing what the first reader was thinking).

Three worlds, one number#

Start with what a pass rate is made of, because the whole difficulty is visible in its construction. Every case in the set is graded pass or fail, and the pass rate is the count of passing cases divided by the count of cases, which makes it a sum over a population with one term per case. The failures can be summed a second way. Each failing case carries exactly one primary mode from the codebook, so the failure count is the total of the per-mode counts, and the pass rate is one minus that total divided by the size of the set.

A targeted fix acts on one term of that second sum. If it works and touches nothing else, the target mode's count falls, every other count stays where it was, and the pass rate rises by exactly the fall in that one count divided by the size of the set. Everything in this chapter follows from dropping the assumption in the middle of that sentence. Addition does not record which terms moved, so any collection of per-mode changes adding to the same total produces the same movement in the pass rate. Four points is a total, and a total is consistent with every set of changes that sums to it.

The specific problem with an aggregate as evidence for a targeted claim is that several distinct outcomes produce the same movement, and they call for different next actions.

Each row names a different set of terms, and each would require something different to be true. World A requires the fix to have worked and the release to have contained nothing else that moved a case, and its row is the half-point bound from the opening being reached exactly — which is why an honest success here arrives looking like a disappointment. World B requires the target mode to have been left alone and some other mode to have improved by around eight cases, which a prompt instruction shipped in the same release can easily do. World C requires two terms to move in opposite directions, the target falling to zero while a mode that did not exist before appears beside it. World D requires several terms to move at once with the balance coming out positive.

One measurement tells all four apart, and it is the count of each mode taken before and taken again after. In world A the target's term falls and no other term moves. In world B the target's term is where it was. In world C the target is at zero and a term that was zero is now positive. In world D several terms have moved in both directions. None of those distinctions is subtle once the terms are visible; they are invisible only for as long as the terms are added together.

World C is the one worth dwelling on, because it is both common and invisible from the top. A fix that constrains a behaviour usually over-constrains it, so the mode you targeted goes to zero and a new mode appears next to it — the assistant now declines to quote promotions that are perfectly valid. On a general eval set those two effects partly cancel, and the residue is a small positive number that reads as success.

There is a fifth story the table has no row for, and it catches the people who have got everything else right. The four rows treat the counts as facts about the system, and they are counts taken on a sample. Two hundred cases drawn from production are two hundred draws, and a mode occurring at a low rate contributes a count that differs from one draw to the next while the product stays exactly as it was. A term that moves by a case or two is therefore compatible with the term not having moved at all. No amount of care in the reading removes that, because it is a property of how many cases were read rather than of how well they were read.

The rule underneath all of this is short enough to carry around. An aggregate is not a measurement of a mode. The measurement of a mode is a count of that mode, taken on cases where the mode can occur, using the definition that produced the first count.

Measure the mode, on cases where the mode can happen#

The first part of the check is straightforward: rerun the same 200 cases on the new build, take the failures, and code them with the same codebook, including the excludes blocks. Same definitions, same procedure, same coder if possible, and a rate per mode rather than a pass rate.

Why that re-code compares like with like is worth spelling out, because it is the step the rest of the chapter stands on. A mode's definition in the codebook is a rule stated from conditions present in the trace — a valid_to earlier than the conversation timestamp, a policy figure that appears in no retrieved chunk — so deciding whether a trace exhibits the mode is a matter of looking rather than of judging. A rule of that shape returns the same verdict on the same trace whoever applies it and whenever they apply it. Applying it to the after-traces therefore puts the identical question that was put to the before-traces, which is what makes the difference between the two counts a difference in the traces. Had the modes been called "wrong" and "bad", the second reading would have measured the reader as much as the build, and the difference between the counts would carry both at once.

The excludes blocks do a specific job at this moment that is easy to skip. A fix that over-constrains a behaviour produces cases sitting right beside the mode it was aimed at, and without an explicit boundary those cases land in whichever existing bucket is nearest. The trade in world C then disappears into a count that was already on the chart. Where the original coder is not available, the agreement check from chapter 4 — twenty traces coded blind by a colleague against the written definitions, scored with Cohen's kappa so that labels two readers would land on anyway are not banked as agreement — is what makes a second reader's counts comparable with the first reader's.

before = {"expired_promo_quoted": 3, "preamble": 17, "re_asks": 6,
          "invented_policy": 1, "promo_over_refusal": 0}   # of 44 failures
after  = {"expired_promo_quoted": 0, "preamble": 4,  "re_asks": 6,
          "invented_policy": 1, "promo_over_refusal": 5}   # of 36 failures

N_BEFORE, N_AFTER = 200, 200
for mode in sorted(set(before) | set(after)):
    b, a = before.get(mode, 0), after.get(mode, 0)
    print(f"{mode:22} {b:2d}/{N_BEFORE} -> {a:2d}/{N_AFTER}  "
          f"{(a-b)/N_BEFORE:+.1%} of cases")

# expired_promo_quoted    3/200 ->  0/200  -1.5% of cases
# invented_policy         1/200 ->  1/200  +0.0% of cases
# preamble               17/200 ->  4/200  -6.5% of cases
# promo_over_refusal      0/200 ->  5/200  +2.5% of cases
# re_asks                 6/200 ->  6/200  +0.0% of cases

That is world C, and the four points of aggregate movement turn out to be mostly a preamble instruction that went in the same release, with a new over-refusal mode consuming a third of the gain. None of it was visible in 0.78 → 0.82, and all of it changes what happens next.

The five modes printed there do not account for every failure, and the gap is worth a moment. Their counts fall by three on the expired promotion and by thirteen on preamble, and rise by five on the over-refusal, a net fall of eleven cases across the five. The failure total fell by eight. Whatever else was coded in those two runs must therefore have risen by three cases between them, spread over modes nobody has looked at. That residue is invisible in the aggregate and invisible in this printout for the same reason: a total is a total, and the only way to see a term is to print the term.

The second part addresses statistical power, meaning whether the measurement was capable of detecting the change at all supposing the change had happened. Three cases going to zero out of 200 is consistent with the mode being fixed and also with three cases happening not to recur, and a general set cannot settle which.

It is worth putting a number on how consistent, because "could be noise" is the kind of objection that gets waved away when it is left as an adjective. Each of the two hundred cases either exhibits the mode or does not, and one conversation exhibiting it says nothing about whether the next one will, so the cases can be treated as independent draws at some small underlying rate. Suppose the fix changed nothing and that rate stayed where it was. The chance that not one case in the set exhibits the mode is then the per-case chance of not exhibiting it, multiplied by itself once for every case in the set, and when the rate is small that product sits very close to the exponential of minus the expected count. Expected count here is the rate times the size of the set, which the before run put at around three. The exponential of minus three is about one run in twenty, and one run in twenty is not a rare event for a team shipping weekly, so a zero after the change is an observation an untouched mode hands you from time to time.

What that argument does not license is the reverse reading. It does not say the fix failed, and it does not say the three cases were noise; both remain perfectly possible. It says this measurement, on this set, cannot separate them, which is a statement about the instrument rather than about the change. A real improvement can sit inside the noise band of a measurement while being entirely real, because the band's width is set by how many cases could have exhibited the mode and not by how large the improvement was. An instrument, unlike a release that has already shipped, is something you can rebuild.

What settles it is a targeted slice: a set of cases constructed so that the mode's precondition holds in every one of them. For an expired-price mode that means conversations where a promotion whose valid_to is in the past is retrievable and relevant — thirty of them, pulled from stored conversations by the same search that turns a single-case mode into a real count (chapter 4, where a rare mode's definition is run as a pattern or a similarity search across a month of stored traffic).

The slice buys power by raising the rate rather than by raising the sample size, and the distinction is the whole reason thirty cases can outperform two hundred. In the general set the mode's expected count was around one, because almost every case could not have exhibited the mode whatever the build did, and a term that small has nowhere to move that is distinguishable from where it started. In a set where the precondition holds everywhere, the mode's expected count is a large fraction of the set, so the same change in the underlying behaviour presents itself as a change of many cases instead of one. Nothing about the product differs between the two measurements. Only the cases they are taken on differ.

Twenty-one of thirty down to two of thirty is a claim you can defend, because the slice was built so that a change in the mode has to show up in it. How far outside the noise that movement sits comes out of the same reasoning as before. The cases in the slice are independent draws at one rate, and for a sum of independent yes-or-no draws the spread is the square root of the number of draws multiplied by the share that come out yes and by the share that come out no. Take twenty-one of thirty: thirty times seven-tenths times three-tenths, under a square root, is between two and three cases. The observed drop is nineteen cases, which is several times that spread, and the spread on the after count is smaller still because the share failing there is small. A set of thirty does not rearrange itself by nineteen cases.

What the slice licenses is narrower than it looks, and the narrowing matters as soon as somebody repeats the result in a room you are not in. The slice was constructed so that the precondition holds in every case, and production is not built that way, so two of thirty is the mode's rate given its precondition rather than a rate over traffic. Getting back to a rate over conversations needs the share of traffic in which the precondition holds, which is the projection step from chapter 5, where a count out of a sample is carried through a failure rate measured on a random audit to become a share of conversations. The defensible sentence is that the mode has become much rarer on cases where it could occur at all. A claim about the release is a different claim and still needs the general set.

The control slice is what catches the trade, and constructing one is the habit that separates a fix from a swap: for any fix that suppresses a behaviour, build the set of cases where the behaviour is correct and measure them too. The precondition slice then becomes a permanent artefact — added to a golden dataset of the kind topic 3.4 assembles, as a named slice with its own threshold, so that the mode you spent a week on has a regression test rather than a memory.

Four things a flat dashboard is consistent with#

Six months later the same team has a pass rate that has sat between 0.87 and 0.89 the entire time, no error analysis since, and a new product manager asking for a quality roadmap. Two engineer-weeks of reading transcripts before proposing any fixes sounds like two weeks of not shipping, and the flat line sounds like evidence that quality is stable.

Quality genuinely being stable is one of four readings. Improvements in one area cancelling regressions in another is the second, which is world D above running for six months. A ceiling effect is the third, where the remaining 12% of cases are ones the set cannot distinguish and the score cannot move regardless of what happens. The fourth, and the most likely at six months, is a set that no longer describes the product: the eval set was built when the assistant handled six intents, the product has since added subscription management, gift orders and a returns portal, and if 41% of current traffic falls in intents represented by fewer than five cases, the score is a precise measurement of a system that no longer exists.

Those four are the earlier table again, with months in place of one release. A flat total is the sum of terms that may all be flat, or may be moving and cancelling each other out, and addition cannot tell you which. Separating them takes the same move as before: hold the count per mode over time instead of the total, and check the composition of the set against the composition of traffic. The ceiling reading carries one extra step worth making explicit. Cases the set cannot distinguish are graded the same however the system behaves, so their terms are constants in the sum, and a constant term contributes no movement no matter how much engineering is aimed at it.

A set that no longer describes the product is the reading to make the case for, and it is a case about measurement validity rather than about craft. The dashboard has been flat because it stopped being connected to the product, which means the team has been flying on an instrument that is no longer wired to anything, and two weeks of reading is what re-wires it. Framed against the previous round it is also a claim with a track record: the last time this was done it found an $8,300-a-month leak that no dashboard had surfaced in the four months it had been running, and it cost five days.

Attaching the practice to decisions that already exist#

The failure mode of that argument is winning it. A quarterly error-analysis review gets approved, someone runs it, produces a document, and by the third quarter the document is written the week before the review, read by nobody, and produced because it is on the calendar. Rituals detached from decisions decay reliably, and the decay is not a discipline problem. What prevents it is attaching each round to a decision that is already being made on a date that already exists, so the output has a consumer and the round is skipped only if the decision is skipped.

The loop closing back to the triggers is the part that matters. A round of analysis whose output is a document produces nothing; a round whose output is three roadmap items with dollar figures, two new slices in the eval set and a judge criterion has changed four artefacts other people are already using.

The weekly half hour is the least glamorous row and the one that keeps the codebook alive between quarters, because it is where new modes first appear. Five traces a week is 260 a year, read by different people, and it costs less than the meeting that usually precedes it.

A judge only counts modes somebody named#

The obvious next step, once a codebook exists with tight definitions, is to turn each mode into a judge criterion and count it continuously. That works, and it is where most teams end up: the expired_promo_quoted definition becomes a judge prompt, runs on a sample of daily traffic, and produces a per-mode time series that makes the quarterly count a live number. It is a genuine improvement and it has a hard limit, which is that a judge can only count modes somebody already named. Every automated counter is a monument to a previous round of reading, and the modes that will matter next quarter — the ones a new subscription flow is about to produce — are not in it.

Both the promotion of a definition into a judge prompt and the ceiling on what any judge can count come out of the property the re-code depended on. A rule written from conditions visible in the trace is a question that can be put to anything capable of reading a trace, a colleague or a model, and the answer does not depend on having been in the room when the mode was named. That portability is what lets the definition become a prompt. It also means a question has to exist before it can be asked, and nothing in the counting apparatus proposes questions, so the set of modes any dashboard can report was fixed on the day somebody stopped reading traces by hand.

That limit is why the practice does not converge on a finished artefact. The codebook has to be re-derived by hand periodically against fresh traces, with the previous codebook set aside during open coding so it does not become the frame that finds only what it already knows. Hamel Husain's write-up on error analysis in LLM evals is worth reading for the version of this procedure practitioners have converged on, and Alex Strick van Linschoten's worked run of it on real traces shows what the intermediate artefacts actually look like.

A claim about a mode and a claim about a release are different claims, and the evidence for one cannot be borrowed for the other. What makes the mode claim checkable is that it is measured where the mode can occur, with the same definitions that produced the original count, alongside the cases where the suppressed behaviour was right all along. That is three measurements instead of one number, and it is the difference between knowing what a week of work did and believing a chart.

What you have at the end of a round is not a report. It is a codebook with definitions somebody else can apply, a set of counts converted into rates over real traffic, a ranked list with money on it, a short constraint list with owners, two or three new slices in the eval set, and a date in the calendar when the next round runs because a decision needs it. The next thing to do with all of that is the round itself — pull the pool, draw the fifty, and start reading at turn one.