Turning counts into rates and cost

The output of a round of coding is a list of modes with counts out of fifty, and those counts are not yet a measurement of anything about the product. Two things have to happen to them. The denominator has to change, because a share of failures answers a different question from a share of conversations and the two differ by a factor equal to the whole failure rate. And frequency has to be multiplied by something, because ranking modes by how often they happen is only correct if every failure costs the same, and the cheapest mode to produce is usually also the cheapest to suffer.

Both conversions run here on a shopping assistant handling about 60,000 conversations a month, whose failure modes were named by reading fifty traces drawn out of a pool of nominated candidates. The mode at the top of that product's chart, sorted by count, is the one that ends up worth nothing, and a mode with a sixth of its frequency is the one leaking money. That inversion is common enough that the arithmetic is worth doing before anyone argues about priorities rather than after.

Two denominators, and only one of them is the product#

A random audit of 200 conversations, read and judged by hand, found 16 that went badly, giving a failure rate of 8%. That is roughly 4,800 failing conversations a month, and it is the number every mode count has to be projected through.

Before projecting anything, it is worth writing out the sentence the projection exists to prevent, because it gets said in almost every one of these meetings. A bar labelled 17 sits at the top of a chart of failure modes, and the reading offered is that the mode affects 17% of conversations. That reading is wrong twice over, and the two errors are worth separating, because somebody who fixes only the first arrives at 34.0% and is then wrong in the more expensive way.

The first error is reading a count as a percentage. The 17 on that bar is a count of traces out of a sample of fifty, so as a proportion it is seventeen fiftieths, which is thirty-four per cent. Taking 17 to mean 17% quietly assumes the sample held a hundred traces, and the sample held fifty. That slip is the easier of the two to catch, because the sample size is usually printed on the same chart, and correcting it doubles the figure.

The second error survives the correction untouched. Thirty-four per cent is a correct proportion of the fifty traces that were read, and every one of those fifty had already been judged a failure before anybody opened it. So thirty-four per cent is a share of failures: among conversations that went badly, about a third carried the preamble mode. It says nothing whatever about how often conversations go badly, because the population it was measured on had that question settled in advance, by selection, before the reading started.

Neither 17% nor 34.0% is therefore a share of conversations, and a hundred conversations taken off the traffic is enough to see how far off both are. Eight of that hundred go badly, which is what an 8% failure rate means. Of those eight, about a third carry the mode, which is not quite three conversations. The mode reached fewer than three of the original hundred, then — not seventeen and not thirty-four — and the exact figure is 2.72%. The naive reading overstates it by more than six times and the corrected-but-mislabelled one by a factor of 12.5, which is the entire failure rate.

What that hundred-conversation walk did was chain two ratios, and the chain is the reason multiplication is the right operation on them. The count over the sample estimates the mode's share within failures — occurrences of the mode per failing conversation. The failure rate is failing conversations per conversation. Multiply the two and the failing conversations cancel, top and bottom, leaving occurrences of the mode per conversation, which is the quantity wanted. A share of a share is a product in exactly the way that half of a third is a sixth.

Adding the two would require them to count the same kind of thing, and they do not: one is a ratio measured inside the failures and the other is the ratio of the failures to everything. Nobody is seriously tempted by addition. The temptation is to combine them not at all, which is what leaves a share of failures standing next to the word "conversations" with nothing having been done to it.

CONVERSATIONS_PER_MONTH = 60_000
FAILURE_RATE = 0.08                      # 16/200 in the random audit
FAILING_PER_MONTH = CONVERSATIONS_PER_MONTH * FAILURE_RATE   # 4,800

def project(count, sample=50):
    share_of_failures = count / sample
    rate_of_traffic   = share_of_failures * FAILURE_RATE
    per_month         = share_of_failures * FAILING_PER_MONTH
    return share_of_failures, rate_of_traffic, per_month

for name, n in [("preamble", 17), ("expired promo", 3), ("invented policy", 1)]:
    s, r, m = project(n)
    print(f"{name:16} {s:6.1%} of failures  {r:6.2%} of conversations  {m:7.0f}/mo")

# preamble          34.0% of failures    2.72% of conversations     1632/mo
# expired promo      6.0% of failures    0.48% of conversations      288/mo
# invented policy    2.0% of failures    0.16% of conversations       96/mo

The two percentage columns differ by a factor of 12.5, which is the whole failure rate, and mixing them up is the most common arithmetic error in this work. "A third of our failures" becomes "2.7% of conversations", which is a number an exec can compare against something else the company measures. The monthly column is more useful still, because it converts to money and to support hours without further translation.

The 8% has to come from the random arm, and the reason is a property of how the nominated pool was assembled rather than a matter of tidiness. Candidates were nominated by selecting on signals correlated with failing — a retry, a rephrasing within a couple of turns, a judge flag. A failure rate computed inside that pool is therefore the probability that a conversation failed given that it tripped one of those surfaces, which is a conditional probability. The projection needs the other one, the probability that a conversation failed at all, taken over the whole traffic including the great majority that nothing flagged. Selecting on a correlate of the outcome pushes the conditional above the marginal by however well the surfaces discriminate, and a surface that discriminates well drives it close to 100% by construction, which is how substituting it produces monthly projections roughly an order of magnitude too high.

The share of failures carries a condition of its own, and it is the step to make somebody demonstrate rather than assert. The fifty traces came out of that same nominated pool, so a count over the sample estimates the mode's share among failures that some surface caught. It equals the mode's share among all failures only if nomination is uncorrelated with which mode caused the failure. A mode that produces a confidently wrong answer nobody bothers to retry is invisible to a retry surface, so it is under-represented in the pool and its projection comes out too low. The per-surface recall measured on the random audit is where that gets checked; where it cannot be checked, the monthly figure should be reported as a floor rather than as an estimate.

Severity has to be a unit, not an adjective#

Sorting a chart by count and reading the top of it as the priority is not a careless habit, and it is correct nearly everywhere else it gets used. On a product dashboard the rows are commensurable: every row is a signup, a crash, an abandoned checkout, and one of anything on the list is worth about what one of anything else on the list is worth. Where that holds, cost is count times a constant, and multiplying every row by the same constant leaves the order unchanged — so sorting by count is sorting by cost, for free, and a decade of dashboards has trained everyone in the room to read it that way. Carried across to failure modes, the belief becomes that the tallest bar is the biggest problem, and the assumption it smuggles in is that constant. Failure modes break it outright: the multiplier on one row is zero and on another it is tens of dollars, and no ordering by count survives multipliers that vary by that much.

To multiply frequency by anything, severity has to be expressed per occurrence in a unit that is the same across modes. Per occurrence is the part doing the work. A severity attached to a mode as a whole — "this one is bad" — carries no unit, and multiplying an adjective by 1,632 is not an operation. A severity attached to one occurrence is harm per event, a count is a number of events, and their product is harm, in the same way that a price per item times a number of items is money.

Where money exists, use money. A company that honours a quoted price even when the promotion has expired loses an average of $40 per occurrence, which is a real number somebody in finance can confirm. Chatty preamble has no cost to attach: nobody refunds anything, no support ticket is opened, no order is lost, so its severity per occurrence in money is zero, and zero times 1,632 a month is zero.

Where money does not exist, an anchored ordinal scale works — ordinal meaning the levels are ranked but the gaps between them carry no arithmetic — as long as each level is tied to an observable consequence rather than to a feeling.

The gaps in the scale are deliberate; consecutive integers invite averaging, which is meaningless on an ordinal scale. The numerals are names for ranks, and the middle level was chosen to sit above the one below it and beneath the one above it rather than to be some particular number of times worse than either. Averaging assumes the gaps are equal and known, and nothing in the way the scale was built supplies them, which is why adding two levels together yields a level that means nothing. What the levels are for is forcing a conversation to happen once, in writing, rather than implicitly and differently in every meeting.

Getting the money figures is a matter of asking the right team rather than of estimating. The $40 comes from finance, who can pull the average adjustment on orders where a promotional price was honoured outside its validity window; the request is specific enough to be answerable in an afternoon. Level 1 and level 3 modes usually convert through support: an extra clarification turn costs nothing directly, but a mode that drives a contact to a human agent costs whatever a contact costs, which support operations already tracks — typically somewhere between $4 and $9 for a chat contact at this size of company. Converting a re-asks mode at 576 occurrences a month into money requires knowing what fraction of those escalate, which is a join between the mode and the handoff table rather than a guess. Modes whose cost genuinely cannot be sourced stay on the ordinal scale and get argued about explicitly, which is worse than money and much better than an adjective.

Putting both conversions together gives a table where the tallest bar is no longer at the top.

The mode occupying a sixth of the frequency of the largest bar is leaking somewhere around $11,500 a month, and the largest bar is leaking nothing measurable. That inversion is the reason the conversion is worth doing at all, and it is not a rare result — cosmetic failures are common precisely because they are cheap to produce and nobody has ever been motivated to prevent them.

How much of this survives a sample of fifty#

Every number in that table rests on counts out of 50, and a count of 3 out of 50 is not a precise measurement. Somewhere in the traffic there is one true rate at which the mode occurs among failing conversations, and the fifty traces are a single draw against it; a different fifty would have come back with a different count. The useful question is therefore not what rate a count of three implies, but which underlying rates would have produced a count near three often enough that seeing it would not be a surprise. Gather all such rates together and the collection is an interval, and that is what a confidence interval is before it is anything else.

The one appropriate for small proportions has a name: the Wilson score interval — the band of underlying rates that could plausibly have produced the count you observed, derived in a way that stays sensible when the count is close to zero. Operationally the band means that if the true rate sat anywhere inside it and you redrew fifty traces many times over, a count like the one you saw would come up regularly; at a rate outside the band it would come up rarely enough to be worth disbelieving.

The interval most people were taught instead takes the observed proportion and adds and subtracts a fixed multiple of a spread computed from that same proportion, and near zero it fails in a way visible on inspection. At one in fifty the observed proportion is small and the spread computed from it is small too, but not small enough: the margin comes out wider than the proportion itself, so the lower end of the band lands below zero. No traffic could have produced a negative rate, so the interval has returned a set containing impossibilities. The reason is that it is symmetric by construction, while the set of rates consistent with a count of one is not symmetric — a rate cannot be pushed below zero, but it can be pushed a long way above the observation. Wilson's interval is obtained by asking which rates are consistent with the count rather than by drawing a symmetric band around it, and that is why its ends stay inside zero and one.

from math import sqrt

def wilson(k, n, z=1.96):
    p = k / n
    d = 1 + z*z/n
    centre = (p + z*z/(2*n)) / d
    margin = (z/d) * sqrt(p*(1-p)/n + z*z/(4*n*n))
    return centre - margin, centre + margin

for name, k in [("preamble", 17), ("expired promo", 3), ("invented policy", 1)]:
    lo, hi = wilson(k, 50)
    print(f"{name:16} {k}/50 -> {lo:.1%} to {hi:.1%} of failures "
          f"({lo*4800:.0f} to {hi*4800:.0f} per month)")

# preamble         17/50 -> 22.4% to 47.9% of failures (1077 to 2297 per month)
# expired promo     3/50 ->  2.1% to 16.2% of failures ( 99 to  779 per month)
# invented policy   1/50 ->  0.4% to 10.5% of failures ( 17 to  504 per month)

The width is startling the first time it is computed. The expired-promotion mode is somewhere between 99 and 779 occurrences a month, which is between $4,000 and $31,000. That range is wide and it is still decisive, because the alternative it is being compared against is zero. Uncertainty matters when it spans a decision boundary and not otherwise, so the right response to a wide interval is usually to check whether it changes what you would do rather than to go and read another fifty traces. Reading more would help less than it sounds: the width of these intervals shrinks roughly with the square root of the sample size, so doubling the reading budget narrows the band by less than a third, at double the cost in somebody's afternoons.

The 8% failure rate carries its own uncertainty — 16 of 200 gives a Wilson interval of 5.0% to 12.6% — and the two uncertainties compound when multiplied. The monthly figure is a product of two estimated quantities, and a product moves under both of its factors at once. The low end of the mode's share can meet the low end of the failure rate, and the high end can meet the high end, so the ends of the product are driven further apart than the ends of either factor on its own. Multiplying the ends together is the pessimistic form of that, since it assumes both estimates are wrong in the same direction in the same month, and it is the form worth quoting when the number has to survive being challenged. That is why "roughly $10,000 a month, somewhere between $4,000 and $30,000" is both more honest and harder to argue with than a point estimate carrying two decimal places.

A second correction comes from the sampling floor that made small strata readable. Every intent was given a minimum number of cases in the draw so that a small intent would not arrive with one or two traces, and a floor applied to a small stratum necessarily lifts it above its proportional share. Promotions is about four per cent of the pool of candidates and it supplied four of the fifty traces, which is eight per cent of the sample — roughly twice its share. So promotions cases turn up in the sample at about twice the rate they occur in the pool, and each one therefore stands for about half as many pool cases as an average trace does.

Counting each promotions case as half a case restores the pool's proportions, and that is the design weight. Stated generally: a stratum over-sampled by a factor of F contributes F times too many cases, so each of its cases has to count 1/F to put the pool's proportions back. F is the stratum's share of the sample divided by its share of the pool, which makes 1/F the stratum's share of the pool divided by its share of the sample, and that is the rule as it gets written down.

# promotions: 170/4000 of the pool but 4/50 of the sample -> weight 0.53
# returns:    880/4000 of the pool but 10/50 of the sample -> weight 1.10
weights = {"promotions": 0.53, "returns": 1.10, "order_status": 1.21}

expired_promo_cases = ["promotions", "promotions", "returns"]
weighted = sum(weights[s] for s in expired_promo_cases)   # 2.16
print(f"{weighted/50:.1%} of failures")                    # 4.3%

Dividing that weighted total by 50 rather than by the sum of all the weights is only legitimate because the weights are built to preserve the sample size, and the check is short enough to do here. A stratum contributes its number of cases multiplied by its weight, the weight has the stratum's share of the sample in its denominator, and the share of the sample is that same number of cases over fifty — so the number of cases cancels and the contribution collapses to fifty times the stratum's share of the pool. Promotions supplies four cases at 0.53, which is a little over two, and it is a little over four per cent of the pool; returns supplies ten cases at 1.10, which is eleven, and it is a little over a fifth of the pool. Sum that across every stratum and the pool shares add to one, leaving exactly fifty.

Two of the three expired-promotion cases came out of the deliberately over-sampled promotions stratum, so the reweighted share is 4.3% of failures rather than 6.0%, and the monthly figure drops from 288 to about 207 occurrences, or $8,280. That does not change the ranking here. It changes the ranking often enough that skipping it is a habit worth not forming, and it is the price of the floor that made the small strata readable in the first place.

Averages do not describe a tail#

Expected cost is an average over many occurrences, and averaging is exactly the wrong operation for a class of failure that is not about accumulation. The invented policy exception at 96 occurrences a month is not a $96,000 problem or a $96 problem; it is a mode where one occurrence in front of the wrong person becomes a regulatory finding and the other 95 cost nothing at all. Multiplying it by anything produces a number whose only function is to make it rank low.

The honest treatment is to keep two lists. One is ranked by expected cost, where multiplication is valid because harm accumulates roughly linearly and the distribution has no meaningful tail. The other is a constraint list, where modes appear because their worst case is unacceptable regardless of rate, and the question asked of them is not "how much does it cost" but "is this rate below what we are willing to accept, and who signed that". The invented policy exception belongs on the second list, and it belongs there at n = 1.

The other thing multiplication cannot do is validate the severity number it is multiplying. The $40 average loss is a mean, and if the distribution behind it is 90% of cases at $8 and 10% at $330, the mean is real and the planning implications are different. The same doubt applies harder to the zero attached to preamble, which is an assumption that verbosity has no effect on conversion, retention or support volume, and which nobody has measured. A 0.4% relative drop in conversion across 1,632 conversations a month would put preamble back in contention, and there is no way to find that out from the traces.

Rates are what make two rounds of this work comparable, and that is a further reason to store them rather than counts. It shows up a quarter later. Counts out of fifty are specific to one sample of one month, and the next round will not draw from the same month or the same volume of traffic. Suppose it reads fifty failures from a month carrying 74,000 conversations at a failure rate of 6%, and suppose a mode sits at 3 of 50 in both rounds. Its share of failures is six per cent both times, and that share is the only thing that held steady. In the first round six per cent of 4,800 failing conversations is 288 occurrences a month; in the second, six per cent of 4,440 failing conversations is 266. The rate over conversations has fallen from 0.48% to 0.36%, and it fell purely because the failure rate the share is projected through fell. The monthly count dropped less steeply than the rate did, because the traffic grew over the same period and partly offset it.

Comparing raw counts across rounds therefore compares two different denominators and produces confident statements about trends that are artefacts of traffic growth. Keep the sample size, the failure rate with its source and interval, the monthly volume and the per-mode counts with their design weights, and the rate becomes a derived quantity rather than a remembered one. A mode's history is then a series of numbers that can be put beside one another, instead of a series of counts out of samples that happened to be fifty.

Two numbers, then, decide whether a mode is worth anybody's attention, and a count out of fifty is neither of them. The first is how often the mode reaches a real user, which needs the base rate from an arm that had no opinion about what a failure looks like. The second is what one occurrence costs, in a unit that survives being multiplied, sourced from whoever owns that cost rather than estimated in the meeting.

That is the point at which the arithmetic stops and a decision has to be made. Ranking by expected cost orders the modes; it does not say which one to work on, because the modes cost different amounts to fix, one of them cannot be priced at all, and the work has to be justified to people who did not read the traces.