Prioritising modes by expected cost

Prioritisation here means turning a ranked list of failure modes into one decision about what an engineer does next week. The ranking arrives with a monthly cost attached to each mode, which is more than most planning conversations have, and it is still only one of the three quantities the decision needs. The second is what each fix costs to build, because a mode leaking thousands a month behind a week of infrastructure work and a mode leaking hundreds behind half a day are not ordered the way their harm figures order them. The third is whether the mode's harm accumulates at all, because expected value is an average and averages are the wrong operator for a failure whose worst outcome is categorically different from its typical one.

A ranked list looks like a plan, and that belief is the one to dislodge first. It is tempting for a good reason: the ranking is total, every mode has a place in it, and the places were assigned by arithmetic rather than by advocacy, so it carries the authority of a measurement. What it orders is harm. What a plan orders is work, and the two lists come apart wherever a fix is expensive, wherever a mode has no price to enter into the ranking in the first place, and wherever two modes turn out to share one cause and are therefore one piece of work rather than two.

Return, and the horizon it is measured over#

Expected harm and the cost of a fix are not quantities of the same kind, and nothing in the analysis so far has made them comparable. One is a flow of money leaving the business each month, assembled from traffic, a rate and a cost per occurrence, and owned between analytics and finance. The other is a stock of engineering time held by a manager whose units are people and weeks. There is no operation on weeks that returns dollars. What supplies one is a price, and the price has to be stated out loud before the two can appear in the same table.

A loaded engineer-week — salary plus the benefits, tooling and overhead the company pays around it — is about $6,000, which is the number to set the benefit against. The payback column is how long a fix runs before the money it saves covers what it cost to build.

That figure is what the company pays to have the engineer available for a week. It is not what the week is worth. The worth of a week is whatever the best other use of it would have returned, and nobody has measured that, so wherever there is a queue of valuable work — and there always is — the loaded rate understates what spending a week actually costs. Using it anyway is a judgement with a stated exchange rate rather than an arithmetic fact. Its virtue is that the judgement sits in one number in one place, and somebody who thinks a week is worth more can argue about that number without reopening the traffic, the rate or the per-occurrence cost underneath it.

The payback column is a division, and it is worth doing once by hand, because the units are what make it legitimate rather than arbitrary. A fix cost is dollars. A benefit is dollars per month. Dividing dollars by dollars per month cancels the money and leaves a number of months, which is why the answer reads as a time instead of as a ratio with no interpretation. The expired-promotion row divides $6,000 by $8,300 a month, which comes to about seven-tenths of a month; on the thirty-day month the table is using, seven-tenths is the twenty-two days printed beside it. The row under it divides $600 by $6,000 a month, which is a tenth of a month, and a tenth of a month is three days.

Three things have to hold for that division to mean what it appears to mean, and none of them is checked by the arithmetic that produces it. The benefit has to be a flow that continues at the same rate for at least as long as the payback takes, so a mode that only fires during a sale does not pay back at the sale's rate for the rest of the year. The fix has to remove the whole of the benefit it is credited with. And the cost has to be genuinely one-off, so a validity field added to an index carries a week's price only until the first schema migration that has to carry it too.

The middle condition is the one the table half-admits, and it deserves its own name. A benefit column is never a mode's expected cost; it is the expected cost multiplied by the share of occurrences the fix removes. Every row here except one quietly sets that share to one, which is a claim about a rate after a change that nobody has observed at the time the table is written. A check at quote time is credited with $6,000 against the mode's $8,300, a bit under three-quarters of the loss, because refusing to state an expired price stops the quote without touching the rows the price came from. The step to make somebody demonstrate is that share, measured on traffic once the fix is live rather than asserted in the meeting where the work is scheduled.

The third row is the one most teams never write down, and it changes the answer. The week of reindexing is the correct fix; a validity check applied where the price is quoted is a mitigation that removes most of the loss in half a day and leaves the index as wrong as it was. Splitting every candidate into "mitigate" and "fix properly" turns a single week-long decision into a half-day decision plus a scheduling decision, and the half day pays back in three days.

Splitting works for a mechanical reason worth stating, because the same move does not help everywhere. Ordering candidates by benefit against cost only becomes a schedule if the work can be taken in pieces. With one person and one week of capacity, a candidate costing a week is taken whole or not at all, and a greedy walk down a ratio-ordered list can commit the whole week to the item at the top while two cheaper items further down would together have returned more. Writing a mitigation row beside a fix row is what makes a mode divisible: the same mode now appears twice at two prices, and the cheaper price fits inside a capacity that the expensive one fills completely.

Divisibility is not free, and the cost lands on the root fix. Every payback in the table is priced against the mode as it stands today with nothing done. Once the half-day check is live and most of the loss has stopped, the week of reindexing is no longer buying $8,300 a month — it is buying whatever the mitigation left behind, which on these figures is a little over two thousand dollars a month. Dividing $6,000 by that puts the payback closer to two and a half months than to three weeks. The reindexing still has to happen, and the last column is where the argument for it now lives, since an index that is wrong stays wrong for every other reader of it. A team that ships the mitigation and keeps quoting the original payback is quoting a number that its own decision invalidated.

The preamble row does not fail because it is cheap. It fails because the benefit column is zero. A payback is a fix cost divided by a benefit, and dividing by zero does not return a very large number of days; it returns nothing at all, which is what the undefined entry in that row records. No horizon is long enough for a benefit of zero to cover a cost of $150. An hour is a small enough cost that shipping it anyway is defensible on the grounds that someone will otherwise raise it in every meeting for a quarter, which is a real cost even if it is not the one being measured.

The cost column has a failure of its own, and it is the mirror image of the benefit column's. A week spent on the ingestion job, the index schema and the retrieval filter is one week whether it repairs one mode or three. Where a second mode also traces back to a row in that index being wrong, the two modes are not two candidates competing for the week; they are one candidate whose benefit is the sum of both, and pricing them as separate lines charges the week twice while crediting it once. The relationship runs the other way as well. A fix aimed at one mode moves the rate of another without anybody intending it, which is why a ranking of modes is not a list of independent items and why the arithmetic has to be redone after anything ships.

Modes that do not belong in the ranking#

The invented policy exception cannot be ranked against the other two, because expected value is the wrong operator for it. Expected value is an average, and an average is the right summary when harm accumulates across occurrences at a roughly constant rate per occurrence. That condition carries more weight than it looks. Multiplying a rate by a cost per occurrence is only legitimate if the cost per occurrence does not depend on how many occurrences there are, and if the harms add rather than interact: the hundredth expired price honoured costs what the first one did, a hundred of them cost a hundred times one of them, and the product and the sum therefore agree.

Both halves fail for a mode with a tail. The cost per occurrence is not a constant that a rate can be multiplied by, since it is near nothing for almost every occurrence and enormous for the one that is not; and the harms do not add, because a regulatory finding is not the sum of smaller findings. A mode whose distribution is 95 harmless occurrences and one that produces a regulatory finding does not have a meaningful mean, and computing one produces a small number that gets used as a reason to do nothing.

Two separate things go wrong when somebody computes it anyway. The mean of a distribution shaped like that is almost entirely one small probability multiplied by one large loss, and those are exactly the two quantities nobody holds — the probability rests on a single observation among fifty, and the large loss is what the "not priceable" entry in the table is admitting cannot be sourced. A product of two unmeasured quantities is not a measurement, however many decimal places it prints with.

The second failure is the worse one, because it survives even if both numbers were known exactly. An expectation is the right summary for somebody who will face the same draw many times and can settle each outcome out of the same pool of money. That is why it works for expired prices: the losses arrive in forty-dollar units, one budget line absorbs all of them, and over a year the average is close to what the company actually experiences. A regulatory finding is not one draw among many from a shared pool. It arrives once, it is not settled out of the promotions budget, and it does not average with the harmless occurrences beside it, because the average describes a run of events the company will never live through.

What such a mode gets instead is a constraint. The quantity that replaces the mean is a rate held against a ceiling, and the substitution is what makes the mode decidable again: nobody can sign a mean, since a mean is not an event anyone will observe, whereas a rate states how often the thing is currently allowed to happen and can be checked again next quarter. The question asked of it is whether the current rate is below what the organisation is prepared to accept and who is prepared to say so in writing. Either the answer is yes, in which case it is accepted explicitly with a named owner and a review date, or it is no, in which case it jumps the queue regardless of cost. A violated constraint is not traded against anything, because it entered the decision as a bound rather than as a term in a sum, and the trade-off it embodies was made once already by whoever set the acceptable rate. Both outcomes are better than the mode sitting in a bar chart at 2% and being quietly deprioritised for six months by nobody in particular.

The practical move for these modes is almost always mitigation rather than root-cause work. Preventing a model from ever inventing an entitlement is an unbounded research problem; detecting entitlement language in an outgoing message and routing those conversations to a human is a day of work with a bounded failure surface. Cheap containment is available for most tail-risk modes — the ones whose worst outcome is far worse than their typical one — precisely because you only need to catch the shape of the output rather than fix the reasoning that produced it.

Diagram

The order of the top two questions is doing real work, and reversing them quietly undoes the section above. A mode only reaches the expected-cost box after its worst case has been established as bounded, so a figure named for a regulatory finding by somebody willing to name one — and in a large enough meeting somebody always is — never gets as far as being multiplied by a rate. A number that can be produced is not the same as a number that may be multiplied. The two entry conditions at the top are what keep the rare severe modes out of the arithmetic, and the mitigation branch is what keeps the constraint list from becoming a list of things nobody has capacity for.

Sorting a chart by count buries severity#

An exec chart with four bars, produced by merging every mode under three cases into "other", is the specific mechanism by which this fails in practice. Sorting by count and merging the tail is defensible if frequency determines importance, and frequency is half of what determines importance. The half it omits is exactly the half that is anti-correlated with count: severe failures are usually rare, because systems that produce severe failures frequently do not survive long enough to be analysed.

So an invented policy exception at one case lands in the same bucket as inconsistent markdown at two cases and a wrong-language answer at one. The bucket is labelled "other, 4 cases", it is the smallest bar on the chart, and every subsequent conversation about the roadmap is anchored on a picture in which the compliance exposure does not appear. Nobody decided to ignore it. The chart decided, by applying a merge rule chosen for readability.

The rule operates on the count column because the count column is the only one that reached the chart. Severity was measured, given a per-occurrence unit and written down; it was then dropped at the moment of rendering by a default that groups small categories together, and that default has no way of knowing that smallness is the property worth looking at here. The alternative that keeps the chart readable is to sort by expected cost rather than by count, and to render the constraint list separately as a short table of modes with owners and accept-or-mitigate decisions. Four bars plus three lines of text is not less readable than five bars, and it is the difference between a chart that summarises the analysis and one that inverts it.

ICE, RICE and the largest bar#

Expected cost against fix cost is not the only prioritisation frame in use, and it is worth knowing what the alternatives do to the same modes.

RICE and ICE are not wrong; they are general-purpose product frames built for a setting where impact cannot be measured directly. When you have read fifty traces and can put a rate and a per-occurrence cost on a mode, using a 1–10 impact score instead throws away a measurement you already paid for. The scores are also the most negotiable artefact in any planning meeting, since an impact of 7 versus 4 is a matter of advocacy and $8,300 a month is a matter of arithmetic somebody in finance can check.

A second objection holds even where no measurement is available. An impact score from one to ten is ordinal, which is to say its levels are ordered and the distance between any two of them is undefined, and both frames multiply such a score by other scores of the same kind. Multiplication needs those distances to mean something, since doubling a quantity has to double whatever the quantity stands for, and there is nothing about a hand-assigned 8 that is twice a hand-assigned 4 in any respect a second person could check. The anchored severity ladder these modes were priced on (chapter five, where every level is pinned to an observable consequence — the company absorbing a direct cost, say, rather than a reviewer's sense of how serious something felt) is built with uneven gaps for that reason, and nothing multiplies it either; it argues an ordering and is then set aside as soon as money is available.

The ease and effort terms deserve one caution. Dividing by effort systematically favours prompt changes over infrastructure work, because prompt changes are always cheap and infrastructure work never is. Across a few quarters that produces a system with an enormous system prompt full of patches and an ingestion pipeline nobody has touched, and the modes that keep recurring are the ones whose real fix was always the expensive one.

Five dependencies nobody has measured#

Presenting a decision like this honestly means presenting its dependencies, and five of them would change the answer.

The $40 average loss per honoured expired price is a mean supplied by finance. If it is the mean of a distribution where most cases are $8 and a few are $300, the monthly figure holds and the variance is much larger than the plan assumes, and nobody has looked at the distribution. The 8% failure rate came from 200 randomly sampled conversations read by hand, and its confidence interval runs from 5.0% to 12.6%, so every projected monthly figure carries that same factor of two and a half.

The one-month window is a single sample in time. An expired-promotion mode is mechanically tied to how many promotions run and how often they expire, so a month with a summer sale and a month without are not the same product, and a second month of data would say whether $8,300 is a run rate or a seasonal peak. The zero attached to preamble is an assumption that verbosity has no effect on conversion, task completion or support volume. Across 1,632 conversations a month, a relative conversion drop of half a percent would be worth more than the promotion mode at this product's basket value, and the basket value is another figure the analysis never writes down. Nothing in the traces could reveal it in any case, because a conversation that converted and one that did not look alike on the page, and measuring it means an experiment rather than more reading.

Finally, every fix estimate is an estimate, and fixes introduce modes of their own. A validity check that refuses to state expired promotions will, on some fraction of conversations, refuse to state a promotion that is perfectly valid, and the rate at which it does that is unknown until it ships. The same mechanism runs in the other direction and is easier to miss, since a fix changes what the system does on inputs that were never its target. A retrieval filter that drops expired rows changes what every query against that index returns, and any mode living downstream of those results moves whether or not anybody counted it. The arithmetic above treats the modes as separate lines with separate benefits, and from the moment the first fix ships they are not.

Stating those five out loud is not hedging: it converts the decision from a claim about the world into a claim about the world under stated assumptions, which is the form that survives being wrong. When the seasonality assumption turns out to be false in November, the fix is to revise one line rather than to relitigate the whole analysis.

The frame buys a defensible order rather than a correct one. Expected cost is not a truth about a mode, it is a product of two measured quantities and one sourced one, and its virtue is that each factor can be argued about separately by whoever owns it — traffic by analytics, rate by the read, cost per occurrence by finance. A number assembled that way can be checked and revised by someone who did not read the traces, which is exactly the audience the decision has to survive. The one join that cannot be checked the same way is the exchange rate in the middle, where a week of a person's time became $6,000 so that a monthly flow of loss could be set against it. That conversion is a judgement, and its only defence is that it was written down in one place where a person who disagrees can point at it.

The week is committed, the mitigation is out, the guardrail is up, and the dashboard has moved. What remains is the question of whether the mode anybody actually targeted is the one that moved, and an aggregate is the wrong instrument for answering it.