Forty percent cheaper and three points worse is not yet a decision

Two numbers arrive together often enough that they deserve a procedure. A bundle of optimisations makes an agent cheaper by some percentage and less successful by some number of points, and everyone in the room can see both figures and disagree about what to do. The disagreement is real and it is usually conducted in the wrong units, because a percentage saved and a percentage of tasks lost are not comparable quantities and no amount of conviction makes them one.

Converting them into one quantity takes two steps and both are empirical. The first is finding out what happened to the tasks that stopped succeeding, since they did not evaporate and their destinations differ in cost by two orders of magnitude. The second is dividing: a saving per task on one side, an increase in failure probability multiplied by the price of a failure on the other. What comes out is a break-even price, which is a number either side of an argument can attack with evidence rather than with seniority.

Follow the thirty tickets#

Three points of task success at 1,000 tickets per unit of accounting is 30 tickets that used to be resolved and now are not. Those 30 tickets did not evaporate. Each of them went somewhere with a cost attached, and the costs differ from each other by two orders of magnitude, so the aggregate is not knowable from the headline pair at all. Pulling those 30 out of the replay and reading their traces sorts them into four groups, and the sorting is mechanical rather than a matter of judgement, because the run record from chapter one carries ended_reason and outcome on every attempt.

The saving over the same 1,000 tickets is model spend of $138.30 before and $80.10 after, which is $58.20. The change costs about twenty-nine times what it saves.

Before doing anything with that ratio, read the total for what it hides. The entire imbalance comes from eight tickets: the wrong credits are a quarter of the lost tasks and about three fifths of the cost, because their unit price is twenty times an escalation's. A total is compatible with every distribution that sums to it, and this one is not evenly spread at all, which means the decision in front of the room is really a decision about eight tickets. That matters because eight tickets have a specific cause and might have a specific fix, whereas "the bundle costs three points" has neither.

The three destinations behave completely differently and it is worth knowing which one you are looking at before pricing anything. Retries spend the saving back and are self-limiting, because a retry policy has a maximum. Escalations cost a known amount and are visible, because somebody is paid to handle them and the volume appears on an invoice. Silent wrong answers cost whatever the wrong action costs and are invisible until something else finds them, which means the count you have is a lower bound produced by whatever detection happens to exist.

The first row being zero is deliberate and worth noting. Success here is measured after the retry policy has done its work, which is what the first chapter's definition requires, so a task that failed once and succeeded on the second attempt is inside the 88% and its extra cost is inside the $0.091. If your success rate is measured per run instead, the retry row will be large and the comparison will be wrong in both directions at once — the numerator will carry costs the denominator has already counted as successes.

The number that decides it#

Reduce all of that to one comparison. Take the change when the saving per presented task exceeds the increase in failure probability multiplied by the loaded cost of a failure. Written as a condition on the price of a failure rather than on the saving, that is: the change is worth taking when the loaded cost of a failure is less than the saving divided by the extra failure probability. The right-hand side of that inequality is a price, in the same units as the thing it is compared with, and it is the only number the argument needs.

def worth_taking(spend_before: float, spend_after: float,
                 success_before: float, success_after: float,
                 loaded_cost_per_failure: float, tasks: int) -> tuple[bool, float]:
    saving = (spend_before - spend_after) / tasks          # per presented task
    extra_failures = success_before - success_after        # as a probability
    breakeven = saving / extra_failures                    # loaded cost that ties
    return loaded_cost_per_failure < breakeven, breakeven

worth_taking(138.30, 80.10, 0.910, 0.880, 56.99, 1000)
# -> (False, 1.94)

Both inputs to that division come from the table above. The saving per presented task is $58.20 spread over a thousand tickets, and the extra failure probability is three points expressed as a fraction. Dividing gives a break-even of $1.94: a failure in this workload has to cost less than that, fully loaded, for the 40% saving to be worth three points. The measured loaded cost is $56.99, which is the $1,709.60 spread over the 30 tickets.

That is the number to put on the whiteboard, because it converts an argument about risk appetite into a question with a measurable answer that either side can attack: dispute the $131 for a wrong credit, dispute the $180 for an abandoned customer, dispute the mix. Nobody has to dispute whether 88% feels acceptable.

The rule also shows immediately when the director is right. Take the same 40% saving and the same three points on an internal code-search assistant, where a failed task means an engineer scrolls the raw results themselves at about 45 seconds of their time, or $1.10. Loaded cost $1.10 against a break-even of $1.94, so the change ships, and it should ship the same afternoon. The identical pair of headline numbers gives opposite answers in the two workloads, which is why arguing from the headline pair alone cannot converge.

The break-even column is the same in every row because it is a property of the change rather than of the workload, and the decision column varies because the loaded cost is a property of the workload rather than of the change. Keeping those two straight is what lets a result travel: a saving measured on one agent can be evaluated for another by looking up one number.

Defending the director's side properly, rather than caricaturing it, means putting the case where the loaded cost is genuinely low and the saving compounds. If the eight wrong credits were eliminated by the argument validator from chapter four — a check that the shipment id belongs to this customer's ticket would have caught all eight — the remaining twenty-two failures land as escalations and abandonments worth about $661 per thousand tickets against a $58 saving, which still loses, but only by a factor of eleven rather than twenty-nine.

Push further: if the escalation desk has idle capacity, the marginal cost of an escalation is closer to $1.10 than $6.40, and if abandonment is measured rather than assumed and turns out to be zero, the loaded cost falls under the break-even and the change ships. Each of those is an empirical claim someone can go and check, and a director who makes them is arguing correctly. What does not work is asserting that 88% is fine, because 88% of what and costing whom are exactly the unknowns.

The retry destination deserves a note of its own, since it is the one that quietly cancels a saving without changing the success rate at all. An optimisation that makes a step 30% cheaper and causes a second attempt on 20% of runs has spent roughly two thirds of its saving on the extra attempts, and the headline success number is untouched because the retries work. That change shows up as a small, confusing gap between the predicted saving and the observed one, and the way to see it is to report mean attempts per successful task alongside cost; if attempts rose, the difference between the arithmetic and the bill is sitting in that column.

Averages hide the failures that matter#

The 30 tickets are not drawn uniformly from the workload. Segmenting them by account type shows that wholesale accounts, which are 12% of ticket volume, produced 61% of the new failures and seven of the eight wrong credits. Retail tickets, with one or two shipments each, lost 0.4 points of success for the same 40% saving.

The mechanism is the one from chapter four, and it explains the concentration rather than merely reporting it. Wholesale tickets carry more shipments and longer context, and the disambiguation step that the bundle downgraded is precisely the step whose difficulty scales with the number of candidates. So the failures land where the candidates are, which is a property of the account rather than of the ticket, and account type is known before the run starts.

That reframes the decision a third time. The choice was never binary, because the change can be applied where it is clean and withheld where it is not.

The last column is the same accounting as the four-row table, applied to each arrangement: the baseline's failures are escalations at $6.40, and the bundle's are those plus the 30 expensive ones, which is where the jump between the first two rows comes from. The segmented rollout keeps 35 of the 40 points of saving and gives up 0.2 points of success instead of 3.0.

It is more configuration to maintain and it requires the routing decision to be made on an attribute known before the run starts, which account type is. Reaching it requires the segmentation to have been in the evaluation from the beginning; an A/B test that reports one aggregate number per arm, an arm being the whole set of tickets run under one of the two configurations, can only ever produce the argument the meeting started with.

Running the comparison so the numbers mean something#

None of the above survives a badly run experiment, and the failure modes are ordinary. The success criterion has to be fixed before either arm runs and applied identically to both, which for this workload means the ticket was resolved, the credit reconciled against the correct shipment, and no reopen within 24 hours. Graders have to be blind to the arm, or the criterion has to be automated end to end; a reviewer who knows which arm produced a reply reads it differently.

The arms have to run concurrently on randomised tickets rather than in consecutive weeks. Chapter seven's Monday is the reason: traffic mix, rate-limit pressure and carrier behaviour all vary by day, and a week-over-week comparison attributes those to the change. Randomising at the ticket level, with the assignment stored on the run record, also makes the segmentation above possible after the fact.

Sample size is the constraint people discover last. Detecting a difference between 91% and 88% with 80% power — an eight-in-ten chance that the experiment registers the difference when the difference is genuinely there — at the conventional significance level needs roughly 1,600 tickets per arm; the pilot that produced these numbers first ran 300 per arm, where the 95% interval around a three-point difference comfortably includes zero. A pilot that size can tell you the change did not catch fire, and it cannot tell you whether success moved, so treating its point estimate as a result is how a regression reaches a decision meeting dressed as a fact.

# tickets per arm to detect p1 -> p2 at 80% power, alpha 0.05 (two-sided)
def per_arm(p1: float, p2: float) -> int:
    z_a, z_b = 1.96, 0.84
    p_bar = (p1 + p2) / 2
    return ceil(2 * (z_a + z_b) ** 2 * p_bar * (1 - p_bar) / (p1 - p2) ** 2)

per_arm(0.910, 0.880)   # 1568
per_arm(0.910, 0.900)   # 14108  -- a one-point regression needs 9x the traffic

The shape of that formula is worth reading even if the constants are not. The difference you are trying to detect appears squared in the denominator and nowhere else, so the sample size scales with the inverse square of the effect. Cut the effect to a third — three points down to one — and the required traffic multiplies by nine, which is exactly the relationship between the two outputs above. Nothing about the model, the workload or the criterion changes that; it is a property of the arithmetic.

The arithmetic has a domain#

The last row of the workload table is there because the formula is for workloads where failures are recoverable and their cost is genuinely economic. Where a failure carries regulatory, safety or legal consequence, the loaded cost is not a number anyone should be producing from a spreadsheet, and the decision is made by a different process. Producing a figure there is worse than declining to, because a number in the right units invites the comparison the domain forbids.

The second limit is detectability, and the second line of that listing is the uncomfortable one. Regressions smaller than about two points are effectively undetectable at the volumes most agent workloads run at, which follows directly from the inverse-square relationship above: a one-point regression already needs nine times the traffic, and a half-point regression needs thirty-six. So a sequence of individually invisible one-point regressions can take a system from 91% to 85% over two quarters with every individual change having passed its evaluation and none of them having been wrong about its own result.

The defence is a fixed evaluation set replayed against the current configuration on a schedule and compared against the original baseline rather than against last month, so drift accumulates in a number somebody sees. That comparison is cheap and it is the only one that can detect an effect no single experiment is powered for, because the effect it measures is the sum of them.

The levers in this topic are now measured rather than believed: caching removed 64% of input tokens without touching behaviour, tiering removed a third of the remainder and cost nine points of success when applied to the wrong step, a checkable stopping rule removed 15% and truncated 0.9% of the runs it fired on, a budget cost 2% and bounded a run that wrote 118 notes to a customer account, and fan-out bought five seconds of median latency for 13% more spend and a race that took eleven days to surface.

Using them is an ordering rather than a menu, and the ordering follows from what each one needs in order to be evaluated. Instrument cost and time per successful task first, because every later number is expressed in those units. Take the changes that cannot alter behaviour next — caching, breakpoint placement, removing a redundant tool call — since they need no ablation and no argument. Then ablate one step at a time against a fixed criterion, segment the results before aggregating them, and price a failure in the workload you actually have, so that when the meeting happens the decision turns on a number rather than on who is more senior.