Total cost per decision

Cost comparisons between an LLM classifier and a trained one are usually settled with two figures that are both correct: the provider's price per call, and the observation that the marginal cost of one predict call on a sparse matrix is somewhere around a millionth of a dollar of CPU time. The first of those is a fully loaded cost — it includes everything that had to happen for that input to get a label, because there is nothing else to buy. The second is a marginal cost, the price of one more unit given that the thing already exists, and it includes none of the work that made a predict call possible.

The reason this particular mismatch is so hard to see is that neither figure is wrong and neither party is being dishonest. A price per call really is the total price of a call: the provider has already absorbed the model training, the serving fleet and the on-call rota, and the arithmetic that turns those into four ten-thousandths of a dollar happened inside somebody else's business. A marginal cost of nearly nothing is equally real; running the sparse multiply one more time genuinely costs almost nothing. The error is not in either number, it is in the subtraction. Dividing one by the other answers the question "if both systems already exist and are already maintained, which is cheaper to invoke", and nobody in the room is asking that. They are asking which is cheaper to have.

That mismatch — a fully loaded cost divided by a marginal one — is what produces the claim that a classifier is three orders of magnitude cheaper. Building the honest version means picking a horizon, listing every line item that would not exist if you did not ship, and dividing by the decisions made over that horizon. Twelve months is the right horizon because it captures at least one taxonomy change and at least one model deprecation, both annual-ish events that a monthly view makes invisible.

The horizon choice is not a convention and it is worth walking, because getting it wrong is how a comparison ends up flattering whichever arm has lumpy costs. Costs on both sides come in two shapes: ones that recur every month, like maintenance, and ones that arrive once or twice a year, like a provider deprecating a model or a policy team splitting a class. Run the comparison over one month and the lumpy costs are either absent — because that month happened not to contain one — or they land whole, inflating that month by the cost of an event that happens twice a year. Neither reading is representative. A horizon long enough to contain at least one of each lumpy event lets it be counted once at its true frequency, and twelve months is the shortest horizon for which that is true of both the taxonomy change and the model deprecation. The discipline that follows from it is that every figure below this point is annual. A rate given per month gets multiplied by twelve before it enters the table, and the single most common way this exercise goes wrong is a monthly rate carried into an annual total without that step.

Putting both sides in the same column#

The rates below are one engineer-day at $800 fully loaded and one analyst-hour at $45, which are assumptions rather than facts and should be replaced with your own. Everything else follows from a volume of 1.8 million decisions a month, or 21.6 million a year.

Every cell in that table is one multiplication, and doing three of them is enough to make the rest checkable. An engineer-day row is days times $800: three days of initial build for the prompt arm is $2,400, five for the classifier is $4,000. A per-month row is the monthly figure times twelve, which is the step the previous section insisted on: half an engineer-day a month of prompt maintenance is six days a year and therefore $4,800, and a quarter-day a month of feature maintenance is three days and $2,400. A labelling row is labels times seconds, converted to hours, times $45: three thousand labels at forty-five seconds each is a hundred and thirty-five thousand seconds, which is thirty-seven and a half hours, which is the $1,690 in the classifier column. The retry row is the only one that adds two shapes together — two engineer-days to build the queue and the backoff, which happens once, plus a quarter-day a month to keep it working, which happens twelve times.

The inference row is the one that behaves differently from all the others, and it is the reason the exercise is worth doing at all. Every other row is priced in somebody's time and does not care how many tickets arrive. This one is 21.6 million decisions multiplied by $0.0004, which is $8,640, and it is the only figure in either column that moves when traffic moves. Volume enters the LLM arm once, here, and enters the classifier arm nowhere: the sidecar costs $300 whether it labels one ticket or twenty million, until it needs a second one. Hold on to that asymmetry, because the next section is entirely about what it implies.

The bottom two rows are the totals divided by the year's decisions. The LLM column sums to $26,260, and $26,260 spread over 21.6 million decisions is a little over $0.00122 each. The classifier column sums to $17,290, which over the same 21.6 million comes to a hair under $0.00080. Dividing one by the other gives about 1.5.

The per-call gap was 400 times. The all-in gap over a year is about 1.5 times. That is the single most useful thing the exercise produces, because a 1.5× difference is small enough that quality, latency and operational risk can legitimately outweigh it, whereas a 400× difference ends the discussion before those factors get a hearing.

Both ratios come out of the same division and it is worth seeing them side by side, because the collapse between them is the entire result. The first divides $0.0004 by roughly a millionth of a dollar, which is a factor of about four hundred. The second divides $0.00122 by $0.00080. What moved is not the price of a call, which is unchanged, but what got counted as part of one: the numerator gained the engineer-days and the migration and the retry queue, and the denominator gained everything the classifier's zero was hiding. Two orders of magnitude vanished without a single figure being disputed, which is why the useful challenge to a cost slide is never "is that price right" and always "what is in that price".

Four of those rows deserve a note. Relabelling costs the classifier less per event than people expect, because a policy change usually affects one or two classes rather than the whole taxonomy, and 800 targeted labels is two analyst-days rather than a project. Provider version migration is a real recurring cost the LLM arm cannot avoid: when a model is deprecated with six months' notice, somebody re-runs the evaluation on the successor, discovers that two of the twelve few-shot examples now push the model toward the wrong class, and spends three days including the relabelling of disagreements. Drift monitoring appears on both sides and is the standing cost of checking that live traffic still resembles the traffic the evaluation set describes — the class mix, the length and vocabulary of the inputs, and the rate at which the system meets something it handles badly. Retry handling is not optional infrastructure: at 1.8 million calls a month, even a 0.2% transient failure rate is 3,600 inputs that need a queue, a backoff and a dead-letter path, and none of that exists on the classifier side because a local function call does not fail intermittently.

The retry row is the one worth following link by link, because it is the row teams are most surprised to owe and the reasoning generalises past this comparison. A hosted classifier is reached over a network. A network call can fail for reasons that have nothing to do with the input — a dropped connection, a rate limit, a provider incident — and a failure of that kind is transient, meaning the same input would have succeeded a moment later. A ticket that failed transiently therefore has no label and is not wrong, it is missing, so it cannot be dropped and it cannot be counted as an error either. Something has to hold it and try again, which is a queue; something has to decide when to try, which is a backoff; and something has to hold the ones that keep failing, which is a dead-letter path. Three components exist because one call crosses a machine boundary. At 1.8 million calls a month, a failure rate as low as 0.2% is 3,600 inputs a month arriving in that machinery, which is well past the volume at which somebody could handle them by hand.

The classifier side owes none of that, and the reason is worth stating precisely rather than as folklore. An in-process function call does not fail intermittently: it either returns, or raises deterministically on that input, or the process is gone and every call is failing at once. There is no state in which the same input succeeds on the second attempt for reasons outside the program. That is what removes the queue, the backoff and the dead-letter path together, and it is a property of where the code runs rather than of what kind of model it is. Deploy the classifier behind an HTTP service and the row comes straight back.

The break-even is a volume, and you should know yours#

Nothing in that table is a property of the two techniques. It is a property of the two techniques at 1.8 million decisions a month, and the ranking can flip at either end of the volume range, because one arm's costs are almost entirely fixed and the other's are almost entirely variable.

That sentence is close to true and the code below shows exactly how close, which is the more interesting result. The classifier arm's variable cost is zero and everything it spends is fixed. The LLM arm is the one people describe as pay-as-you-go, and its fixed costs come to $17,620 a year — within a few hundred dollars of the classifier's. Nearly all of that is engineer-days, because building a prompt, migrating it when the provider deprecates a model, and standing up a retry path are all somebody's time and none of them get cheaper if traffic halves. The intuition that a prompt has no fixed costs because there is no training run does not survive being itemised. What the LLM arm has that the classifier does not is a variable cost on top of a comparable fixed one.

LLM_FIXED = 2_400 + 20 + 4_800 + 1_600 + 2_400 + 4_000 + 2_400   # $17,620/yr
LLM_PER_CALL = 0.0004
CLF_FIXED = 4_000 + 1_690 + 300 + 2_400 + 3_200 + 900 + 4_800    # $17,290/yr
CLF_PER_CALL = 0.0

def annual_cost(volume_per_month):
    n = volume_per_month * 12
    return (LLM_FIXED + n * LLM_PER_CALL, CLF_FIXED + n * CLF_PER_CALL)

# volume at which the two annual totals would meet
breakeven_per_month = (CLF_FIXED - LLM_FIXED) / (LLM_PER_CALL * 12)

The expression on the last line is worth deriving rather than accepting, because its sign is the finding. Each arm's annual cost is a straight line in volume: a fixed amount, plus a per-call price multiplied by twelve months of that volume. The two lines meet where the totals are equal, so setting them equal and solving leaves the difference in fixed costs divided by the difference in per-call prices — and since the classifier's per-call price is zero, that denominator is just the model's price times twelve. The numerator is the classifier's fixed costs minus the model's.

With these assumptions that expression comes out at about −69,000 a month, and the minus sign is the result rather than an artefact: the classifier's fixed costs sit $330 a year below the model's, so the two lines never cross at any volume you could actually run and the classifier is cheaper everywhere. What changes with volume is only how much cheaper.

A negative break-even is not a failure of the calculation and it is worth saying why, because it looks like one. The formula asks how much extra volume it would take for the model's per-call spending to make up a fixed-cost advantage it does not have. Since the model starts behind on fixed costs and pays more per call, there is no positive volume at which it catches up — the only solution is a volume in the past. Read the sign first and the magnitude second. A positive break-even is a threshold to compare your traffic against; a negative one means the arms are ordered the same way at every volume and the only remaining question is how large the gap gets.

Every row of that table is the same two lines evaluated at a different point, and working one row makes the shape obvious. At twenty thousand decisions a month the year holds two hundred and forty thousand of them; multiplied by $0.0004 that is under a hundred dollars of inference, added to the model's $17,620 of fixed costs. The classifier column does not move at all, because none of its costs depend on volume. At a thousand times that traffic the inference term has grown to nearly a hundred thousand dollars and now dwarfs the fixed costs it is added to, while the classifier column is still $17,290. The ratio column is one divided by the other, and it climbs from barely above one to well past six for the same reason a sum of a constant and a growing term eventually looks like the growing term alone.

Two readings follow. At small volumes the arms are indistinguishable on cost, which means a team processing forty thousand documents a month should choose on quality, iteration speed and operational simplicity and stop modelling the money. At large volumes the variable cost dominates everything else and the classifier's fixed costs become a rounding error, which is the arithmetic behind the claim, developed two chapters from now, that high volume is a reason to avoid per-call pricing rather than a reason to justify it.

The other reading is that a ledger with three significant figures is fiction. The engineer-day rate is a guess, the label rate depends on who is labelling, and the maintenance rows are estimates of future behaviour. What you want out of this exercise is an order of magnitude and a crossover volume, and if the crossover sits within a factor of three of your actual traffic, the honest conclusion is that cost does not decide this and something else must.

Costs that belong to neither column#

Some line items get claimed by whichever side is arguing, and they should be struck from both. The independently labelled evaluation set is the largest: you need it to report either arm's accuracy, so its twenty-five person-hours are the cost of having a measured system rather than a cost of the classifier. The written label policy is the same. So is the human review queue for low-confidence or failed cases, which both arms need.

Charging shared costs to the trained-classifier column is how a lot of these comparisons quietly get their result, because the labelling line is the one that sounds expensive. The 3,000 training labels are a genuine classifier-only cost; the 500 evaluation labels are not.

The test that sorts a line item into one column or neither is a single counterfactual, and applying it mechanically is more reliable than arguing about it. Ask what happens to this cost if the arm is not shipped. Drop the classifier and the 3,000 training labels are never produced, so they belong in its column. Drop the classifier and the 500 evaluation labels are still produced, because the other arm's accuracy figure needs them too, so they belong in neither. The same question strikes the label policy, the human review queue and the monitoring dashboard from both columns. It is worth being strict here, because the temptation runs one way: labelling is the line that sounds expensive, so it is the line an advocate for the model arm reaches for, and quietly moving shared labelling costs into the classifier column is enough on its own to reverse the ranking on a ledger this close.

Errors cost differently depending on their shape#

Both arms make mistakes, and a ledger that stops at the error rate misses that the same rate can cost very different amounts to live with. A logistic regression's errors are deterministic and reproducible: the same input produces the same wrong label every time, so a reported misroute can be reproduced in a unit test, the feature driving it found, and the gap either closed with training examples or accepted. Fixing it fixes every future instance of that pattern.

An LLM's errors are distributed across inputs that share no obvious surface feature, and they are not stable across model versions or, at nonzero temperature, across calls. Reproducing a reported misroute may require the exact input, and a prompt edit that fixes it may move three other cases nobody was watching, which is why the evaluation set has to be rerun on every prompt change rather than eyeballed. The per-error handling cost is therefore higher for the model arm even at an equal error rate, and it shows up as engineer-hours in the maintenance rows rather than as a separate line.

There is a countervailing point that belongs in the same paragraph rather than a triumphant one of its own. The classifier's errors are systematic, which means they cluster, which means a whole category of input can be wrong at once and stay wrong for a quarter — a new product name absent from the vocabulary is the standard example. Systematic errors are cheaper per instance and more dangerous per incident.

A ledger converts "essentially free" into "$17,290 a year and 1.5× cheaper at current volume", which is a much better sentence, and it still leaves the decision open, because the two arms differ by 2.4 accuracy points and nothing in the cost model says whether those points are worth $9,000. It also does not survive contact with change on its own. Every number above is conditional on the volume, the taxonomy, the model's price list and the input distribution staying roughly where they are, and each of those has moved for somebody in the last year: provider prices for a given capability tier have fallen substantially year over year, which pushes the crossover volume up, and a product change that doubles volume pushes it down.

A ledger computed once and pasted into a document becomes wrong quietly, and the failure mode is not that somebody notices it is stale but that nobody remembers which assumptions it rested on. That is the gap the next chapter closes. The numbers you have just assembled are only useful to a future reader if the thresholds they were compared against — the quality bar, the traffic assumptions, the latency budget — are recorded next to them, so that when one of those assumptions breaks, reopening the decision is a small documented job rather than a fresh argument.