Conditions that justify an LLM premium

The instinct that high volume justifies investing in a model is imported from a different kind of engineering, and in that setting it is correct. When the thing you are buying has a large fixed cost and a near-zero marginal cost — a cache, an index, a compiled query plan — volume is exactly the argument for buying it, because the fixed cost amortises over more units.

The word doing the work in that sentence is amortises, and it is worth unpacking because the whole chapter turns on where it applies. A cache is built once. Whatever it cost to build, that cost does not recur, so the cost per request is the build cost divided by the number of requests — a fraction whose denominator grows and whose numerator does not. Ten times the traffic makes each request a tenth as expensive to have served. Nothing about that argument is about caches specifically. It follows from one property: the money was spent before the requests arrived, and no request causes any more of it to be spent.

An LLM call inverts that structure. The fixed cost is a few engineer-days of prompt and schema work, and the cost of the thing itself is charged per invocation, so each additional unit of volume adds to the bill instead of diluting what is already spent. The break-even arithmetic behind that has already been worked through: the arm with fixed costs wins by more as volume rises, and the arm with per-call costs wins only at volumes low enough for its fixed costs to matter. What follows from it is a set of conditions for spending the premium anyway, and they are narrow enough to list and check.

The bill on two jobs of very different size#

Take two extraction jobs. Job A reads notification emails from a logistics partner and pulls out the order ID, the ship date and the carrier; the partner's template has not changed in three years, and the job runs about 2 million times a month. Job B reads the free-text box where a customer explains why they are cancelling, and maps it onto a taxonomy of cancellation reasons that a retention team revises roughly monthly; it runs about 40,000 times a month.

Job A's emails run about 900 tokens each once the quoted thread is stripped, and an extraction call at that size costs somewhere near $0.0009 all in, so 2 million a month is $1,800 a month or $21,600 a year, forever, to parse a document whose structure has been stable since before the current team joined. Job B's free-text reasons are short, around 120 tokens, and 40,000 calls a month at $0.0004 is $16 a month. The job with fifty times the volume costs a hundred and twelve times as much to serve with a model, and it is the job where a model buys the least.

Both of those bills are one multiplication, and both are worth doing on paper because the comparison at the end of them is the point of the section. Two million calls at nine ten-thousandths of a dollar each is eighteen hundred dollars a month, and twelve of those months is $21,600 a year. Forty thousand calls at four ten-thousandths of a dollar each is sixteen dollars a month, which is under two hundred a year. Now divide the two bills into each other. Job A costs a hundred and twelve times what Job B costs. Divide the two volumes into each other and Job A is fifty times larger. The leftover factor of a bit over two is the ratio of the per-call prices, and that is the entire story: the bill is volume multiplied by price, so the ratio of two bills is the ratio of the volumes multiplied by the ratio of the prices, and neither factor ever divides out. There is no denominator anywhere in that expression for the volume to grow into.

One thing not to read off those figures is a rule connecting input length to price. Job A's emails are more than seven times the size of Job B's free-text boxes, and its calls cost a bit over twice as much. A call's price is not the input alone: the instructions, the response schema and the generated output are all charged too, and none of them shrink when the input does. The consequence for planning is that a job's bill has to be measured on that job's own calls rather than scaled from another job's, and the consequence for this comparison is that even a much shorter input would not have rescued Job A. Halve its per-call price and it still costs an order of magnitude more than Job B, because the fifty is doing most of the work.

What actually earns the premium#

The first condition is a label space that keeps moving. Changing the label set is expensive for a trained classifier in a specific way: new classes have no coefficients, so you need labelled examples of them, which means a relabelling pass over historical data before a retrain can even start. If a retention team splits too_expensive into price_increase and found_cheaper_alternative on the first of the month, the classifier's path to supporting that is a labelling job, a fit, an evaluation and a deploy — call it a week and a half of elapsed time with two people involved. The model's path is an edit to the enum in the response schema and a sentence in the prompt describing the distinction, deployed the same afternoon. When the revision cadence is monthly and the retrain cadence is six weeks, the trained artifact is never describing the current taxonomy. That is the justification that holds, and it is a statement about cycle time rather than about accuracy.

"Never" there is arithmetic and not rhetoric, which is worth checking rather than accepting. A revision lands roughly every four weeks. A retrain takes about six. So a retrain that starts the day a revision lands finishes two weeks after the next revision has already landed, and the artifact it publishes is describing a taxonomy that was superseded before it shipped. Start the retrain later and it is further behind, never closer. The gap does not accumulate without bound, because each retrain catches up to some recent state, but it never reaches zero either: the artifact's lag is the retrain duration, permanently. That is the shape to look for in any comparison of this kind — not "is the trained model accurate" but "is its cycle time shorter than the thing it is tracking". Where it is not, its accuracy on the current taxonomy is unmeasurable, because the current taxonomy has no labelled data yet.

The second is semantic variation that no pattern tracks. Cancellation reasons are written by people who are annoyed, in their own words, at whatever length they feel like: "honestly it's just not getting used, my partner signed us up and neither of us opens it", or "the price went up and I found the same thing for £6". Neither contains the vocabulary of its label, and the second requires knowing that a competitor comparison and a price complaint are different reasons even though both mention price. That is the same failure mode a linear model has on anything needing knowledge outside its own corpus, and it is where the published head-to-heads consistently favour the model.

The third is an open-ended label space, meaning one where the set of possible outputs is not enumerable in advance. Normalising a free-text job title onto a canonical role, or summarising a cancellation reason into a phrase for a dashboard, has no fixed class list to train against. This is worth keeping separate from the first: a moving taxonomy is one where the classes change, an open-ended one is where there is no class list at all.

None of the three is volume, and none of them is difficulty. A task can be hard for a human and still be perfectly served by a trained model, because "hard" and "unstable" are unrelated properties.

The last row decides it. Job A's failure trigger is discrete and observable: the partner changes the template, and every message after the change fails the same way at the same moment, which is the easiest kind of failure to detect and alert on. Job B's trigger is continuous and scheduled, and there is no moment at which a trained model breaks loudly — it just describes last quarter's taxonomy while the report is built against this quarter's.

The parser, and the three checks that verify it#

Job A gets a parser, and the parser is not a heuristic. The template is fixed, so the extraction is a set of anchored patterns with a validator on each field.

import re
from datetime import date

ORDER = re.compile(r"^Order(?: number)?:\s*(ORD-\d{8})\s*$", re.M)
SHIPPED = re.compile(r"^Shipped(?: on)?:\s*(\d{4}-\d{2}-\d{2})\s*$", re.M)
CARRIER = re.compile(r"^Carrier:\s*(DHL|UPS|Evri|Royal Mail|DPD)\s*$", re.M)

CARRIERS = {"DHL", "UPS", "Evri", "Royal Mail", "DPD"}

def parse(body: str) -> dict[str, str | date]:
    order = ORDER.search(body)
    shipped = SHIPPED.search(body)
    carrier = CARRIER.search(body)
    if not (order and shipped and carrier):
        raise TemplateMismatch(missing=[
            name for name, m in
            (("order", order), ("shipped", shipped), ("carrier", carrier))
            if m is None
        ])
    ship_date = date.fromisoformat(shipped.group(1))
    if ship_date > date.today():
        raise FieldOutOfRange("shipped", shipped.group(1))
    return {"order_id": order.group(1),
            "ship_date": ship_date,
            "carrier": carrier.group(1)}

Three details in that code are load-bearing and none of them is the pattern text. The first is the anchoring: every pattern starts at ^ and ends at \s*$ under re.M, so a match has to occupy a whole line. A forwarded logistics email carries the quoted thread below it, and an unanchored search for an order number would happily find one in last week's quoted message. Requiring the line to be the field rather than to contain it is what makes the match a claim about structure. The second is that the function raises on a missing field instead of returning a partial result — three fields are searched, and if any one of them is absent the caller is told which, by name. The third is the range check on the date. A mis-parsed YYYY-MM-DD string is still a valid date, so the only cheap thing the parser can say about it is that a shipment cannot have left in the future, and it says exactly that and nothing more.

Knowing it is good enough is three separate checks rather than one accuracy number. The first is the same fixed evaluation set the model arm would have been scored on, labelled independently, with per-field precision and recall — for a job like this the honest result is 100% on every field, and anything below that means the template is not as fixed as the partner claims. The second is a live no-match rate exported as a metric, because the evaluation set is a curated snapshot and the population is not; a parser that raises TemplateMismatch on 0.02% of traffic is healthy, and one that starts raising on 3% has found a template variant nobody mentioned. The third is a small canary: send 500 messages a day to the model as well, diff the fields, and alert on disagreement. That canary costs about $13 a month, which is the entire model budget for this job reduced to a monitoring line, and it converts "the template has not changed" from an assumption into an observation.

The canary's price is the same multiplication as before with a much smaller first factor. Five hundred messages a day is fifteen thousand a month, and fifteen thousand calls at nine ten-thousandths of a dollar is a little over thirteen dollars — against the eighteen hundred a month the same model would cost on all of the traffic. Sampling is what buys that: a template change affects every message, so a sample large enough to contain one is large enough to find it, and there is no reason to pay for the other ninety-nine in a hundred.

There is a specific failure the canary catches and the parser cannot, and it is the reason this is a third check rather than a redundant one. The three regexes verify shape. If the partner renames a header or reorders the fields, the pattern stops matching, the function raises, and the alert is immediate — that is the loud, detectable event the table's last row promises. What the patterns cannot see is a change in meaning behind an unchanged shape. Suppose the partner begins putting the estimated delivery date in the Shipped: line. The line still starts with Shipped:, still holds a well-formed YYYY-MM-DD string, and the value is still in the past on most messages, so it passes the pattern, passes date.fromisoformat and passes the range check. Every defence in the parser is a defence about form, and this change is about reference. Only something that reads the message and reaches an independent conclusion can notice, which is what the model in the canary is for. This also names the limit of the "loud failure" argument used elsewhere in this chapter: a parser fails loudly on the failures it can see, and quietly on the one class it cannot.

Even that live no-match rate is a shape check, and the two thresholds are worth converting into messages. At two million a month, a rate of 0.02% is about four hundred messages — small enough to be the ordinary tail of malformed forwards and autoreplies. A rate of 3% is around sixty thousand, which is nobody's tail. The gap between "healthy" and "investigate" is not a matter of a few percentage points of judgement; it is two orders of magnitude in the count, which is why a threshold set anywhere in between will do.

The raise in that code is doing more work than the regexes. A parser that returns None for a missing field hands the downstream system a value that looks like an answer, and that specific failure is the subject of the next chapter.

One code path, two jobs#

The predictable counter-proposal is to put the model on both jobs, on the grounds that one code path is cheaper to maintain than two. The maintenance intuition is correct in general and does not survive the specifics.

What the shared path actually saves is one deployment target, one set of credentials and one retry implementation, which is perhaps a quarter of an engineer-day a month. What it costs is $21,600 a year for Job A, roughly twenty-seven engineer-days at the rates the ledger used. The trade is not close, and it does not become close at any volume, because the saving is fixed and the cost scales with traffic.

Putting both sides in the same unit is what makes that obvious, and the conversion is one division each. A quarter of an engineer-day a month is three engineer-days a year. Job A's model bill of $21,600 divided by the day rate the previous chapter's ledger used comes to twenty-seven of them. Three against twenty-seven: the shared code path saves about a ninth of what it costs, in the same currency, in the same year. And the two numbers move differently. Triple Job A's traffic and the saving is still three engineer-days, while the cost becomes eighty-one — the gap widens with every unit of the quantity the proposal cites in its own favour.

The claim that it is one code path is also not quite true. Job A's job is to parse a fixed template and Job B's is to interpret free text, and those have different prompts, different schemas, different evaluation sets, different failure semantics and different quality bars, so what they share is the client library. Both still need the retry, rate-limit and dead-letter handling that the ledger costed at two engineer-days plus ongoing maintenance, and Job A does not need any of it today, because a local function call does not get rate-limited.

The part that most deserves a direct answer is what happens to Job A's failure mode. A parser fails loudly, on the field, at the moment the template changes, and the exception carries the name of the field that stopped matching. A model handed a changed template usually keeps working — it finds the ship date wherever the partner moved it — until the day it silently returns the delivery estimate instead, in a valid YYYY-MM-DD string, on a subset of messages. Deterministic errors are reproducible and clustered; that general property becomes concrete here, and it means the model version of Job A would be both more expensive and harder to operate.

The premium buys implementation lag and nothing else#

Paying per call for a moving taxonomy buys the ability to change the prompt today. When the retention team splits a label, the prompt can be updated the same day, but the evaluation set still describes the old taxonomy, and until somebody relabels the affected slice you have a system whose current quality is unmeasured. That relabelling is smaller than a classifier's — needed for measurement rather than for training, so hundreds of examples instead of thousands — but it is not zero, and the failure mode of teams who move to a model for this reason is a prompt edited eleven times against an evaluation set last touched in March.

The premium also does not make flexibility free at the point of change. Adding a fourteenth label to an enum can move behaviour on the other thirteen, because the classes are defined relative to each other in the prompt, and a class described as "the customer found a cheaper alternative" will pull examples away from "too expensive" in ways that only show up when the evaluation is re-run. Prompt edits are cheap to deploy and not cheap to verify, and the verification is the part that has to stay in the schedule.

None of this is an argument about model quality. Job A is not too easy for a model; a model would do it perfectly. It is that a job with no semantic variation, no moving label space and 2 million monthly executions is the exact profile where per-call pricing is worst and determinism is worth most, and that profile is readable from a table before anything is built.

Job A's parser now has to actually replace something, because there is usually a model already in production doing this work while somebody argues about whether to keep it. Swapping it out has a specific hazard the evaluation set will not show you: the two systems can agree on every field of every row in the set and still behave differently the first time an input arrives that neither the pattern nor the lookup table covers.