A saving without a quality delta is not a result

The quarterly slide said model routing: −41% inference spend, with the underlying figures in the notes — $47,200 of baseline-equivalent model cost across the routed surfaces, $27,850 actually spent, $19,350 saved. Nobody disputed the arithmetic, and the next slide asked for routing to be extended to the two remaining surfaces in Q3.

The question that should have been asked first is what the 41% bought and where. That number aggregates across three products with different traffic mixes and different tolerance for a weak answer, in exactly the same way that a mean judge score aggregates across easy and hard requests, and it hides the same kind of thing. Broken out, the same quarter looked like this:

The headline is a weighted average over one surface that saved a great deal, one that saved moderately, and one that was never routed at all. Extending routing to the search surface on the strength of "41%" is reasoning from a number that contains no information about the search surface. A cost report and a quality report have the same aggregation disease, and the cure is the same in both cases: publish the slices, and put them side by side.

The two halves have to be on the same row#

The reason this is an organisational problem as much as a measurement one is that the two numbers usually live in different places and belong to different people. Spend comes from the provider's billing export into a finance dashboard, refreshed monthly, owned by whoever owns the infrastructure budget. Quality comes from an evaluation harness, run on demand before releases, owned by the team that builds the assistant. Neither dashboard contains the other's number, so the trade between them — a certain amount of money against a certain amount of answer quality — is never made by any individual person. It gets made by default, in one direction, because the cost number is available every month and the quality number requires somebody to go and run something.

Putting them on the same row is what forces the trade into the open. It works the way an error budget does: chapter 15.3 established the pattern of a measured allowance that spending decisions are made against, and the equivalent here is a stated quality allowance per slice, published beside the saving that slice produced. A saving that consumes more than its allowance is a regression with a discount attached, and it should read that way on the page.

For the support assistant, with continuous sampled judging running on production traffic, the quarter looked like this:

The last column is the one that makes the decision. Routing the FAQ and order-status traffic bought several thousand dollars a quarter for no detectable quality cost at all, and routing the extraction workflow bought $382 a quarter at a cost of 0.85 rubric points on 108,000 requests — under $450 per point of quality surrendered, against more than a hundred thousand dollars per point on the billing slice. Ranked that way, the extraction decision is obviously wrong and the FAQ decision is obviously right, and neither conclusion is visible from a 41% headline or from a 4.31-against-4.34 aggregate. The ratio also gives a principled way to answer "should we route this new surface", which is to estimate both terms rather than one.

Two cautions about that column. It is undefined and should be left blank when the quality delta is within its own confidence interval, because dividing by a number indistinguishable from zero produces a spectacular ratio that means nothing. And it should never be used to justify a slice that falls below the acceptance bar in absolute terms; a task that drops from 4.1 to 3.6 is unacceptable at any price, and the ratio is only a tool for choosing between acceptable options.

What the report has to contain to be buildable#

The report above is a rollup of per-request records, and it is only possible if the records carry the labels to group by. That means deciding, at request time, to emit the routing decision alongside everything else on the request's tracing span — the per-request record your observability stack already writes, one row per request with whatever attributes you attach to it:

def emit(req, result, span):
    span.set_attributes({
        "llm.task":              req.task,             # FAQ, extraction, ...
        "llm.difficulty_band":   band(req.input_tokens),
        "llm.route":             result.route,         # "cheap" | "escalated" | "pinned_strong"
        "llm.escalation_reason": result.reason,        # "signal" | "schema" | "verifier_timeout" | None
        "llm.signal_value":      result.signal_value,
        "llm.models":            result.models_called, # ["small-8b@2024-11", "large@2025-03"]
        "llm.tokens_in":         result.tokens_in,
        "llm.tokens_out":        result.tokens_out,
        "llm.cost_usd":          result.cost_usd,      # summed across every call made
        "llm.latency_ms":        result.latency_ms,
        "llm.shadow_sampled":    result.shadow,        # in the 1% dual-run cohort?
    })

Cost belongs on the span rather than being reconstructed from the billing export, because the billing export cannot tell you which of the two calls on an escalated request was which, and the ratio in the table above requires per-task attribution. Computing it in-process from token counts and a price table is a few lines and stays correct as long as somebody keeps the price table current.

The quality half comes from a continuous shadow sample: a fixed 1% of production requests are also run through the strong model out of band and both answers judged, using the harness from chapter 5 with the same pinned judge. At 1.2 million requests a month that is 12,000 extra strong-model calls at $0.0054, plus 24,000 judge calls at roughly $0.0004, which comes to about $75 a month — around a seventh of what the false escalations from chapter 4 cost, and the only mechanism in the system that can notice a quality drift before a customer does. Stratify that 1% rather than sampling uniformly, for the reason chapter 5 gave: a uniform 1% gives the extraction workflow 360 rows a month and the FAQ traffic 4,900, which is the wrong way round relative to where the risk lives.

The operational half is what chapter 15.3's machinery already handles, with three indicators that need to be present specifically because this is a cascade: escalation rate per task, latency on the direct path, and latency on the escalation path. Chapter 2 showed why aggregate p95 alone is unreadable here — it moves when escalation rate moves without any component slowing down — so the escalation rate has to be on the same chart as the latency it drives.

Per-request spend is harder to compute than it looks#

The cost column above is the one people assume is trivial and it is the one most often wrong, because a modern inference bill has several terms that do not scale with the token counts on the span.

Cached prompt tokens are the largest of them. If the system prompt and the retrieved policy documents are shared across requests and served from a prompt cache — the provider keeping the processed form of a repeated prefix on its side so it does not have to be run through the model again — those tokens bill at a fraction of the normal input rate — often a tenth on a cache hit, with a surcharge on the write — and the cache hit rate differs between the two models because the cheap path is called on every request while the strong path is called on 12%, so the strong model's cache is colder and its effective input price is higher than the rate card suggests. A cost model that multiplies total input tokens by the list price overstates the cheap path and understates the escalation path, which biases the saving upward in exactly the direction that flatters the project.

Batch and off-peak discounts do the same thing in the other direction for asynchronous surfaces, where the strong model can often be called at half price with a several-hour completion window. That changes the economics of a cascade substantially: a document pipeline that can batch its escalations is trading quality against a strong model that costs $0.0027 rather than $0.0054, which moves the break-even escalation rate and can make routing not worth doing.

The remaining terms are small individually and add up. The verifier's own calls, at $0.0002 per request on 100% of traffic, cost $634 a quarter here — more than the extraction workflow's entire potential saving. Retries after a 429 or a malformed response bill again. The continuous shadow sample bills. Embedding calls for retrieval do not change under routing and should be excluded from both sides rather than left in to dilute the percentage. The discipline that keeps this honest is to compute the baseline the same way as the actual: baseline-equivalent spend means replaying the same requests' token counts through the strong model's price with the strong model's realistic cache behaviour, rather than taking last year's bill as the counterfactual.

Putting the pieces in one artifact#

A quarterly page that supports a decision has roughly these rows, per surface, with the previous quarter beside each for drift:

Three of those rows are doing work that the headline number cannot. The worst-slice row is the gate from chapter 5 promoted into the running report, so that a regression concentrated in one workflow has a fixed place to appear every quarter rather than being discovered by a data team. The missed-escalation row carries the labelled measurement forward as a trend — the improvement from 4.0% to 1.2% here came from the deterministic checks and the verifier added in chapter 4, and it is the only row that speaks to the failure mode that produces no telemetry. And the escalation-reasons breakdown is the row that catches a specific kind of silent change: verifier timeouts at 13% of escalations means a dependency is degrading, which chapter 7 takes up as its own problem.

What this report does not establish#

It is worth being clear that a production report of this kind is a monitoring instrument rather than a causal one. The 1% shadow sample compares the cascade's answer against the strong model's answer on the same request, so it is a paired quality delta and it is trustworthy for detecting drift. It cannot tell you what would have happened under a different escalation threshold, or with a different cheap model, because those counterfactuals were never run. Decisions about changing the design still need the offline evaluation with its stratified sample and its pinned versions; the report tells you when to go and run one.

There is also a Goodhart risk that shows up quickly once escalation rate is on a dashboard next to a spend target, Goodhart's law being the observation that a measure people are asked to move stops being a good measure of the thing it stood for. Escalation rate is a driver, not an objective, and a team asked to reduce it will reduce it — by raising the threshold, by narrowing the schema checks, by declaring a task ineligible for verification — every one of which reduces cost and increases exactly the error nobody can see. The defensible target is the quality delta per slice, with the escalation rate reported as the mechanism that produced it.

The design assumes throughout that the components it depends on are answering. The verifier that caught 61 of the 80 missed escalations in chapter 4, and now accounts for a quarter of all escalation traffic, is a network call to somebody else's service, and the interesting question is what the cascade should do on the morning that service starts timing out.