Unit-cost alerting per feature

A unit cost is spend divided by a count of things the product delivers — model calls, user actions, or successfully completed user actions. It is the only spend metric that holds still while a business grows, which is what makes it the basis for alerting: total spend rises when traffic rises, and an alert that cannot tell growth from regression is an alert somebody silences in the first week.

Underneath that is a fact about the arithmetic rather than about alerting practice. A feature's spend is a product of two factors: how many times it did the thing, and what each one cost. An alert on the product cannot report which factor moved, and the two factors are wanted in opposite directions — traffic is what the business is trying to increase, unit cost is what engineering is trying to hold down. So a spend alert fires on success and stays quiet on regression whenever the two move against each other, and it is quiet whenever the regression is smaller than the day-to-day variation in the traffic factor. Dividing spend by a count of deliveries cancels the factor nobody wants paged about and leaves the factor they do, which is the entire reason the alertable quantity has to be a ratio.

The cancellation is only clean when cost is roughly proportional to the count, and it is worth knowing where that stops. A feature carrying a fixed hourly cost — a cache-warming job that runs whether anybody arrives or not — has a unit cost that falls as traffic rises and climbs on a quiet Sunday, with nothing regressed and nobody at fault. What the ratio needs is a denominator counting the thing the money was spent on, which is a choice with its own section below.

Getting from that idea to an alert that fires takes three decisions: what the metric is scoped to, what it is compared against, and what goes in the denominator. This chapter makes all three, works through why the monitors most teams already have cannot catch a regression of this shape, and pairs the resulting alert with the reconciliation residual from chapter 4, because each of the two is structurally blind to what the other sees.

Four monitors that do not fire#

Take a deploy that adds nine words to a draft-reply feature's instructions, moving output tokens up 77% and the feature's unit cost from 2.10 to 2.55 cents. Four reasonable monitors watch that model's spend, and each fails for its own instructive reason.

Total daily spend sits at $2,840 a day with a ±11% weekday-to-weekend swing. Draft-reply is 31% of Opus 5 traffic, so the regression adds about $190 a day — 6.7% of the total, comfortably inside a band that already has to tolerate ordinary demand variation. Any threshold loose enough to survive a normal Monday is loose enough to swallow this.

The $190 is not an independent measurement, and following where it comes from is what makes the failure predictable rather than unlucky. The feature's unit cost rose 21.4%, so the feature's own spend rose 21.4%, and because the feature is roughly 31% of the traffic on that model its rise reaches the daily total scaled by that share — about a fifth of a third, which is the 6.7%. Now set that against the band it would have to be seen through. The weekday-to-weekend swing is 11% of the total, so in dollars the ordinary swing is larger than the regression. A threshold is a single line, and it cannot separate two quantities that overlap.

Tokens per request moves from 14,074 to 14,261, which is 1.3%, because the nine words that went in were input and the 178 tokens that came out were output. No threshold catches that, and no threshold should.

That 1.3% is the most instructive of the four, because it sits beside a 21.4% move in the money for the same event. The difference between the two token counts is the 178 tokens that came out plus the handful the nine words went in as, and against a request already carrying more than fourteen thousand tokens it disappears. The gap between the two percentages is the price gap between categories from chapter 3 (token categories and their rates), where rates inside one model span fifty to one. A metric counting tokens weights a cache read and an output token identically and the invoice does not, so counting tokens is not a cheaper proxy for counting money. It is a different measurement that happens to be built from the same raw counters.

Error rate and p95 latency — the response time 95% of requests come in under — both move, latency by about 900 ms, since 178 more output tokens take time to decode, and both stay inside their service level objectives, the response-time and error-rate targets the team has committed to publicly. Latency is a real leading indicator of output-length growth and a weak one, because it is confounded by queueing, model routing and load, and because a change that makes calls cheaper and slower looks identical to one that makes them dearer and slower.

Cost per request, blended across all features on the model, moves 6.6%: the feature's 21.4% diluted by its 31% share of traffic. That is the one that comes closest, and dilution is exactly the mechanism keeping a global metric from ever firing on a single feature's regression. The more features share a model, the more a real problem in one of them looks like rounding in the aggregate.

Dilution is one multiplication and it is worth writing out, because seeing it as arithmetic rather than as a vague loss of signal is what tells you it cannot be tuned away. Cost per request blended across a model is the traffic-weighted average of the per-feature unit costs, so a change in one feature's unit cost reaches that average multiplied by the feature's share of the requests: 21.4% against a 31% share is the 6.6%. The multiplier is the share, which means every new feature that lands on the model shrinks every other feature's share and delivers the same regression to the aggregate smaller than the quarter before.

The repair that suggests itself is to lower the threshold on the blended metric, since 6.6% is a real move and a line drawn under it would have fired. It does not work, and the reason is that a weighted average moves for two independent causes: a change in one of the unit costs, and a change in the weights. Traffic mix shifts every day with nothing regressed — a marketing send, a batch window, a public holiday in one region — so a threshold low enough to catch a diluted regression is a threshold that fires on ordinary mix. Scoping to a single feature is therefore not a more sensitive version of the same metric. It removes the second cause outright, because inside one feature there is one weight and it is one.

The metric that fires#

The metric that moves 21.4% on Tuesday afternoon is cost per request for that feature, compared against that feature's own recent history, with the token categories broken out so the shape of the change is visible in the same alert that reports it.

Both halves are load-bearing. Per-feature scoping removes the dilution. Baselining against the feature's own history removes the need for a threshold that means something across features whose unit costs differ by two orders of magnitude — classification at 0.04 cents a request on Haiku, research at 45 cents a request on Opus at high effort, and no single dollar threshold is meaningful for both. A ratio against each feature's own trailing median is meaningful for both.

WITH hourly AS (
  SELECT feature,
         hour_bucket,
         count(*)                                              AS requests,
         sum(cost_usd)                                         AS usd,
         sum(cost_usd) / count(*)                              AS usd_per_request,
         sum(output_tokens)::numeric / nullif(sum(input_tokens
             + cache_read_tokens + cache_write_tokens), 0)     AS output_ratio,
         sum(cache_read_tokens)::numeric / nullif(sum(input_tokens
             + cache_read_tokens + cache_write_tokens), 0)     AS cache_hit_share
  FROM   llm_cost_facts
  WHERE  environment = 'prod'
    AND  hour_bucket >= now() - interval '15 days'
  GROUP  BY feature, hour_bucket
),
baseline AS (          -- same hour-of-day, previous 7 days, excluding today
  SELECT feature,
         extract(hour FROM hour_bucket)                          AS hod,
         percentile_cont(0.5) WITHIN GROUP (ORDER BY usd_per_request) AS median_upr,
         percentile_cont(0.5) WITHIN GROUP (ORDER BY cache_hit_share) AS median_chs
  FROM   hourly
  WHERE  hour_bucket <  date_trunc('day', now())
    AND  hour_bucket >= date_trunc('day', now()) - interval '7 days'
  GROUP  BY 1, 2
)
SELECT h.feature, h.hour_bucket, h.requests,
       h.usd_per_request, b.median_upr,
       h.usd_per_request / b.median_upr AS unit_cost_ratio,
       h.cache_hit_share, b.median_chs
FROM   hourly h
JOIN   baseline b
  ON   b.feature = h.feature AND b.hod = extract(hour FROM h.hour_bucket)
WHERE  h.hour_bucket >= now() - interval '3 hours'
  AND  h.requests >= 200
  AND  (h.usd_per_request > 1.15 * b.median_upr
        OR h.cache_hit_share < 0.60 * b.median_chs);

Four choices in that query are worth defending. Baselining against the same hour of day matters because unit cost has a daily shape — overnight traffic on many products is batch work with different prompt sizes, and comparing 03:00 against a 24-hour average generates a nightly false alarm. The median rather than the mean keeps one bad hour from poisoning the baseline it will later be compared against. The minimum of 200 requests stops a low-traffic feature paging on three unusual calls; below that count the ratio is dominated by which requests happened to arrive. And the 1.15 threshold with a two-hour confirmation — implemented by requiring two consecutive breaching hours before notifying — sits above ordinary hour-to-hour drift, which on the features measured here runs around ±6%, while still catching a 21% step change within about two hours of the deploy.

Each of those numbers is a decision with a cost on both sides, and the two costs are not symmetric. Set the threshold too high and a regression is missed, which costs the whole interval until something else notices — twenty-three days in the case this chapter is built on. Set it too low and somebody is paged for nothing, and the price of that is not the interruption but what the second false page teaches: that the alert is noise. After that it has a detection rate of zero while continuing to look configured. The asymmetry argues for missing small regressions rather than catching them alongside false alarms, which is why 1.15 sits well clear of ±6% rather than just above it — a drift figure describes the spread of ordinary hours, not a bound on them, so a line drawn at its edge fires on the ordinary hours in the tail.

The two-hour confirmation and the 200-request minimum are both buying fewer alarms from hours that were merely unusual, at opposite ends of the noise. Requiring two consecutive breaching hours replaces one unusual hour with two in a row, which is a much rarer event when the hours are close to independent, and it is paid for in a knowable way: two hours of exposure after every deploy. The request floor works on the other end, because an hour's unit cost is a mean over that hour's requests and the spread of a mean narrows as the count of requests grows. Below the floor the ratio reports which requests happened to arrive rather than what the feature costs, which is why it is a count and not a dollar amount. None of the four figures is derived from anything; they come from measuring one feature's own drift and adjusting after the first few false alarms, and copying them without that measurement is copying somebody else's noise floor.

The cache condition is a separate clause on purpose. A cache-reuse collapse and an output-length regression are different failures with different owners and different fixes, and an alert saying only "unit cost up 18%" sends someone to read a chart before they can start work. An alert saying "unit cost up 18%, cache hit share 0.87 → 0.04" has already named the bug.

Choosing the denominator#

"Cost per request" hides a decision, and the three available denominators catch different regressions. Counting model calls gives cost per call; counting distinct correlation ids gives cost per user action; counting only actions that returned successfully gives cost per delivered outcome.

What makes one of those a legitimate denominator rather than merely an available one is that it counts the thing the money was spent on. Spend and count then move together when nothing has changed, which is the property that holds the ratio still through growth. Dividing by a count that is not what the money bought is the commonest way to build a metric that looks stable and is not — total HTTP requests to the service, say, most of which never reach a model. Ship a cheap endpoint that doubles request volume and cost per request halves with no model call altered, and the dashboard reports an efficiency win that did not happen.

The gap between cost per call and cost per user action is fan-out, and fan-out drifts. A research feature that goes from 8 model calls per click to 14 — because a planner started spawning more sub-agents, or a tool loop started iterating further — has a 75% cost increase per user action while cost per model call sits perfectly still, since each individual call is unchanged. That regression is invisible to the per-call denominator and glaring to the per-action one. The reverse case exists too: a change that consolidates six calls into two longer ones raises cost per call while lowering cost per action, and alerting on the per-call figure would page someone about an improvement.

The third row matters when a feature starts failing after doing most of its work. If 12% of research requests now error out at the synthesis step, the tokens spent on the planner and the sub-agents are still billed and the user got nothing, so cost per delivered outcome rises 14% while cost per attempted action is flat. Alert on distinct correlation ids as the primary denominator and put cost per successful action on the dashboard next to it; the gap between the two lines is the cost of failure, and it is a number most teams have never seen.

What the alert says on Tuesday#

Concretely, at 16:00 — two breaching hours after a 14:07 deploy — the notification carries enough to skip the investigation entirely:

{
  "alert": "unit_cost_regression",
  "feature": "draft-reply",
  "window": "2026-03-17T14:00Z..16:00Z",
  "requests": 3814,
  "usd_per_action": 0.02546,
  "baseline_median": 0.02101,
  "ratio": 1.212,
  "category_deltas": {
    "input_tokens":      "+0.5%",
    "cache_read_tokens":  "0.0%",
    "output_tokens":     "+76.7%"
  },
  "cache_hit_share": {"now": 0.866, "baseline": 0.866},
  "deploys_in_window": ["a91c4f0 draft-reply: tune instructions"]
}

Nobody has to know what the deploy did. Output up 77% with input and cache flat, one deploy in the window that touched the feature's instructions, and the change is identified before anyone opens a dashboard. Including the deploy list in the payload is a small piece of glue and the single largest determinant of whether the alert gets acted on, because an alert that names a commit is a ticket and an alert that names a percentage is a chore.

The reverse smell, and the pair that closes it#

A per-feature unit-cost alert watches spend that is in the ledger. It is structurally blind to spend that never reached the ledger, because a call that writes no row contributes nothing to either the numerator or the denominator — cost per request stays exactly where it was while the invoice climbs. The residual alert from the reconciliation job is the other half, and the two together cover the four combinations:

Wiring both means the specific combination that produces a month-end surprise — flat dashboard, rising invoice — has an alert whose entire job is that cell. Neither alert alone makes it impossible; the pair does, and the pair is the minimum.

Quality signals, deploys and per-tenant noise#

The unit-cost alert has nothing to say about quality. Dropping effort from high to low on a feature shows up as a clean 30% cost improvement and a green dashboard while the model quietly under-thinks on the hard 15% of requests. Cost metrics and quality evaluations answer different questions, and a cost win with no accompanying evaluation is an unvalidated change wearing a success badge. The honest form of the metric is cost per action tracked alongside whatever quality signal the feature already has, so a tradeoff is visible as a tradeoff.

It also cannot distinguish an accident from a decision. Enabling web search on a research feature raises its unit cost by design and pages whoever is on call, and after that happens twice people start ignoring the alert, which is a worse state than not having it. The fix is mechanical: emit a deploy annotation into the same store, and suppress the alert for a feature within two hours of a deploy that touched it while still recording the level shift so the new baseline establishes cleanly. An unsuppressed alert that fires on every intentional change trains the team to close it.

Per-tenant alerting looks like the obvious next step and mostly is not, at least in the form of alerts. Three hundred tenants times five features is 1,500 series, each with less traffic and more variance than the feature-level series, and at any reasonable sensitivity several breach every day. What works instead is a ranked daily digest — the ten tenants whose spend moved most against their own trailing median, with absolute dollars alongside the percentage so a tenant that went from $0.40 to $1.60 does not outrank one that went from $900 to $1,400. That is a report somebody reads with coffee, and it catches a runaway integration a day or two after it starts rather than at month end.

Two alerts, then, are the whole of the monitoring for this topic: unit cost per feature against its own hour-of-day baseline, and the reconciliation residual as a share of provider spend. Both are queries over the fact table and its nightly comparison, both are cheap, and both exist only because the columns underneath them — feature, correlation id, the four token categories, the status — were written at the moment each call was made.

Every mechanism in this topic has that property. The price on the row, the correlation id, the split counters, the status column, the fact table, these two alerts: each is something that has to be true when a model call happens. None can be added later by querying harder, which is the argument the last chapter has to make in the language of engineering weeks.