Error-budget burn beside the share of traffic measured

The weekly platform review has one slide per system, and yours currently has four lines on it: latency objective met on 99.2% of requests against a 99% target, error objective 99.9% against 99.9%, quality 94% against a 95% target, cost $0.014 per request. The VP of Engineering asks, reasonably, whether those four can be collapsed into a single AI health score, because every other platform on the slide deck has one number with a colour behind it and yours takes forty seconds to read.

Two weeks before that meeting, the evaluator silently timed out on 60% of samples for four days. The quality figure that week was computed from the 1,240 samples that came back, and it read 96.1%. When someone reran the dropped samples eleven days later the true figure for the week was 92.4%, because judge calls time out preferentially on long answers with large retrieved context, and long answers with large retrieved context are where groundedness failures live. The number was not merely imprecise; it was biased in the direction that made the week look good, and nothing on the report said it was computed from two-fifths of the intended sample.

That incident is the reason to decline the composite, and the argument is worth making in those terms rather than as a general objection to summaries.

Three of the four are counted and the fourth is estimated#

Latency, errors and cost are computed from complete data. Every request emits its own timestamps, its own status, and its own token counts, so the objective is an arithmetic over the whole population and the only judgement calls are the window and the eligibility filter. The quality objective is a sample estimate produced by a model that can be wrong, can drift, and can fail to answer at all, over a 2% draw from eligible traffic. Those two kinds of number do not average, and a blend inherits the weakest property of its inputs while advertising the confidence of its strongest.

The composite has a second defect that shows up the first time it goes red. Weighting a latency breach against a quality breach establishes an exchange rate between a slow answer and a wrong one — say that a point of quality is worth three points of latency attainment — and nobody in the room can defend that number, because it is not a technical quantity. The following quarter, when the score is amber and one team wants to argue that their latency work should offset another team's quality regression, the discussion is about weights rather than about either problem. The pattern also has a predictable lifecycle: the components are published underneath the composite for the first two cycles, and then everyone reads the headline and the components stop being maintained.

What leadership actually needs is legitimate, and it is not a weighted average. They need one glance, comparability across weeks, and a rule for when to ask a question. All three are available from a status derived by rule rather than by arithmetic: red if any objective has consumed more than 100% of its error budget for the window, amber if any is above 75%, green otherwise, with the rule printed next to the light. That fits in a corner of the slide, cannot be gamed by trading one objective off against another, and says which objective caused the colour.

The report that survives an evaluator incident#

The two columns that were not there before are the ones that matter. Budget consumed converts every objective into the same unit — the fraction of the window's permitted failures already spent — so that 99.2% against 99% and 94% against 95% become 80% and 120%, comparable at a glance, and the request-success line at exactly 100% is visibly out of headroom rather than looking like a pass. Traffic measured is what makes a number self-describing: 100% on three rows, 2.0% on the fourth by design, and on the week of the incident that cell would have read 0.8% in red, which is a question anyone reading the slide would have asked without knowing anything about evaluators.

That column is doing a different job from an alert on evaluator health, and both are needed. The alert fires while the timeouts are happening and gets somebody to restart a worker; it lives in the operational path and, once resolved, leaves no trace on any historical figure. The coverage column is a permanent property of every number the report has ever published, so that six months later, when someone compares this quarter's quality attainment against last quarter's, the weeks computed from a degraded sample are marked as such in the record. The incident above went unnoticed for eleven days precisely because there was an alert that did not fire and no column that would have shown its absence.

Cost is worth stating per accepted answer rather than per request. Failed and retried requests consume tokens too, so $0.014 per request at a 94% pass rate is $0.0149 per answer a user could actually use, and the two figures diverge exactly when quality degrades — which means the per-request figure quietly improves during a quality incident, since a model producing shorter, worse answers is cheaper.

Attributing burn when eight teams ship prompts#

On a platform serving eight product teams, "the quality budget is 120% consumed" is true of nobody in particular. Each team ships its own prompt versions on its own cadence, and the dimensions established earlier make the attribution mechanical: every scored sample carries prompt_version and the owning service, so budget consumption partitions by owner.

WITH budget AS (           -- 28d window, 95% target
  SELECT count(*) * 0.05 AS allowed FROM quality_samples
  WHERE scored_at > now() - interval '28 days'
)
SELECT s.owning_team,
       s.dims->>'prompt_version'                  AS prompt_version,
       count(*)                                   AS scored,
       count(*) FILTER (WHERE verdict='fail')     AS failures,
       round(100.0 * count(*) FILTER (WHERE verdict='fail')
             / (SELECT allowed FROM budget), 1)   AS pct_of_budget
FROM quality_samples s
WHERE s.scored_at > now() - interval '28 days'
GROUP BY 1, 2
ORDER BY pct_of_budget DESC;

A platform-level figure of 120% resolves into one team at 6.2% of traffic consuming 41% of the budget, which is a conversation with a named owner and a named prompt version rather than a line item on a slide. It also reframes the cost row, since a blended $0.014 conceals a twenty-fold spread between a classification call and a document summary, and any efficiency target set on the blend will be met by whichever team happens to grow fastest.

Two mechanics make this workable in practice. Attribution needs a rule for shared failures — when a retrieval index regression damages three teams at once, the burn is charged to the platform rather than to the teams, and the report needs a platform row so that the charge has somewhere to go. And a team's burn is only interpretable against the traffic they send, which is why the share-of-traffic column sits next to it; 20% of the budget from 27% of traffic is unremarkable, and 41% from 6% is not.

Generating it without anyone assembling it#

The report has to be produced by the same job that feeds the alerts, from the same store, or the two will disagree and the disagreement will be discovered in a review. The pipeline is short and its outputs fan out from one place.

The generator is a scheduled job that queries the sample store and the metrics backend, renders the two tables above plus the worst-slice list, writes the result to a file in the repository, and posts it. Committing it matters more than posting it: a report in a chat message is gone in a week, and a report in git is a series that can be diffed, which is how anyone answers "when did this start" six months later. It also means the report's own code is reviewable, so a change to how burn is computed shows up in a pull request rather than in a spreadsheet formula somebody edited.

Confidence belongs on the report as well, because the quality row is an estimate. With 8,400 scored samples and an observed 6% failure rate, the standard error on that proportion is 0.26 points, so the 95% confidence interval on attainment — the range the true figure falls inside on all but one occasion in twenty — is roughly 93.5% to 94.5% and the corresponding budget burn is between about 110% and 130%. Publishing that interval prevents the specific argument in which a team points out that 94.0% against 95.0% could be sampling error — it could not be, at this sample size, and the interval says so — and it prevents the opposite argument on a slice with 60 samples, where the interval genuinely is wide enough to make the number unusable on its own.

One further consistency requirement is easy to miss and expensive to discover late. The report's eligibility filter and the alert rules' eligibility filter have to be the same expression, evaluated from the same definition, or the weekly figure and the alert history will describe different populations. The failure mode is mundane: someone excludes a noisy internal route from the alert to stop it firing, does not touch the report, and three weeks later the report shows a burn nobody was paged about. Keeping the population definition in one versioned file that both the alert evaluator and the report generator import is the whole fix, and it also means a change to the population is a reviewable diff with a date attached, which the annotated dashboard can then show.

What the report cannot do#

A weekly cadence detects nothing. Everything on it was already visible in a dashboard or an alert days earlier, and a regression that only the weekly report catches has been serving users for up to seven days. Its purpose is deliberation — deciding whether a target is right, whether a team's burn justifies pausing feature work, whether the sampling rate should move — and treating it as a safety net will produce a slow, expensive incident eventually.

It is also not a substitute for the escalation path that found the German-language failure. The worst-slice scan makes that failure visible on the report a week after it starts, and an account manager's email will still sometimes beat it, which is fine as long as the escalation ends up recorded as a slice with a burn attached rather than as an anecdote.

The objectives themselves are provisional, and the quality one most of all. A 95% target chosen in a planning meeting is a guess until it has survived a quarter of being missed and a quarter of being comfortably met, and the evaluator behind it will get stricter, cheaper or deprecated inside a year. Phil Hetzel's talk on the maturity phases of running evals is a useful map of that trajectory — the first version of a quality objective exists to force the arguments about population and sample into the open, and the version that is still standing two years later usually shares only its name with it.

What remains is to run the thing. The next step is a first review meeting where these numbers are put in front of the product owner who agreed the 4.2-out-of-5 line, with the mutation test, the slice table and the coverage column as the evidence for replacing it, and with a concrete answer to the question that follows immediately: what actually happens the first month the quality budget is gone.