Fifty samples an hour crosses any threshold on its own

The incident log for the week has four entries against the quality alert. Tuesday 03:12, resolution note "recovered on its own, no action taken". Wednesday 21:40, same note. Thursday 02:55 and Thursday 16:07, same note with an added remark that the responder checked the deploy log and found nothing. Two of the four woke someone up. None of the four produced a change to anything, and the fourth was acknowledged and closed in ninety seconds by a person who had learned by then what the answer would be.

The alert is doing exactly what it was configured to do. Fifty requests an hour are sampled and scored, the hourly pass rate is compared against a floor of 90%, and a breach pages the on-call rotation. What nobody worked out before switching it on is how often fifty draws from a 96%-pass process land below that floor by themselves.

The floor is inside the noise#

The hourly pass rate is a binomial proportion, meaning a count of successes over a fixed number of independent pass-or-fail draws, and it carries the spread that comes with being counted from fifty of them. With a true pass rate of 96% and fifty samples, its standard deviation is the square root of 0.96 × 0.04 / 50, which is 2.77 percentage points. An hourly reading is therefore routinely five or six points away from the truth in either direction with nothing wrong, and the distribution is discrete enough that the arithmetic is easier to read in counts than in percentages.

A 1.44% chance per hour across 168 hours a week is an expected 2.4 pages a week from a system in perfect health, and weeks with four are entirely ordinary — the count of noise pages is itself a random variable, and a week with four when the mean is 2.4 happens about one week in eight. Every one of those pages is genuinely a recovery, in the sense that the next hour's sample really does come back near 96%, because nothing was ever wrong.

Two consequences follow, and the second is the expensive one. The alert cannot be acted on, since there is no state of the world in which the responder can distinguish this page from a real one at the moment they receive it. And the alert trains the rotation, so that when a genuine regression eventually pages at 03:00, the acknowledged-and-closed reflex fires before anyone reads the graph. An alert that is wrong five times out of six is worse than no alert, because no alert at least leaves the responder's attention intact.

Raising the sampling rate fixes the wrong half of the problem#

The tempting repair is more samples. It works, in the narrow sense: at 500 samples an hour the standard deviation of the hourly figure falls to 0.88 points and a 90% floor sits nearly seven standard deviations away, so the false pages stop. What it costs, and what it fails to buy, are both worth writing down.

A judge call on this workload runs about 1,800 input tokens and 120 output tokens, which comes to roughly $0.0024 at current mid-tier pricing. At 50 samples an hour that is 8,400 samples and about $20 a week, which nobody notices. At 500 an hour it is 84,000 samples and $200 a week, still not a large number in absolute terms but now 20% of eligible traffic passing through a second model. Scoring every request — 420,000 a week — costs about $1,008, and since the product itself runs around $0.014 a request, the evaluator would be adding 17% to unit cost to compute a number that is already accurate to a fraction of a point at a twentieth of the volume.

The larger objection is architectural. An evaluator scoring a meaningful share of traffic stops being an offline consumer and starts being a capacity dependency: it needs its own rate limit headroom, its own retry policy, its own queue depth alarm, and it will have an incident of its own at some point, during which the system reporting on your system's health is the thing that is down. That is a real trade to make deliberately at high sampling rates and an easy one to fall into by tuning a number upward until an alert stops firing.

None of that spending addresses the failure from the previous chapter. A perfectly estimated aggregate is still an aggregate; the tenant at 0.81% of traffic moved the fleet number by 0.66 points whether it was measured on 8,400 samples or on all 420,000. Noise and blindness are separate defects of the same metric, and more samples treats only the first.

An alert needs a sample floor and a sustained condition#

The rule that survives has three parts: it may not fire until enough samples exist to support a conclusion, it requires the condition to persist rather than to occur, and its threshold is expressed as a rate of budget consumption rather than as a distance below target. The third part is what makes the numbers choose themselves.

With a 95% target over a 28-day window, the error budget is 5% of scored samples. Burn rate is the multiple of the budget's steady-state consumption you are currently running at, so an observed failure rate of 20% is a 4× burn and would exhaust a 28-day budget in seven days. That framing rescales cleanly, and it is worth noting that the burn-rate multipliers quoted in most SRE material are calibrated for 99.9% availability targets where the budget is tiny; a 14.4× burn against a 95% target means a 72% failure rate, which is not an alert threshold so much as an outage. Quality objectives sit at 95% or 90% and need their own multipliers.

The pair of burn rules is the configuration to run. The fast rule exists for the case where something has genuinely broken — a prompt deployed with a truncated system message, a retrieval index that came back empty, a model swap that went to the wrong tier — and at a 40% failure rate it fires inside an hour with a false-alarm rate of roughly once every six years. The slow rule catches the 12%-failure regression that the fast rule will never see, and it deliberately files a ticket rather than paging, because a quality regression of that size is not something the 03:00 responder can fix and a ticket at 09:00 loses nothing.

Expressed as code rather than as a table, the evaluation is unremarkable, and the sample gate is the line that does the work.

from dataclasses import dataclass

@dataclass(frozen=True)
class BurnRule:
    name: str
    window_hours: int
    min_samples: int
    max_fail_rate: float
    severity: str            # "page" | "ticket"

RULES = [
    BurnRule("quality_fast_burn", 1, 50, 0.20, "page"),
    BurnRule("quality_slow_burn", 24, 1_000, 0.08, "ticket"),
]

def evaluate(store, rule: BurnRule, now):
    scored, failed, errored = store.counts(
        since=now - timedelta(hours=rule.window_hours)
    )
    # An evaluator that is failing is not a system that is failing.
    if errored > 0.10 * (scored + errored):
        return Signal("evaluator_degraded", severity="ticket")
    if scored < rule.min_samples:
        return None                      # not enough evidence to say anything
    if failed / scored >= rule.max_fail_rate:
        return Signal(rule.name, severity=rule.severity,
                      observed=failed / scored, n=scored)
    return None

Returning None when the window is under-populated is the part that most implementations get wrong, because the natural default in an alerting language is to compare whatever is there. An hour in which the evaluator queue backed up and only nine samples were scored will show a pass rate of 0.78 if two of them failed, and a rule without the gate will page on it. The gate turns that hour into silence, which is correct in the sense that nothing was learned, and the separate evaluator_degraded signal is what says so.

Alerting on slices multiplies the problem you just solved#

The slice dimensions from the previous chapter make this arithmetic worse before they make anything better, because every series that can be watched is a series that can cross a line. Forty named slices each evaluated against the current hourly rule would produce forty times 2.4 pages a week, which is a hundred pages, and the multiple-comparisons problem — the more series you watch, the more of them cross a line by chance alone — is not solved by tuning the threshold — it is solved by not alerting on most slices at all.

The workable split is that alerting is reserved for slices with both a commitment and enough volume to support a rule, while everything else is reviewed rather than alerted. A slice qualifies for its own alert when stratified sampling guarantees it at least the rule's minimum sample count within the rule's window, which for the 24-hour slow-burn rule at 1,000 samples means the slice needs about 42 scored samples an hour. On the stratified allocation from the previous chapter, the enterprise tier as a whole qualifies and no individual enterprise tenant does, so tenants are covered by the weekly worst-slice scan and the tier is covered by an alert. Writing that rule down as a policy — a slice gets an alert only if its guaranteed sample rate supports the window — prevents the slow accretion of per-customer alerts that each fire twice a month for no reason.

What the responder receives matters as much as when. A page that says "quality fast burn: 22% failure over 68 samples" is not actionable on its own; the same page carrying the top three failure_reason values, the two slices contributing most of the failures, three sample rationales in full, and a link to the traces behind them lets someone form a hypothesis before they have opened a dashboard. All of that is available from the sample store at alert time, and assembling it into the notification is a couple of hours of work that changes the character of every quality page thereafter.

Sequential methods, and what none of them fix#

Fixed windows with fixed thresholds are the most legible option and not the most statistically efficient one. Exponentially weighted moving average control charts carry a running estimate of the mean with an adjustable memory — at α = 0.15, roughly the last seven hourly readings dominate — and signal when the smoothed value leaves control limits derived from the process's own variance, which detects a small sustained shift several hours earlier than a 24-hour window at the same false-alarm rate. CUSUM does something similar by accumulating signed deviations and is the sharper tool for detecting a step change of known size. Both are well-understood and both are harder to explain in a review than "failure rate over 24 hours exceeded 8%", which is a genuine consideration when the person asking is a product owner deciding whether to accept the objective.

There are limits worth stating plainly rather than discovering during an incident. An alert on a sampled quality metric will always be slower than an alert on errors, because errors are counted from complete data and quality is estimated; the 12-to-18-hour detection above is the honest cost of a 2% sample, and shrinking it means either more samples or a larger tolerated regression. A sustained-change rule buys quiet with delay, by construction. And a quality alert is not a deploy gate, since by the time enough samples exist to be confident, the change has been serving traffic for hours — gating belongs to the offline suite, and the alert exists for regressions that the offline suite did not contain.

Both of the week's problems now have distinct answers: the four pages were noise crossing a line, and the six-week tenant failure was an aggregate with no dimensions and no per-slice review. Neither is fixed by tuning the other. What the burn-rate framing adds is a vocabulary in which a quality objective and a latency objective can be discussed in the same units — budget consumed, at what rate, with how much of the window left — which is what the weekly report has to speak, and what leadership will try to compress into a single number in the next chapter.