A two-point drop on thirty cases is noise, not a regression

The alerting channel has 47 messages in the last 30 days and eleven members, nine of whom muted it in early March. The rule that produces them is one line in a config: fire when a tracked segment's daily evaluation score falls more than two points from the day before, across all 40 segments. Segment sizes run from 30 cases a day on the smallest to 4,000 on the busiest flow.

In the same 30 days, the invoice-adjacent contract_clauses segment — 60 cases a day — fell from 0.88 to 0.79 and stayed there. It was found by a customer, eleven days after the fact. Somewhere in that channel are two or three messages about it, sitting between a Portuguese segment that wobbled three points on a Tuesday and a tool-calling segment that does this every week.

Both facts come from the same line of config, and it is worth being precise about why, because the fix that suggests itself — raise the threshold until the noise stops — makes the second symptom worse while barely touching the first.

Score deltas do not have one variance#

An evaluation score is a mean over cases, and the run-to-run spread of a mean depends on how many cases went into it. When each case passes or fails independently with probability p, the standard deviation of the daily score is sqrt(p(1-p)/n), and a day-over-day difference compares two such measurements, so its standard deviation is larger by a factor of sqrt(2). At a pass rate of 0.85 that gives numbers which settle the argument.

A two-point rule on the 30-case segment fires four days in five when absolutely nothing is happening. Across 40 segments of mixed sizes, the expected count of spurious alerts per day is somewhere between eight and fifteen, which matches the observed 47 messages once you account for the alerts being batched into digests. The channel is not noisy because the world is noisy. It is noisy because the rule is stated in a unit — raw score points — that means something different on every segment it is applied to.

The same arithmetic explains the missed regression, though not in the way the shallow reading suggests. On the 60-case segment, a rule at six points still fires on 36% of quiet days, so raising the threshold does not restore anyone's trust in the channel; it moves the line and leaves the design error in place. More decisively, the 9-point drop on contract_clauses did not arrive as a step. Rescoring the archived outputs afterwards put the onset over about six days, roughly 1.5 points a day, which is what a provider rollout at partial traffic weighting looks like from the outside. A rule that only ever compares today with yesterday sees 1.5-point deltas buried under a 6.5-point noise band and never fires, no matter where you put the threshold, because the quantity it watches genuinely never moved. The level shifted; the daily difference did not.

A rule with four terms instead of one#

The replacement compares a current window against a long reference window, and it carries a minimum sample, a statistical test, a severity floor, and a persistence requirement.

Comparing against a trailing reference rather than yesterday is what makes gradual drift visible: a 30-day baseline of 1,800 cases against the last 3 days of 180 will show the whole accumulated distance, not one day's step. Pooling into windows is also how small segments acquire enough cases to say anything, and the window length should be set per segment from its daily volume rather than globally.

The test converts a difference into standard errors so that the threshold means the same thing on every segment. A two-proportion z-test is adequate and requires no libraries, and the number it produces — the z score, the observed drop divided by the sampling error the two window sizes lead you to expect — is the quantity the threshold is stated in. What matters is that the comparison is against the reference window's rate with the current window's sample size driving the error term.

The severity floor exists for the opposite failure. On the 4,000-case flow, a drop of 0.4 points is statistically certain within days and is not worth waking anyone for, so the rule requires the point estimate to fall at least 3 points as well as being significant. Significance and importance are separate conditions and a rule needs both, in the same way a latency alert needs both "clearly above baseline" and "above the number in the SLO", the service level objective the team has committed to keeping.

Persistence handles the residual. Requiring two consecutive runs, or two of three, roughly squares the false-positive rate per segment while costing one extra day of detection latency, and with 40 segments being tested daily the multiple-comparison arithmetic — the fact that running many tests at once multiplies your chances of one of them looking significant by accident, which the Bonferroni correction in the code below answers by dividing the tolerated false-alarm rate across all forty — makes that trade obviously worth taking — 40 independent tests at a nominal 5% would produce two false alerts a day on their own.

# alerts/drift_rule.py
from math import sqrt
from statistics import NormalDist

MIN_CURRENT_N = 90          # below this, extend the window backwards
MIN_ABS_DROP  = 0.03        # severity floor: 3 points
Z_CRIT        = 3.2         # 0.05 two-sided, Bonferroni over 40 segments

def segment_verdict(cur_pass, cur_n, ref_pass, ref_n):
    if cur_n < MIN_CURRENT_N:
        return "insufficient_sample"
    p_cur, p_ref = cur_pass / cur_n, ref_pass / ref_n
    drop = p_ref - p_cur
    if drop < MIN_ABS_DROP:
        return "ok"
    pooled = (cur_pass + ref_pass) / (cur_n + ref_n)
    se = sqrt(pooled * (1 - pooled) * (1 / cur_n + 1 / ref_n))
    z = drop / se if se else 0.0
    if z < Z_CRIT:
        return "watch"       # goes to a daily digest, not to the pager
    return "regression"      # fires only if the previous run also said watch or regression

Run against the previous 30 days of history, that rule produces three alerts instead of 47, and it fires on contract_clauses on the third day of the ramp — when the trailing 3-day window sat 4.4 points below the reference at z = 3.4 — rather than on day eleven via a customer.

Windowed comparison catches ramps because it measures a level rather than a step, and there is a sharper instrument for the same job if you want one. A CUSUM accumulates signed deviations from the reference rate and alarms when the running sum crosses a bound, which means a persistent drift of half a standard deviation per day accumulates towards the bound even though no single day is remarkable. On the contract_clauses history, a CUSUM with a slack of half a standard deviation and a bound of five crossed on day two rather than day three. The cost is a second concept to explain and a state variable to maintain, and it is worth adding for the handful of segments where a day of detection latency has a price attached, not for all forty.

The reference window needs a little hygiene or it will quietly absorb the thing you are trying to detect. A trailing 30 days that includes last week's regression drags the baseline down until the new, worse level looks normal, which is how a permanent degradation becomes invisible after a month. Two rules handle most of it: exclude days already marked as an incident from the reference, and re-baseline explicitly and with a note whenever you accept a change — a model migration, a deliberate prompt rewrite, a scoring rubric revision. A baseline that moves only when someone decides it should is auditable; one that moves continuously is a slow-motion way of agreeing with whatever happened.

What the rule cannot buy, and what to buy instead#

The sensitivity you get is fixed by how many cases you evaluate, and no threshold arrangement changes that. Detecting a drop of size d with 80% power at this significance level — power being the share of the times a genuine drop of that size gets caught, so four occasions in five — needs roughly n = p(1-p)(2.8/d)² cases in the current window, which at p = 0.85 comes to about 400 cases for 5 points, 100 for 10 points, and 1,600 for 2.5 points. A segment sampled at 30 cases a day will not detect a 5-point regression inside a week regardless of how the rule is written, and the honest response to "we need to catch 5 points on this segment within two days" is to raise its daily sample to 200, which costs money and is a decision someone can actually make. Reading the alert design as a sampling-budget question is more productive than reading it as a threshold question.

The rule also does not tell you what happened. It says that contract_clauses is 4.4 points below its reference with a z of 3.4, and the replay ordering from the previous chapter is what turns that into a cause. Teams that skip that step tend to attribute every alert to whatever changed most recently in their own repository, which is the one explanation the alert has already made unlikely.

There is a variance the formula does not include, which is the grader's own. The sqrt(p(1-p)/n) figure assumes each case has a fixed pass probability and the only randomness is which way it lands, whereas an LLM judge re-scoring the same output can change its mind, and a judge scoring two fresh outputs of the same case has two sources of movement stacked. On the segments here that inflates the spread above the binomial prediction by 10% to 40%, most at the segments whose cases sit closest to the pass boundary. The empirically measured noise floor you established for the CI gate is the number to use in the alert rule, with the formula serving only as a check that the two agree to within a factor; where they disagree badly, the grader is usually the reason.

Nor does any of this help if the metric is wrong. A grader that rewards length will hold perfectly steady through the exact regression from the first chapter, and a well-calibrated alert on a badly specified score is a precise instrument pointed at the wrong thing. Thresholds control false alarms about the number you have; they cannot tell you that the number is not the one you care about.

The eleven days are a separate piece of work#

Fixing the rule does nothing about the interval that already happened, and that needs treating on its own terms. Because the outputs were archived rather than only scored, the history can be rescored against the corrected grader to date the onset — which is how the six-day ramp above was established — and that date bounds the exposure. Cross it with production volume in that segment, not evaluation volume: 60 evaluation cases a day corresponds to roughly 2,400 real contract-clause requests over eleven days, spread across a known set of tenants, and that is the number that goes to the account team and, if the segment carries a contractual commitment, to the customer.

The muted channel deserves the same treatment as the regression, because it is the more durable failure. Nine people muting an alert is a decision the organisation made about the signal's value, and it will be made again about the replacement unless the routing distinguishes severity: confirmed regressions page, watch verdicts accumulate in a daily digest nobody has to read in real time, and anything that fires more than about twice a month without producing an action gets deleted or re-specified. An alert that everyone ignores is worse than no alert, since it also provides the reassurance of having one.

Dating the onset to a particular Tuesday is where this chapter's work stops and the next problem starts. Knowing the day something changed is useful only if you can go back to that day and show what the system did, and that turns out to depend on decisions made months earlier about what you kept.