Every dashboard green and the answers wrong is still a page

The page fires at 02:14 on a Tuesday. The body reads eval.contract_summary.mean_score: 60.2 (baseline 71.4, threshold -5.0) and carries a link to a job log. The on-call engineer opens the service dashboard out of habit: availability over the trailing six hours is 100.00%, p95 latency — the duration under which 95% of requests finish — is 1.84 seconds against a 1.9-second baseline, the error rate is 0.00%, the queue depth is flat, and the last deploy shipped eleven days ago. Every graph is the shape it is supposed to be. The only number that has moved is produced by a job that re-runs 240 stored contracts through the summariser every six hours and scores what comes back.

The vocabulary in that paragraph is worth making explicit, because the rest of this topic is built on it. An incident is an unplanned failure serious enough to interrupt normal work while it is still happening, and incident response is the practice of handling one: noticing it, containing it so the damage stops growing, working out what changed, telling the people affected, and deciding when it is over. The engineers who agree to be interrupted are the on-call rotation, the automated message that wakes one of them at 02:14 is a page, and the apparatus around both is standard across the industry — an alert with a threshold, a runbook of steps written in advance, an incident commander who owns the decisions while the incident runs, and a written review once it closes. All of that was designed for software that fails loudly, where a process crashes or a dependency times out and a stack trace names the line that broke.

A system built on a language model has a failure class that fails quietly instead. It answers, in the right format, at the usual speed, and the answer is worse than it was last week; nothing raises an exception, so nothing that watches for exceptions reports anything. Incident response for AI systems is the same lifecycle and the same command structure adapted to that class, which is what this topic covers — detecting a quality drop by measuring outputs rather than by counting errors, rolling back when the thing that changed may live on a vendor's side of an API, and telling customers something true while the cause is still unknown. An engineer meets it in the ordinary places — the alert that woke this on-call, the incident channel, the status page, the email to twelve enterprise accounts — each of which needs a kind of evidence the existing tooling was never built to collect.

The alert body carries the first of those adaptations. A fixed evaluation set is a stored collection of inputs with known-acceptable outputs, replayed through the live system on a schedule and scored so that a number moves when behaviour moves, and a slice is a named subset of it — here the sixty contract-summary cases — kept separate so that a drop in one kind of work is not averaged away by three kinds that are fine.

The first thing to settle is whether the measurement is real, and that takes about three minutes. The eval job has its own failure modes: a scorer that was updated in the same window, a fixture file that was regenerated, a truncated fetch of the source documents, a change to the rubric prompt that tells the scorer what a good summary contains. Check the job's git SHA against the previous run, check that the input hashes match, and check that the other three slices — support-reply, clause-lookup, redline — are still where they were. In this case they are: support-reply moved 0.4 points, clause-lookup 0.9, redline 0.2. One slice moved, by eleven points, and the harness did not change. That is a real observation about the system, and it means someone is awake for a reason.

What the green dashboards were measuring#

Availability, latency and error rate are all computed from the envelope of the request. Availability asks whether the HTTP handler returned a 2xx. Latency asks how long that took. The error rate counts exceptions raised and non-2xx statuses returned. None of the three opens the response body, because none of them was ever designed to have an opinion about it. A summariser that returns a fluent, confident, entirely wrong summary in 1.84 seconds is byte-for-byte indistinguishable, at that layer, from one that returns the right summary in 1.84 seconds — both are a 200 with a JSON body of about the same length.

It is worth retiring one piece of folklore before it costs an hour of an incident. Systems like this are often said to "drift", as though the model were slowly going out of tune the way an unserviced sensor does. Weights that have shipped do not change on their own, and a hosted model is not quietly degrading between your requests. When behaviour moves, something moved it: your prompt, your retrieval corpus, your configuration, the mix of inputs arriving, or something on the provider's side of the API. Treating a quality drop as an event with a timestamp and a cause, rather than as weather, is what makes the rest of the investigation tractable.

The row worth staring at is schema validation, because it is the one teams assume closes the gap. A summariser that must emit {"parties": [...], "effective_date": "...", "obligations": [...]} and does so, with a plausible date and three plausible obligations, passes every structural check ever written for it. Structure is cheap to verify and says nothing about truth. Adding more envelope-level monitoring does not converge on quality monitoring, because a percentile computed over a body-blind metric cannot be sharpened into a statement about correctness.

What justifies waking someone#

The justification for a page is user harm or a measured quality drop, and this alert is the second of those. What it is not is proof of the first. An eleven-point drop on a rubric is evidence that the system's behaviour changed; whether that change is currently hurting anyone depends on what sits downstream of the outputs, which is a property of your product rather than of the model.

For this system the contract-summary slice is 18% of traffic, roughly 4,200 requests a day. About 3,300 of those render a summary into a review pane where a paralegal reads it next to the source PDF and either accepts or edits it. The remaining 900 flow into an automated obligations extract that writes renewal dates, notice periods and auto-renewal flags into the contract lifecycle system, where a scheduler later fires reminder emails to account managers off those dates. The same eleven-point drop means two very different things across those two paths.

That table is the 02:00 decision. Two hours of degraded summaries in the review pane costs the paralegals some annoyance and produces no artefact anyone has to clean up. Two hours of degraded summaries flowing into the CLM produces 75 records with dates nobody can distinguish from the good ones without re-running them, and a handful of those will have already gone out as emails to a customer. So the containment move is to disable the automated extract path and keep the review pane serving with a banner, which takes one feature-flag flip and costs 900 requests a day of automation until the incident resolves. Notice that this decision required no theory about the cause. Containment is chosen from blast radius — how far a bad output travels and what it touches on the way — and reversibility, and those are both known at 02:14.

The same reasoning sets the bar in the other direction, which is the part that keeps a rotation sustainable. A two-point move on any slice is the harness talking to itself and belongs in a weekly review. A single customer complaint with no corroborating measurement is a ticket, because one bad output is the base rate of a system that is working correctly. An eleven-point drop on the redline slice, whose outputs are only ever read by a lawyer already editing them, is also a ticket for the morning — real, worth fixing, and harming nobody while everyone sleeps. What earns a 02:00 page is the intersection of a measurable drop and a downstream consumer that acts without a human in the loop, and writing that intersection down as the paging rule is what stops the eval suite from becoming a source of noise people learn to ignore.

Turning the drop into an alert instead of an escalation#

The alert exists because someone built a job that measures the thing the dashboards cannot. The shape is unremarkable: a fixed set of inputs with known-acceptable outputs, scored per slice, run on a cadence, compared against a rolling baseline with a threshold that accounts for the noise floor.

SLICES = ("contract_summary", "support_reply", "clause_lookup", "redline")

def run_eval(dataset, runner, scorer, run_id: str) -> dict[str, float]:
    """Score a fixed dataset and emit one mean per slice."""
    totals: dict[str, list[float]] = {s: [] for s in SLICES}
    for case in dataset:                      # 240 cases, 60 per slice
        output = runner(case.input)           # one production-path request
        totals[case.slice].append(scorer(case, output))  # 0.0 - 100.0
    means = {s: sum(v) / len(v) for s, v in totals.items()}
    emit_metric("eval.mean_score", means, tags={"run_id": run_id})
    return means


def check(means: dict[str, float], baseline: dict[str, float]) -> list[str]:
    """Alert on a drop larger than the harness's own run-to-run noise."""
    NOISE_FLOOR = 1.8   # measured: same set, same config, 20 repeats, 2 sigma
    THRESHOLD = 5.0     # ~2.8x the noise floor
    return [
        s for s in SLICES
        if baseline[s] - means[s] > THRESHOLD
    ]

The NOISE_FLOOR constant is the part most teams skip and then regret. Generation is not deterministic, so re-running the identical 240 cases against an unchanged system twice gives you two different means. Measure that spread before you pick a threshold: twenty repeats on this set produced a standard deviation of 0.9 points per slice, so anything under about two points is the harness talking to itself. A threshold at five points will not fire on noise, and an eleven-point move clears it by a wide margin.

The cadence is a cost decision. Each case in this set averages 2,400 input tokens and 600 output tokens. Running all 240 against claude-opus-5 at $5 per million input and $25 per million output is roughly $6.50 a run; four runs a day is about $26. A 60-case smoke subset every hour adds about $39 a day. Call it $2,000 a month to know within an hour that quality moved, against an engineering organisation where a single quality incident consumes several engineer-days. Where scoring uses a model as judge rather than a deterministic check, add the judge's tokens to that figure — on this set the judge runs at about 3,000 input and 200 output tokens per case, which roughly doubles the bill.

Two design choices inside that job decide whether its number means anything at all. The first is where the expected outputs came from. Sixty contract summaries reviewed and corrected by two paralegals is a week of somebody's time and produces a set you can defend in a meeting; sixty outputs from last quarter's model, skimmed and accepted, produces a set that scores the old system's habits and calls any change a regression. The second is how much of the score is deterministic. On this suite roughly 60% of the points come from checks that need no model — is the effective date present and does it match the source document, are the named parties a subset of the parties in the text, is the obligation count within one of the annotated count — and the remaining 40% comes from a rubric judged by a model. Deterministic checks are cheap, stable and narrow; judged checks cover the things that actually matter and contribute most of the variance you measured as the noise floor. Reporting the two components separately rather than blending them into one mean lets an incident distinguish "the system started omitting fields" from "the system started writing worse prose", which are different failures with different causes.

Explicit feedback confirms; it does not detect#

The tempting version of this signal is user feedback: put a thumbs-down on every output, chart the negative rate, alert when it rises. It is cheap, it needs no fixture set, and it measures real users rather than stored examples. It is also far too slow to be the thing that pages you, and the arithmetic shows why.

This product sees about 30,000 requests a day with a 0.4% explicit feedback rate, which is 120 ratings a day, of which roughly 6% are negative — seven thumbs-down. A degradation that pushes the negative rate from 6% to 9% takes that to eleven. Distinguishing eleven from seven, when the daily count already varies between four and ten on unaffected days, needs several days of data before the shift is visible above the noise. By then a customer has told your support team, which is exactly the escalation the alert was supposed to replace. Feedback is worth collecting because it tells you which outputs users hated, which is superb material for building the next fixture set, and because it corroborates a drop you have already detected. As a detector it arrives after the humans do.

What the fixed set still cannot see#

The eval set measures the distribution it contains. A regression that only shows up on a document format the set has no example of — a scanned contract, a language you added last quarter, a customer whose master agreement runs to 90 pages — is invisible to it, and the way you learn about those is still an escalation. Coverage is not a solved problem; it is a standing maintenance obligation, and a set that has not gained a case in six months is measuring last spring's product.

Three other blind spots are worth naming before they surprise you. The signal is lagging by up to its cadence, so an hourly job means an hour of degraded output before anyone knows. It measures quality rather than harm, so a system that produces excellent summaries of the wrong document, or leaks one tenant's clause into another tenant's summary, scores well while doing something far worse than a quality drop. And if the expected outputs in the set were generated by an earlier version of the same system and lightly reviewed, the set encodes that version's habits, so a genuine improvement can register as a regression.

What the alert has given the on-call engineer at 02:14 is a slice, a magnitude and a window. What it has not given them is a single comparable pair — one request from Monday that was good and one from Tuesday that was bad, side by side, with everything that differed between them visible. Without that, the investigation is a conversation about hypotheses. The next chapter is about why the tracing already on this system does not produce that pair either.