The most valuable line is how long it took to notice

Two numbers from the Calderwood entry sit at very different scales. The mitigation — a scripted delete of 1,910 chunks across twelve tenants — took eleven minutes to run and about three hours to write, review and get approved. The interval between the first wrong answer at 08:55 UTC on 13 February and the first person outside the system understanding that something was wrong, at 09:15 UTC on 5 March, was twenty days and twenty minutes. Everything the team did well is in the first number and everything it will pay for again is in the second, and the draft write-up recorded only the first.

That ratio is normal for quality failures and is the reason detection deserves its own line. A crashed service announces itself: the error rate goes to 100%, the pager fires, and time-to-detect is measured in seconds because the failure and the signal are the same event. An assistant serving a superseded policy figure produces a 200, a fluent answer, normal latency and a satisfied-looking conversation, so the failure and the signal are unrelated events, and the second one only happens if somebody built it. Between 13 February and 5 March, the system served 63 conversations that had a superseded chunk in context while every operational dashboard stayed exactly where it had been in January.

The awareness bar is the whole chapter. It starts when the system first produces a wrong output and ends when a human first knows, and in this incident it covers 96% of the elapsed time. Shrinking it is the only change available that improves every future incident of every kind, because it multiplies through: a detection interval of one day rather than twenty days would have made this incident 3 conversations rather than 63, would have preserved the index state and the traces that expired, and would have removed the customer-report path entirely.

There is a further reason this interval predicts cost rather than merely describing it. Diagnosis gets harder roughly in proportion to how long the failure ran, because the evidence in chapter 1's table expires on schedules that have nothing to do with your investigation: traces roll off at fourteen days, the index gets rebuilt, the prompt template moves on, the model provider updates the serving stack. Twenty days is long enough for four of those six evidence sources to have degraded, which is why a diagnosis that should take an hour takes four and why some incidents never reach a mechanism at all. The next incident's diagnosis budget is set by this incident's detection interval, and that link is what makes the detection line the most load-bearing sentence in the write-up.

Where the interval starts, and why that timestamp is contested#

Getting the number honestly requires a definition, and there are three candidates for the start. The first bad output is 13 February at 08:55, established by replay rather than observation, because nothing recorded it as bad at the time. The defect's introduction is 12 February at 09:14, when the ingestion run left stale chunks live, which is earlier and measures something different — that interval includes a day when the system was misconfigured and harmless, since nobody asked a parental-leave question until the following morning. The first bad output that a monitor could plausibly have caught is later still, if your candidate monitor needs a handful of events before it can distinguish signal from noise.

Pick the first bad output, and record the introduction time separately. Detection is about how long harm ran unnoticed, and a defect that harms nobody for a day is not detection debt. Being explicit also matters because the number gets compared across incidents later, and a team that silently switches definitions produces a trend line that means nothing.

The end of the interval is easier to get wrong than it looks. It is not when the ticket reached engineering; it is the first moment any human understood that the system was producing wrong output. In this case the employee read an answer he half-doubted on 4 March at around 11:00, mentioned it to his manager that afternoon, and the HR director raised it on 5 March at 09:15. If you take the engineering timestamp you record twenty days and drop a 22-hour report-path delay that is a real, separately fixable cost. Splitting it makes both visible.

def detection_metrics(incident) -> dict[str, float]:
    """Hours, split so each interval has an owner."""
    h = lambda a, b: round((b - a).total_seconds() / 3600, 1)
    return {
        "defect_to_first_bad_output": h(incident.introduced_at, incident.first_bad_output),
        "time_to_notice": h(incident.first_bad_output, incident.first_human_aware),
        "report_path": h(incident.first_human_aware, incident.reached_engineering),
        "time_to_diagnose": h(incident.reached_engineering, incident.cause_identified),
        "time_to_mitigate": h(incident.cause_identified, incident.mitigated_at),
    }

# Calderwood, FM-2025-014
# {'defect_to_first_bad_output': 23.7, 'time_to_notice': 458.1,
#  'report_path': 22.3, 'time_to_diagnose': 4.1, 'time_to_mitigate': 3.3}

Four hundred and fifty-eight hours against four hours of diagnosis is the shape of the finding. It also settles an argument that would otherwise consume the review, because a team looking at this entry with only a total elapsed time will spend its energy on making diagnosis faster, which was already the cheapest part.

Why no monitor fired#

Having the number is half of it; the entry needs the reason, and in practice the reason is one of three things. Naming which one applies decides who owns the follow-up and how much it costs, so it is worth being specific rather than writing "we lacked observability".

The three are easy to confuse in a review, since all of them present as "we did not know", and they have almost nothing in common as work. An earlier incident in the same archive, where the assistant began refusing a class of eligibility questions after a prompt change, was the first kind: no metric anywhere counted refusals, so no amount of re-slicing existing data would have surfaced it and the fix was a new detector plus a month of tuning its threshold. A later one, where a tool timeout silently degraded answers for six hours, was the third kind: an alert did fire, at 04:12, into a channel that had been created for a decommissioned service and had no owner, and the fix was twenty minutes of routing configuration. Recording which kind it was is what makes the follow-up estimable, and a team that writes "detection gap: we lacked observability" three times in a row has recorded nothing that can be scheduled.

Calderwood was the second, which is the most common and the most annoying, because the data was there the whole time. The pipeline emitted retrieval.stale_doc_ratio, defined as the fraction of retrieved chunks whose source document had been superseded, and it was alerted at a 2% threshold computed across all traffic. Across 340 tenants the platform served roughly 3.2 million retrievals a month, and the stale chunks in this incident accounted for about 9,800 of them, which is 0.31% — well under the threshold, and inside the normal band, since a few tenants always have a document mid-supersession.

Sliced by tenant, the same data is loud. Calderwood's stale ratio over the window was 41% of retrievals, and for queries that touched leave or benefits it was effectively 100%. The alert rule that would have fired is not a new metric; it is the same metric with a tenant_id dimension and a max-over-tenants rule with a volume floor to keep small tenants from producing noise.

# before: one series, one threshold
- alert: StaleRetrievalContext
  expr: sum(rate(retrieval_stale_chunks[1h])) / sum(rate(retrieval_chunks[1h])) > 0.02
  for: 30m

# after: worst tenant, with a floor so low-volume tenants do not flap
- alert: StaleRetrievalContextPerTenant
  expr: |
    max by (tenant_id) (
      sum by (tenant_id) (rate(retrieval_stale_chunks[6h]))
      / sum by (tenant_id) (rate(retrieval_chunks[6h]))
    ) > 0.15
    and on (tenant_id) sum by (tenant_id) (rate(retrieval_chunks[6h])) > 0.02
  for: 2h
  labels: {owner: retrieval-oncall}

Backtesting that rule — replaying it over the metric history that was already being stored, as though the rule had existed at the time — puts the first firing at 13 February, 14:10 UTC, five hours and fifteen minutes after the first bad output rather than twenty days. Backtesting also produced two false positives over the preceding ninety days, both tenants genuinely mid-migration for under a day, which is the cost of the change and belongs in the entry alongside the benefit.

The averaging trap generalises past this incident. Any quality metric aggregated over a population where failures are concentrated in a slice will hide them in proportion to how narrow the slice is, and AI failures concentrate by construction — by tenant, by document, by language, by intent, by prompt version. A dashboard showing overall groundedness at 0.91 is compatible with one tenant at 0.34 and a rounding error in the mean, which is why the useful alert is almost always on the worst slice above a volume floor rather than on the average.

In the entry itself this becomes a short section with four fields — the interval, the gap class, the evidence that the gap was what you say it was, and the detection change with its own owner and status. The evidence field is the one that keeps this honest, because "the alert would have fired on 13 February" is a claim about a rule that did not exist, and the only support for it is a backtest over the stored series that someone else can re-run. Without that, detection findings drift into wishful engineering, where every post-mortem concludes that a hypothetical monitor would have caught it and none of those monitors ever get built or measured.

What the number does not do#

Time-to-notice is a property of your detection, and it makes a poor performance review. A team that starts being judged on it will find the cheapest way to lower it, which is to alert on things that are easy to detect and rarely matter, and the average will improve while the incidents that hurt keep arriving from directions nothing watches. If you track it, track it alongside the false-positive rate and the fraction of incidents whose first signal came from a customer, because those three move against each other and only the set of them says anything.

The number also cannot be driven to zero, and pretending otherwise produces monitoring theatre. Some failures are genuinely undetectable by any monitor you would be willing to run: a summary that omits a clause the customer cared about, a tone that reads as dismissive to one user, a recommendation that is defensible and wrong for this particular account. For those, "no feasible monitor" is an honest entry, and the follow-up moves to a different lever, which is the report path — the 22.3 hours between an employee doubting an answer and an engineer hearing about it. An in-product flag on the answer itself, routed to the same queue as alerts, would have cut most of that, and it is a cheaper change than any detector for the same class of failure.

There is a third limit worth stating plainly. Detection tells you something changed; it does not tell you what, and the alert above would have said "Calderwood retrieval is stale" with no view of the null-version mechanism from chapter 2. The value of catching it on 13 February is that the trace is still there, the index has not been re-chunked, and the investigation starts with a system that still contains its own evidence. Detection buys you a cheaper diagnosis, not a diagnosis.

With the interval recorded and its cause named, the entry now says what happened, why, and why nobody saw it. What none of that does is stop the same mechanism from returning under a new prompt version in June, because a document is not a control, and turning the finding into something that executes is the next problem.