A recovery nobody explained will happen again

The 08:00 eval run posts at 08:12: contract-summary 70.9, against a baseline of 71.4 and a bad-window floor of 60.2. The other three slices are unchanged. Nobody touched anything after 03:19 — the prompt was put back to 4.3 at 03:40 when the revert made things worse, and since then the system has been running the same configuration that produced the regression, serving the same traffic, with the extract path disabled. The channel fills with the particular relief of a problem that solved itself, and someone asks whether we can stand down and go to bed.

The honest answer is that the system is in exactly the state it was in at 02:14, minus the symptom. Nothing was fixed, because nothing was identified. Whatever produced the drop is still in the system or still in the supply chain, and it has now demonstrated that it can turn itself on and off.

Everything that could produce this shape recurs#

The set of mechanisms that make a fixed eval score fall and then recover with no change on your side is short, and every member of it has a repeat interval. Working through them is a twenty-minute exercise, and it converts "we don't know" into "we don't know, and here are the four candidates with the earliest one due back on Sunday night".

What makes the table useful rather than decorative is writing the falsifying evidence before running any of the tests. For each candidate, one sentence: what would have to be true in the data for this to be the cause, and what observation would rule it out. Without that, four minutes of querying turns into an argument about whether a chart looks suggestive, and a candidate nobody can kill survives for weeks as a background belief that quietly shapes design decisions. With it, the morning produces a list where three rows read "ruled out, here is the query and its result" and one reads "consistent, cannot confirm without the provider" — which is a finding you can put in a postmortem — the written review that an incident produces on its way to being closed — and hand to a vendor.

For this incident the second row is testable in about four minutes, because chapter 2's retrieval attributes are already being recorded:

-- Was the index complete while the score was down?
SELECT
  date_trunc('hour', ts) AS hour,
  attributes->>'retrieval.index_build_id' AS build,
  count(DISTINCT attributes->'retrieval.results'->0->>'doc_id') AS distinct_top1_docs,
  avg((attributes->'retrieval.results'->0->>'score')::float) AS avg_top1_score,
  avg(jsonb_array_length(attributes->'retrieval.results')) AS avg_results_returned
FROM spans
WHERE name = 'retrieval'
  AND attributes->>'eval.slice' = 'contract_summary'
  AND ts > now() - interval '48 hours'
GROUP BY 1, 2
ORDER BY 1;

If avg_results_returned sits at 3.4 between 21:41 and 07:30 and returns to 5.0 afterwards, the incremental pass is the cause, the recurrence interval is one week, and the fix is a rebuild job that swaps the alias only when the new build is complete. If it sits flat at 5.0 the whole time, that candidate is dead and the remaining ones are all upstream. Either outcome is worth four minutes, and both are much easier to get now than on Thursday, when the trace content-capture window from chapter 2 has aged out.

Mitigated is a state; resolved is a different one#

The vocabulary matters here because the vocabulary is what the on-call rotation, the status page and the follow-up tracker all key off. An incident with the symptom gone and the cause unknown belongs in its own state, with its own exit criteria, rather than being rounded to either "still burning" or "done".

The edge from Mitigated back to Detected is the whole reason the state exists. A recovery you cannot explain is a hypothesis that the problem is gone, and the cheapest way to test it is to keep watching at incident intensity for a defined period rather than to close and rediscover it from a customer escalation.

Four things stay on while the incident sits in Mitigated. The eval cadence stays raised — hourly on the full suite instead of six-hourly, for 72 hours, at the roughly $150 that costs. Content capture stays at 100% on the affected slice, so that a recurrence arrives with evidence attached instead of sending you back to the sampling argument. Containment stays in place: the obligations extract remains disabled, because the justification for turning it back on was never "the score recovered" — it was "we understand why it moved and can tell when it moves again". And the customer cadence continues, which means the 14:00 update goes out saying that quality returned to normal at 08:00, that the cause has not been identified, that automated writes remain paused, and when the next update lands. A customer who is told the symptom is gone and the cause is unknown will trust the next thing you tell them; one who is told it is resolved and then sees it recur will not.

The opposite failure is real too. Heightened monitoring with no exit criteria becomes permanent overhead, and an hourly eval that nobody reads is worse than a six-hourly one that someone does. Write the exit criteria at the same time you enter the state:

incident: INC-2026-0623
state: mitigated
entered_at_utc: "2026-06-23T08:12:00Z"
exit_to_resolved_requires:
  - "72h of contract_summary >= 68.0 on the hourly suite"
  - "either a confirmed cause, or a written falsification of all five candidates"
  - "a detector that would fire within 60 minutes of a repeat"
  - "312 affected obligations records re-run and reconciled"
review_at_utc: "2026-06-26T09:00:00Z"
owner: "@rota-lead-rag"
if_unmet: "extend 72h once, then close as unexplained with the recurrence playbook"

The last line is the one that keeps this honest. Sometimes there is no cause to be had: the provider will not confirm cohort membership, the candidates all fail their tests, and the evidence has aged out. Closing an incident as unexplained is a legitimate outcome as long as you close it with the detector, the containment decision and a written playbook for the next occurrence, rather than by quietly letting the tracker item rot.

Closing as unexplained is only defensible if it ships with a playbook, and the playbook is short. It names the symptom as it appeared — which slice, which metric, what magnitude, what the dashboards showed — so the next person recognises it at a glance instead of re-deriving it at 02:00. It lists the candidates with their falsifying tests and this round's results, so nobody repeats the work. It states the containment move and names the flag that performs it. And it says what evidence to capture immediately, before anything ages out: traces for the affected slice, resolved model identifiers, and a snapshot of the resolved configuration. A recurrence handled in twenty minutes with that page open is a different event from the same recurrence handled over six hours without it.

The close has a customer-facing half as well, and it is the one teams skip because it feels like re-opening a settled matter. The accounts that received the 06:20 and 14:00 messages are owed a final note stating what was established, that no cause was identified, what remains paused and why, what changed so that a recurrence is detected within an hour rather than four, and that the 312 records were re-run with a count of how many changed. Customers who have been told the truth at each step generally take "we could not determine the cause" calmly. What they do not take calmly is silence followed by a repeat.

The postmortem when the suspected cause belongs to someone else#

The tempting root cause statement is "our model provider changed something". It might even be true, and it is useless as a finding, because it produces no action item you can own and it points the discussion outward — at a vendor who was not in the room and may never comment. Blameless postmortem practice is usually described in terms of not blaming your colleagues; the same discipline applies to not blaming your suppliers, and for the same reason. Clément Mihailescu's Blameless Postmortem Culture In Software Engineering makes the general case; the specific version here is that a supplier changing behaviour is a condition your system operates in, like a network partition, rather than an event you get to be surprised by.

The findings that produce work are about your own detection, attribution and recovery, and they fall out of three timestamps.

Change at 21:41, detection at 02:14 — four hours and thirty-three minutes, of which the eval cadence accounts for up to six hours by construction. The action item is the cadence and the trigger, and it costs money rather than cleverness: hourly smoke runs on the slices with automated downstream consumers.

Detection at 02:14, containment at 02:30 — sixteen minutes, which is good, and the reason it is good is that the blast-radius table from chapter 1 existed before the incident and the extract path was behind a flag. The action item is to extend that table to the slices that do not yet have one.

Containment at 02:30, attribution never. This is the expensive finding, and the two hours between 03:08 and 05:00 spent reverting components one at a time is its evidence. The action items are the manifest and the retained builds from chapter 3, so that the next incident's first move is a four-second pin rather than a two-hour search, and the eval-lane bisection that follows it runs against fixtures instead of customers.

There is a fourth item that only appears if someone looks for it. The provider ticket opened at 03:55 was three sentences long and contained no request IDs, no timestamps in UTC, no resolved model identifiers and no example inputs, which is why it had not been answered by 08:00. A vendor escalation is a data submission, and the telemetry from chapter 2 is exactly what makes it one: fifty request IDs with timestamps, the resolved model identifier on each, the rendered prompts for ten of them under whatever NDA covers that, and a one-line statement of the delta with units. That package gets a substantive answer; "our quality seems worse" gets a link to the status page.

What to do with all of this on Monday#

The through-line of this topic has been that every one of these incidents is decided by artefacts that had to exist beforehand. The scored fixture set with per-slice thresholds is what turns "it got dumber" into a page with a magnitude. The semantic attributes on the retrieval and generation spans, plus the slice label and the manifest ID stamped on every request, are what let you compare a good answer to a bad one and produce an exact list of affected records. The release manifest with immutable component hashes and retained index builds is what makes rollback an operation rather than a search. The tenant residency classification and the routing policy file are what turn a provider outage into a lookup instead of an irreversible judgement call. The update template with a cohort, a window, an established-versus-suspected split and a next-update time is what stops the communications decision from being made under pressure by whoever is most senior in the channel.

None of them take long individually. The fixture set is a week of work and a couple of thousand a month; the span attributes are an afternoon; the manifest is a day if your promotion path is already scripted; the tenant classification is a column and a backfill. What they have in common is that they cannot be built during the incident that needs them, and the incident that needs them will be a Tuesday at 02:14 with every dashboard green.

Pick the one your last incident would have wanted most, and build that one first.