A contributing cause names a mechanism, not a verdict

Seven months after the Calderwood incident, a different engineer is designing a document-versioning feature and searches the incident archive for anything about handbooks. She finds one entry, from March, and it reads in full: Root cause: the model hallucinated the parental leave entitlement and cited an out-of-date document. Mitigation: prompt updated to instruct the model to prefer the most recent source. Status: closed. There is nothing in that paragraph she can act on. She cannot tell whether the model invented the figure or read it correctly from a document that should not have been there, and those two possibilities have nothing in common — one is a generation problem addressed with grounding constraints or a different decoding configuration, the other is an ingestion problem addressed with a delete step that asserts on its own return value. The write-up cost someone twenty minutes and returned nothing, and the feature she is designing reintroduces the same defect.

"The model hallucinated" fails as a cause statement because it is a paraphrase of what was reported. The complaint was that the assistant said something untrue; the cause offered is that the assistant said something untrue. Nothing was added between the symptom and the explanation, so there is no place to attach a fix, no component whose behaviour is now understood differently, and no check anyone could write that would fail if the condition returned. The trace from chapter 1 already contained the material for a better statement: two chunks from benefits-handbook-2024.pdf with doc_version: null, outranking the current handbook, cited in the answer. The model read a document that was in front of it and reported its contents accurately.

Following the mechanism back until it lands on something owned#

The useful version of this incident is a chain, and each link is a claim you can go and verify. Retrieval returned two chunks from the superseded 2024 handbook. Those chunks were live in the index because the 12 February ingestion run for that tenant added the 2025 chunks but removed nothing. It removed nothing because the supersede step deletes by filter on doc_version, and the 2024 chunks had been written in April 2024, before that field was introduced, so their doc_version was null and no comparison against it matched. The call returned successfully with a count of zero, and the ingestion job logged that count at INFO and moved on.

def reindex_tenant(tenant: str, docs: list[Document], version: str) -> dict:
    """Write the new corpus, then retire anything from an older version."""
    upsert(tenant, chunk_all(docs), doc_version=version)

    resp = index.delete(
        tenant,
        filter={"doc_version": {"$lt": version}},   # null never satisfies $lt
    )
    log.info("superseded %d chunks for %s", resp["deleted"], tenant)
    return resp

The call and its response, taken from the 12 February job log, are unambiguous once you look at them.

POST /v1/indexes/t_calderwood/documents:delete
Content-Type: application/json

{"filter": {"doc_version": {"$lt": "2025.1"}}}

HTTP/1.1 200 OK
{"matched": 0, "deleted": 0, "took_ms": 41}

Two hundred, zero matched, zero deleted, and a green job. Every layer of the system reported success, which is why this ran for twenty days without anyone looking at it. Writing the chain down in that form gives the September reader something she can check against her own design in about five minutes: does her versioning scheme have a value that means "written before we had versions", and does anything assert that a supersede step removed a plausible number of chunks?

Where the chain stops is a judgement call, and the useful stopping rule is to keep going until you reach something a team owns and could have changed. "The filter did not match null" is a property of the index vendor's query semantics, which nobody here owns. "The supersede step trusts a returned count it never checks" is owned by the ingestion team and is directly fixable. "The doc_version field was added to new writes without backfilling existing rows — a backfill being the one-off pass that goes back over the rows already stored and writes the new field onto them — and no migration recorded that the two populations differed" is owned by whoever ran that migration in late 2024 and is the condition that made the null possible at all. All three go in the write-up, because the third is the one that generalises: any field added to a large index without a backfill produces a silent population of rows that every subsequent filter quietly skips.

What separates a cause you can use from one you cannot#

Four questions sort candidate statements quickly, and running them explicitly is worth the two minutes because a plausible-sounding sentence can fail all four.

The last row is the interesting failure. It is factually accurate, it points at a real omission, and it is still a bad cause statement, because the prevention it implies is that a named person try harder, which nothing can check and nobody can schedule. It also does the thing chapter 6 is about, which is teaching everyone watching that being the last person to touch a system is dangerous. The same fact restated as a condition — a schema addition landed without a backfill and without a migration record — points at a control that can exist, such as a migration checklist item requiring either a backfill or an explicit statement of how the filters handle the pre-migration population.

Causes are plural, and the singular version is where the fix goes wrong#

The phrase "root cause" arrived in engineering from industrial safety and manufacturing, where it fits systems whose components fail in ways that propagate along a mostly linear path, and where finding the one cracked bearing genuinely does end the inquiry. The practice of writing "contributing causes" instead came out of aviation and later out of the web operations community, on the argument that a complex system in production is normally running with several latent defects at once and only produces an incident when they line up. The Google SRE book's chapter on postmortem culture is the most-copied statement of that position for software teams, and it is a reasonable place to take the template from.

The Calderwood failure needed four conditions simultaneously. The version field had a null population. The supersede filter skipped nulls. Nothing asserted on the delete count. And the retriever had no recency or version preference, so a superseded chunk that scored 0.842 beat a current chunk that scored 0.702 purely on embedding similarity. Remove any one of those and there is no incident: with a backfill the filter works, with an assertion the job fails loudly on 12 February, with a version-aware reranker — a second scoring pass that reorders the retrieved candidates before any of them reach the prompt, using signals the similarity search does not see — the current handbook wins and the stale chunks never reach the prompt. That is what makes the counterfactual test useful for separating conditions from narrative — for each sentence you are about to call a cause, ask whether the incident still happens if that sentence were false. "The model hallucinated" survives its own removal, since the answer was wrong either way, which is the formal way of saying it explains nothing.

Writing all four also protects you from the fix that is cheapest to reach for. The March mitigation — an instruction in the prompt to prefer recent sources — addresses the fourth condition and only weakly, because the model cannot see ingestion dates unless you put them in the context, and adding a sentence to a prompt has no test attached to it. The mitigations that came out of restating the causes mechanically were an immediate one, a scripted delete of the 1,910 null-version chunks across the twelve affected tenants, which ran in eleven minutes, and a durable one, an assertion in reindex_tenant that fails the job when a corpus change produces zero deletions.

The shape of the entry, and what it is not#

The write-up that the September reader needed carries five things: the trigger, the scope, the contributing causes stated as mechanisms, the mitigations applied with their status, and the evidence each of those rests on. Evidence is the part most often dropped, and it is what makes the entry survivable — a cause with a job-log line and a trace id attached can be re-checked by someone who doubts it, and a cause with nothing attached is a claim the reader either accepts or ignores.

Scope deserves more care than it usually gets, because it is the field stakeholders read first and the one most often written as an impression. "A small number of tenants were affected" is unusable; twelve tenants, 63 conversations with at least one superseded chunk in context, and 19 answers that contradict the current policy on a figure or a date is a claim someone can act on, and each of those three numbers means something different. The first bounds the exposure, the second bounds what would need re-checking if the standard for harm turned out to be lower than you assumed, and the third is your current best estimate of actual defects. State how each was produced as well, since "19" from a replay of logged queries with an automated comparison against the current handbook is a different quality of number from "19" from someone reading conversations, and the reader in September needs to know which one they are inheriting.

Written out, the entry carries an id of its own — FM-2025-014, a failure-mode prefix followed by the year and a sequence number — so that a regression test, a ticket or a later write-up can cite it in one token rather than by retelling the story.

id: FM-2025-014
title: Superseded policy chunks served after a partial re-index
trigger: >
  12 Feb ingestion run for t_calderwood upserted 2025 handbook chunks;
  supersede delete matched 0 rows.
scope:
  window: 2025-02-13T08:55Z .. 2025-03-05T16:40Z
  conversations_with_stale_context: 63
  answers_contradicting_current_policy: 19
  tenants_with_null_version_chunks: 12
contributing_causes:
  - mechanism: doc_version added to new writes in Nov 2024 with no backfill
    evidence: migration MR-4471, no backfill step; 1,910 chunks with null
  - mechanism: supersede filter uses $lt on doc_version, so null never matches
    evidence: job log 12 Feb, delete response matched=0 deleted=0
  - mechanism: delete count logged at INFO, never asserted
    evidence: reindex_tenant, ingestion@1.9.2
  - mechanism: retriever ranks on similarity only, no version or recency prior
    evidence: trace cv_8b41d2e7, stale 0.842 over current 0.702
mitigations:
  - action: delete null-version chunks across 12 tenants
    status: done 2025-03-05, 1,910 chunks removed
  - action: assert deleted > 0 when corpus changed; fail the job
    status: shipped ingestion@1.10.0
  - action: version-aware rerank penalty for superseded docs
    status: open, owner retrieval team

Mechanical precision has a failure mode of its own, which is worth naming because it looks like rigour. An entry can describe the immediate mechanism in exhaustive detail — the null semantics of a comparison operator, the exact chunk ids, the cosine scores to three decimal places — while leaving out the condition that let a silent no-op run for three weeks. That kind of write-up produces a fix that is correct and narrow: the filter gets patched, the entry gets closed, and the next incident arrives from a different pipeline with the same shape. Precision about the proximate step is worth having, and it is not a substitute for at least one cause stated at the level of the process that produced it.

It also helps to be clear about what this activity is not. Naming mechanisms for one incident is a different job from the taxonomy work in topic 3.6, which reads a sample of failures and produces labels that two reviewers would apply the same way across hundreds of cases. A post-mortem cause is specific, evidenced and singular; a failure-mode label is general, defined by observable conditions and useful for counting. They feed each other, since a recurring post-mortem cause is a candidate label and a label with a rising rate is a prompt to go and write a post-mortem, and confusing them produces either a taxonomy of one-off stories or a post-mortem that says "grounding failure" and stops.

What this entry is not is an explanation of why nobody noticed for twenty days. The trigger line says 12 February and the window did not close until the mitigation ran on 5 March, and the arithmetic between them is sitting there unremarked. That interval is a separate finding with its own causes, its own owner and its own fix, and it is usually worth more than everything above it.