The known-good state was a configuration, not an identifier

At 03:08 the engineer changes one line in the runtime configuration, moving the model identifier back to the snapshot that was serving on Monday afternoon, and triggers a 60-case eval run. Six minutes later it comes back at 61.4 against the bad baseline of 60.2 — inside the noise floor, so effectively unchanged. At 03:19 she reverts the prompt template from version 4.3 to 4.2 and runs again. This time the score is 54.8, and spot-checking ten outputs shows something new: four of them have an empty parties array, which had not happened in either the good window or the bad one. The system is now behaving in a way it has never behaved in production, and the clock has been running for sixty-five minutes.

Nothing here is incompetence. The move is the standard one — change a variable, measure, keep or discard — and it is the right instinct in almost every other kind of incident. It fails here because of what was promoted at 21:41, and because of what the eval score can and cannot tell you between probes.

What actually shipped at 21:41#

No deploy went out on Monday evening, which is why the incident's first hour was spent looking for one. What ran was the weekly index rebuild job, and that job does more than re-embed documents. Its pipeline reads a bundle definition from the retrieval repository, and that bundle pins the chunker version, the embedding model, the prompt templates written against that chunk format, and the model snapshot that the retrieval team last evaluated the bundle against. Someone had merged an upgrade to the bundle on Friday; the job picked it up on its Monday-night run and promoted the whole thing.

The dotted edge is where the trouble starts. Every solid edge in that graph is a dependency that was tested as part of a whole; the dotted one is a comparison against a number produced by a different whole. Chunker 5.2.0 stopped emitting the two-line document header at the top of each chunk, because the new embedding model scores better without it. Prompt 4.3 was rewritten to compensate, instructing the model to derive party names from the body text rather than reading them off the header. Prompt 4.2 still says "the parties are named in the first line of each excerpt", and against 5.2 chunks that line is a clause fragment. So the second revert produced a configuration where the instructions describe a document format that no longer exists, and the model dutifully reported no parties when it could not find them where it had been told to look.

Part of why this took an hour to see is that the word "release" had three different meanings inside the same organisation. The application team deploys twice a week through CI, and their pipeline knows about the service image and nothing else. The retrieval team merges bundle changes whenever they are ready, and those take effect when the rebuild job next runs, at 21:30 on Mondays. The provider ships changes on their own schedule and announces some of them. Three promotion paths, three cadences, one running system — and the incident channel spent its first hour asking "what deployed?", because the only path with a dashboard was the one that had not moved. The unit that needs a version is the combination that serves a request, and none of the three pipelines knows what that combination is unless something is built to record it.

That is the general shape of the problem. The bundle's components are not independent variables; several pairs of them encode assumptions about each other. Reverting one of a coupled pair produces a state that is not "half fixed" — it is a state nobody has ever run, and its failure mode carries no information about which component caused the original regression.

Bisection needs independence, valid intermediate states, and a cheap test#

The instinct to revert one variable at a time comes from an operation most engineers have run dozens of times. git bisect works because three things hold: the history is a line, every commit in it was a state someone intended and CI ran, and the test you run at each step is fast and gives the same answer every time. Change any of those and the search stops converging.

Here, all three fail at once. The four components were promoted together rather than in sequence, so there is no linear history to walk — the search space is the 16 combinations of four binary choices, of which exactly two have ever been evaluated as a whole. The intermediate states are not valid: prompt 4.2 with chunker 5.2, as the 03:19 probe demonstrated, is a combination that no one designed. And the test is neither fast nor deterministic. Each probe costs a 60-case eval run at roughly six minutes and $1.60, and its noise floor is the ±1.8 points from chapter 1, which means a probe that improves the score by three points is not distinguishable from one that changed nothing unless you run it two or three times. Sixteen states, at three runs each to see past the noise, is about five hours of measurement — while the automated obligations path is either disabled and costing the business its automation, or enabled and writing bad data.

There is a subtler failure worth naming. Because the score is a mean over sixty cases, two different defects can offset. A probe that fixes the retrieval regression and simultaneously introduces the empty-parties defect can score exactly the same as the untouched system, and a searcher steering by the mean will read that as "this variable does not matter" and discard the one change that was actually correct. Steering by a single aggregate through a space where multiple defects coexist is how a bisection can walk confidently away from the answer.

So the answer to the engineer who wants to keep going one at a time is no, and it is not a judgement about her reasoning. Bisection is the right tool once the incident is contained and the states are cheap; running it live means using production as the search space and users as the instrument.

Once traffic sits on a known-good configuration, that same search becomes cheap and boring. Sixteen combinations against the 60-case suite is about $26 of tokens, and run in parallel against a staging lane — a full copy of the serving stack that no customer traffic reaches — it is roughly ten minutes of wall clock. Because no user is exposed, you can afford three repeats per state to see past the noise floor, which costs another $52 and no additional time. The two hours the incident spent probing production would have bought the entire factorial — every one of the sixteen combinations, measured — twice over, and the offline version produces a better answer: with all sixteen states measured you can see the interactions, so the prompt-and-chunker coupling appears as a number in a grid rather than as a surprise at 03:19.

Rollback should name a state that was evaluated as a state#

What the system needed at 03:08 was one operation that put every component back to the combination that produced the 71.4 baseline. That requires the combination to have a name, which requires it to have been written down at promotion time.

# releases/rag-summariser/manifest-2026-06-09T03-30Z.yaml
manifest_id: rel_20260609_0330
promoted_at: "2026-06-09T03:30:11Z"
promoted_by: index-rebuild-job
components:
  model:
    id: claude-opus-5
    effort: high
    max_tokens: 4096
  prompt:
    template_id: contract_summary
    version: "4.2"
    sha256: "9f1c8b02d7…"
  retrieval:
    index_name: contracts-v3
    index_build_id: bld_20260609_0330
    embedding_model: text-embed-3-md
    chunker_version: "5.1.4"
    top_k: 5
  tools:
    schema_bundle: obligations_v7
    sha256: "3a44e19c05…"
evaluation:
  suite_sha256: "c07be4…"
  slice_scores:
    contract_summary: 71.4
    support_reply: 78.9
    clause_lookup: 83.1
    redline: 69.6
  noise_floor: 1.8

Two properties make this a rollback target rather than a changelog. Every component is identified by something immutable — a content hash for the prompt and the tool schemas, a build ID for the index — so the manifest resolves to exactly the bytes that ran, not to a mutable pointer that has since moved. And the evaluation block records both the scores and the hash of the suite that produced them, so a later run can tell the difference between "the system regressed" and "the suite changed".

The operation the engineer wants is then a single command with a single argument:

$ rollout pin --service rag-summariser --manifest rel_20260609_0330
resolving components…
  model          claude-opus-5 (effort=high)             unchanged
  prompt         contract_summary@4.2 sha 9f1c8b02       CHANGED from 4.3
  index          bld_20260609_0330 (text-embed-3-md)     CHANGED from bld_20260616_2141
  tools          obligations_v7 sha 3a44e19c05           CHANGED from obligations_v8
  routing        100% -> rel_20260609_0330
all components resolvable; applying atomically… done in 4.2s
run `rollout verify --manifest rel_20260609_0330 --suite contract_summary` to confirm

The word doing the work in that output is "resolvable". A pin fails loudly if any component no longer exists, and the component that most often no longer exists is the index build, because rebuild jobs are usually written to replace rather than to add. Retaining the previous two or three builds costs disk — for this corpus a build is about 40 GB, so keeping three is 120 GB, roughly $3 a month in object storage — and it is the difference between a rollback that takes four seconds and one that takes the six hours it takes to re-embed 1.4 million chunks with the old model. Prompt versions and tool schemas — the JSON declarations that tell the model which functions it may call and with what arguments — have the same requirement and cost nothing: never mutate a version in place, and never garbage-collect one that appears in a manifest.

The command's final line is not decoration. A pin that resolves and applies is not yet a rollback that worked, because the one component you cannot pin — anything living on the provider's side of the API — may be the one that mattered. Running the suite against the pinned manifest and comparing the result to the score stored in its evaluation block distinguishes "we are back" from "we are back on a configuration that no longer produces the score it used to produce". The second outcome is the most useful negative result an incident can get, because it says the cause is not in anything you version, and it says so in six minutes.

The manifest has an organisational precondition that is easy to miss while writing the YAML. Something must be responsible for producing it, and that something is whichever promotion path runs last, since only it knows the full combination. In practice the index rebuild job stops being a job that swaps an alias and becomes a job that resolves every component, writes a manifest, runs the eval suite against it, and moves traffic only if the scores hold — which is a larger change to that job than adding a file to a repository. Teams that skip it end up with three partial manifests, each describing one team's components accurately and the running system not at all.

Neither column identifies the cause, and that is worth being explicit about. A manifest rollback buys back the ability to investigate calmly by removing the users from the experiment. The bisection still has to happen; it happens afterwards, against the eval harness, in a lane where a broken intermediate state costs six minutes and no customer.

What a manifest cannot put back#

Three things fall outside the pin, and each of them ends an incident differently.

State that was written while the bad configuration was live does not revert. The 900 daily obligations extracts from chapter 1 that ran between 21:41 and containment are rows in a system of record, indistinguishable from good rows unless you can list the affected request IDs — which you can, from the traces of chapter 2, joined on the manifest ID stamped onto every request. Stamping release.manifest_id on the root span costs nothing and turns "we think about seven hours of extracts are suspect" into an exact list of 312 record IDs to re-run.

Anything the manifest points at that you do not control does not revert either. If the regression had come from the model behind an alias being re-pointed by the provider, pinning claude-opus-5 would have restored your intention and changed nothing, because the artefact your manifest names is theirs. That case has a different lever — routing traffic somewhere else — and it fails in ways the pin does not.

And a rollback is not a fix when the old state is itself unavailable. The next chapter picks that up in its sharpest form: the primary provider returning errors for forty minutes with no ETA, a runbook that says to fail over to a secondary that has never carried production traffic, and a substitute that can breach commitments the outage itself does not.