Reverting a commit depends on the pipeline that may be broken

At 02:14 the on-call engineer had the diagnosis and the fix. A prompt change had gone to 20% an hour earlier and was producing truncated answers on any conversation past four turns, at roughly 14 affected requests a minute. The fix was a one-line revert of the YAML file that pins assistant_prompt, and the pipeline that ships it takes twelve minutes on a good night: four minutes to build the image, six to run the suite, ninety seconds to push to the registry, two minutes for the rolling deploy.

The revert never merged. The suite includes the evaluation stage from topic 3.16, which runs 240 graded cases against the model provider, and the provider's mid tier was returning 529s, the status code a model provider sends when it is overloaded — which was itself the reason the prompt was misbehaving, since the retry path was silently truncating context. The eval job retried, timed out at eight minutes, and reported red. Branch protection requires a green run to merge. The override needs a second approver from a rota of four people, at 02:20 on a Tuesday, and the engineer got one at 02:41. Total time from diagnosis to fix in production: 51 minutes, about 700 affected requests, against a plan that said twelve minutes.

The team's retrospective action was to make the pipeline faster — split the suite, cache the layers, target eight minutes. That is a reasonable engineering goal and it would not have helped on this night at all.

The two mechanisms depend on different things#

Reverting a commit is not a fast operation with a long constant. It is a request for several independent systems to cooperate, in sequence, each of which can be unavailable for its own reasons.

Every box in A is a dependency you have to be lucky about. The eval stage needs the model provider, which is the thing most likely to be sick during an incident involving a model. The registry needs a network path and quota; teams that have hit an image-pull rate limit during an outage remember it. The rolling deploy needs pods to pass health checks, and if the bad change is causing memory pressure or timeouts, the health checks are exactly what is failing, so the rollout stalls with a mixed fleet. The scheduler needs capacity in a cluster that may already be scaled up unusually because of the incident.

Branch protection and required reviews sit on top of that, and they are the parts most likely to be waived under pressure, which is its own hazard. An engineer with production access, no sleep and a merge override is a configuration that has caused more than one second incident.

Path B depends on a write to a key-value store and a read that every pod is already doing. It shares almost nothing with A. That is what makes them different in kind: the question is not how many minutes each takes but how many independent systems have to be healthy for either to complete, and the incidents that make you want a rollback are correlated with exactly the systems A depends on.

The damage arithmetic follows from that. At 14 affected requests a minute, the nominal twelve minutes is about 170 bad answers and the actual 51 minutes was about 700. Path B at a 20-second refresh interval is around 10, counting the tail of pods that refresh late. Whether 170 or 700 matters depends entirely on what the requests are — 170 truncated answers in a support assistant is an unpleasant morning, 170 in a clinical triage tool is a different conversation — and the reason to care about the difference is that you do not get to choose which kind of incident you have.

None of this is new thinking, and it is worth knowing where it comes from because the vocabulary is borrowed. Progressive delivery, the family of techniques that exposes a change in widening stages instead of to everyone at once, grew out of the practice of separating deploy from release: get the code onto the machines as an inert artifact, then turn behaviour on with a switch that is not a deploy. Feature flags were the mechanism, and the argument for them was always about the release path rather than the build path, which is the same argument here. What is different with prompts and models is that the artifact being switched is not code at all — it is a string and a model identifier — so the temptation to treat it as ordinary application configuration, compiled in and shipped with the image, is much stronger than it ever was for a feature branch. Teams that have separated deploy from release for their application features quite often have not done it for the prompt, because the prompt did not look like a feature when it was first written.

Chapters 1 and 2 have already done most of the work of making B possible. The service resolves a prompt version and a model identifier from flag payloads per request rather than compiling them in, which means the value that determines behaviour is data. If it were still a constant in the source, no config store would help, because there would be nothing for the config store to hold. Rolling back by configuration and selecting artifacts by configuration are the same capability seen from two ends.

What the runtime path needs before it counts as a rollback#

The read has to be cached with a bounded, observable age, and it has to survive the store being down. Fetching config per request puts the store in the critical path of every call, which converts a config-store blip into a service outage; refreshing on a timer and serving the last good value on error keeps the store a control plane rather than a data plane dependency, meaning a system that configures the service rather than one every single request has to reach.

class ConfigClient:
    def __init__(self, store: Store, ttl: float = 20.0) -> None:
        self._store, self._ttl = store, ttl
        self._value: dict | None = None
        self._fetched_at: float = 0.0

    def get(self) -> dict:
        age = time.monotonic() - self._fetched_at
        if self._value is None or age > self._ttl:
            try:
                self._value = self._store.read("flags/assistant")
                self._fetched_at = time.monotonic()
            except StoreError:
                if self._value is None:
                    self._value = BAKED_IN_BASELINE  # shipped with the image
                metrics.increment("config.refresh_failed")
        metrics.gauge("config.age_seconds", time.monotonic() - self._fetched_at)
        return self._value

config.age_seconds is the metric that turns "thirty seconds" from a hope into a claim you can check. Alert when the fleet-wide maximum exceeds three times the TTL, the time-to-live the client is configured with and therefore the longest a pod should ever hold a stale value, because a pod that stopped refreshing is a pod your rollback will not reach, and the failure is silent — it keeps serving happily, with the old value, and it is the one your escalating customer is routed to.

BAKED_IN_BASELINE handles the cold start. A pod that comes up while the store is unreachable must start on a known-good configuration shipped in the image, because the alternative is a process that either crashes on boot or, worse, resolves an empty prompt and serves it.

The rollback target has to be something that has recently run. Recording last-known-good automatically makes the 02:14 decision mechanical rather than archaeological:

$ assistantctl config history assistant_prompt --limit 3
  assistant@61  set 2026-03-11 01:09Z by rgarcia   served 41,220 req  esc 9.8%
  assistant@57  set 2026-02-02 14:40Z by ci-deploy served 3.1M req    esc 3.4%
  assistant@55  set 2026-01-18 09:12Z by ci-deploy served 890k req    esc 3.6%

$ assistantctl config set assistant_prompt assistant@57 \
    --reason "INC-4412 truncation past turn 4" --incident INC-4412
  validated: assistant@57 in supported set, eval suite 2026-02-01 passed
  written at 2026-03-11 02:15:38Z, expected fleet convergence < 20s

The validation line is the part that stops path B from becoming a way to deploy untested configuration at speed. The value has to be an artifact that exists, that is in the supported-combination set from chapter 2, and that has a passing evaluation run recorded against it. That check is cheap because it reads metadata rather than running anything, and it rules out the class of incident where a rollback targets a prompt version that has never been served in production.

Approvals belong on the write path, and they should be asymmetric. Narrowing a rollout, disabling a flag or returning to a previously served value is an action a single on-call engineer takes alone, at any hour, with no second pair of eyes, because requiring a quorum to stop harm reproduces the 02:41 problem in a new place. Widening a rollout, or pointing a flag at a value that has not served production traffic, is a change with review. The two directions have different risk profiles and deserve different gates.

The failure mode you have just bought#

A thirty-second global lever is thirty seconds in both directions. Path A has a test suite in front of it; path B has a CLI and a person. The mitigations above — enumerated values, evaluation metadata, an audit trail with an incident reference — exist because the gate that used to be implicit in "you have to deploy to change this" is gone, and something has to replace it.

There is a version of the "make the pipeline faster" argument that is correct, and it is worth conceding properly rather than dismissing. A twelve-minute pipeline is bad for reasons that have nothing to do with rollback: it slows every change, it discourages small commits, and it makes people batch work in ways that raise the risk of each release. Investing in it is well spent. What it cannot do is change the dependency structure, because a four-minute pipeline still needs the build, the suite, the registry and a healthy rollout, and the specific night described above would have gone the same way with a suite that ran in ninety seconds — the eval stage would still have called a degraded provider, still gone red, still blocked the merge. Speed and independence are different properties, and only one of them is what a rollback needs.

Configuration also becomes a second source of truth about what production is running, and it will drift from the repository. The concrete version of this: the engineer sets assistant@57 at 02:15, the incident closes, and at 09:30 a routine deploy runs a GitOps reconciliation — an automated job that re-applies whatever the repository declares to be the desired state of the system — which syncs flag defaults from the repository, where the file still says assistant@61. The rollback silently un-does itself during business hours, and the second incident is harder to diagnose than the first because nobody deployed anything that looks related. The fix is a decision about ownership: either the store is authoritative and the repository holds only the baked-in baseline, or every runtime write is followed by a mandatory pull request that reconciles the repository within a stated window, tracked as incident follow-up. Both work. Having neither is the default and is where the drift comes from.

Propagation is not atomic, which matters for how you read the graphs afterwards. With a 20-second TTL and a fleet of 90 pods refreshing on independent timers, there is a 20-second window during which some requests are served by the old value and some by the new one, and any minute-granularity metric that straddles the write is a blend. That window is short enough not to matter operationally and long enough to make the exact moment of recovery ambiguous, so recording the config version on each request (chapter 6) is what lets you split the graph by what was actually served rather than by wall-clock time.

And there is a scope limit worth being honest about. Runtime configuration rolls back the things it selects — prompt versions, model identifiers, sampling parameters, retrieval index versions, tool schema versions. It does not roll back a change to the code that assembles the request, a new tool implementation, a schema migration, or a bug in the retry path. If the truncation at 02:14 had come from a change to the context-assembly function rather than from the prompt, path B would have had nothing to write, and the twelve-minute pipeline would have been the mechanism whether or not the provider was healthy. Making prompt and model selection runtime-configurable does not remove the need for a fast, reliable deploy; it removes the most frequent reason for needing one urgently.

The rollback path is now fast enough that the interesting question moves upstream. If recovery takes twenty seconds, the damage is set by how long the problem goes unnoticed rather than by how long it takes to fix, and noticing is what a canary is supposed to do — a deliberately small slice of live traffic put on the new configuration ahead of everybody else and watched for signs of trouble, named after the bird that went down the mine first. The next chapter is about why a canary that looks generously sized routinely fails to notice anything at all.