A rollback rule names a threshold on every axis, not just quality

A rollback rule is a file, written when a prompt is adopted, that says what has to remain true for it to keep serving traffic and what happens automatically when something stops being true. It names a fallback revision, a set of hold conditions, the events that force a re-check, and the action on failure. It exists because an optimised prompt is not a change you make once: it is a standing bet that a particular machine-written text is still worth more than the hand-written one it replaced, and that bet is re-opened by every model upgrade, every retriever change, every incident and every quarter that passes.

The alternative is to settle each re-opening as a conversation, and that conversation goes badly in a specific and predictable way. The moment of the decision is the worst possible moment to make it, for three reasons that have nothing to do with anyone's competence. The information is asymmetric, because the person deciding rarely ran the search and has no sense of how arbitrary the winning wording was. The framing is contaminated by sunk cost, because six hours of compute, ninety minutes of review and a release note all argue for keeping the thing. And the question "is a small margin against it still acceptable?" has no answer at all unless someone previously committed to what enough meant, so the discussion resolves on tone rather than on evidence.

Writing the rule months earlier is easy, because nobody is under pressure then and the person writing it does not yet know which way it will cut. That is the entire mechanism, and it is the same reason an alert threshold is set before the incident it fires on. What follows is the set of axes such a rule has to name, why quality alone is not one of them, and the form that makes it enforceable rather than aspirational.

The axes, and a threshold on each#

Everything that made an optimised prompt uncomfortable — its length, its latency, the fact that its structure was chosen by a machine — is invisible in the score. Each of those needs a number or a test that can come back false.

The quality row is where teams under-specify. "Still beats baseline" is not a threshold, because two prompts within noise of each other will trade places at random on every re-evaluation and you will end up flip-flopping the production pointer on the strength of 0.3-point differences. Requiring a stated margin and a bootstrap lower bound above zero, both measured in the same run against a re-scored baseline, means the pointer only moves when something has actually changed.

A threshold set too low fails in the other direction, and the failure is noisier than it sounds. One team ran the same check with min_margin_points at 0.1 and no bootstrap condition, wired to move the pointer automatically in both directions, and switched production between two prompts five times in three months. Not one of the five switches was supported by a paired interval that excluded zero; each produced a changelog entry, a line in the weekly summary and a period of several days during which nobody could say from memory which prompt was serving traffic. Two prompts within noise of each other will trade places on roughly half of all re-evaluations, so a low threshold on an automatic two-way switch is a random number generator wired to production.

That half is not a turn of phrase, and the reason it comes out at exactly a half is the reason the rule needs a margin rather than a comparison. If two prompts are genuinely worth the same, each re-evaluation draws fresh noise on both, and which one measures higher is settled by which cases happened to suit which prompt that morning. Nothing breaks the symmetry, so each verdict is a coin flip independent of the one before it. That is why the five switches carried no information and why a sixth would have carried none either. A stated margin fixes it by requiring the coin to land the same way and by a set distance, which noise on its own manages far less than half the time.

The rule here avoids that by being one-way. Adopting a prompt requires a merged pull request, a human review and a written annotation, while reverting requires only a failed condition, so the two directions have deliberately different costs and there is no cycle to flap around. If you do want the check to promote as well as revert, the thresholds have to differ: adopt above a margin of +1.0 points and keep above +0.5, so that a prompt hovering near the line stays where it is rather than oscillating. That gap is the same hysteresis a thermostat uses and for the same reason.

The thermostat is a picture rather than an argument, so the mechanism is worth stating in its own terms, and it is worth saying where the picture stops holding. With a single line, a prompt whose true margin sits near it measures above on some runs and below on others, and every crossing moves the pointer. With a gap between the line for adopting and the line for keeping, an already-adopted prompt has to fall all the way through the band before anything happens, and noise that moves a measurement by less than the width of the band cannot move the pointer at all. Where the comparison breaks: a room's temperature genuinely oscillates and drifts back on its own, while a prompt's true margin sits still and only the measurement of it moves. The band here is not damping a physical cycle. It is declining to act on differences that were never distinguishable.

Re-scoring the baseline in the same run is the part most often skipped, and it is the part that catches model upgrades. An upgrade moves both prompts, usually upward, and frequently moves the weaker one further because the compensating wording that earned the optimised prompt its points is no longer needed. Checking only that the optimised prompt "still scores fine" reads a rise in its absolute score as an improvement while the margin underneath has inverted.

Maintainability is the row that looks unmeasurable and is not. The test used here is that the prompt's directory contains a plain-language annotation, written by the human who merged it, saying what each block is for and what would break if it were removed. That document is cheap to produce for a prompt someone understands and impossible to produce for one nobody does, which makes it a real gate rather than a formality. A twenty-minute drill works as well: hand a team member a bug report, ask them to make a targeted edit and predict its effect, and see whether they can.

The rule as a file#

# prompts/drafting/rollback.yaml
active: drafting@v14-optimised
fallback: drafting@v11-baseline      # kept deployed, exercised in CI, never deleted
owner: "@priya"

hold_conditions:
  quality:
    holdout: evalset@a41f2c9#holdout
    min_margin_points: 1.0
    require_bootstrap_lower_bound_above: 0.0
    both_prompts_scored_in_same_run: true
  budget:
    max_prompt_tokens: 600
    max_added_p95_ms: 60
    max_cost_per_1k_requests_usd: 2.00
  maintainability:
    annotation_file: prompts/drafting/v14-annotation.md
    owner_required: true

recheck_triggers:
  - model_version_change        # includes provider-side pointer moves
  - retriever_or_chunker_change
  - tool_schema_change
  - grader_or_rubric_change
  - holdout_refresh
  - incident_touching_this_prompt
  - every_90_days

on_failure:
  action: switch_active_to_fallback
  notify: ["@priya", "#ai-platform"]
  open_issue: true               # re-run the search against the new baseline

The every_90_days trigger exists because the other triggers all depend on someone noticing a change and firing the check, and provider-side model updates are not always announced. Pinning an exact model identifier such as gpt-4.1-2025-08-07, rather than a floating alias such as gpt-4.1 that the provider repoints at newer builds whenever it likes, removes most of that risk, and the calendar floor covers what pinning does not.

Topic 4.10 established prompts as reviewed, versioned, pinned revisions, and this rule leans entirely on that. The two revisions are both real, both deployed, and the active one is named by configuration, so switch_active_to_fallback is a pointer change that takes effect in seconds without a build. That property is what makes the rule enforceable rather than aspirational: a rule whose failure branch requires a release, a rollback of unrelated commits and an approval is a rule people will argue their way around at 15:00 on a Friday.

Enforcing it belongs in the same CI evaluation the team already runs:

def check_hold_conditions(rule, results) -> list[str]:
    failures = []
    margin = results.optimised.mean - results.baseline.mean
    lower, _ = results.paired_ci(results.optimised, results.baseline)
    if margin < rule.min_margin_points / 100:
        failures.append(f"margin {margin:+.3f} below {rule.min_margin_points}pt")
    if lower <= 0:
        failures.append(f"paired CI lower bound {lower:+.3f} includes zero")
    if results.optimised.prompt_tokens > rule.max_prompt_tokens:
        failures.append("prompt tokens over budget")
    if results.optimised.p95_ms - results.baseline.p95_ms > rule.max_added_p95_ms:
        failures.append("p95 regression over budget")
    if not annotation_present(rule.annotation_file):
        failures.append("no annotation; maintainability threshold unmet")
    return failures

Two details in that function carry the design. The division by a hundred is the unit conversion between the file and the scores: the rule is written in points because points are how people argue about quality, while results.optimised.mean is a rubric score between zero and one, so a threshold of 1.0 points has to become a hundredth before the comparison means anything. That is the same hundredth the command-line output below prints as required >= +0.010. The second detail is that margin and lower are both computed off one results object, which is what enforces both_prompts_scored_in_same_run in code rather than in a comment — there is no path through this function that compares today's optimised score against a baseline figure somebody remembered from November.

When that returns a non-empty list the pipeline switches the pointer and opens an issue. The issue matters as much as the switch, because a revert is not a conclusion — the fallback baseline is now running against a newer model that neither prompt was searched against, and the right follow-up is a fresh search seeded from the baseline, which will find different things than the last run did.

The lifecycle has only two live states, and the asymmetry between the transitions is the design.

Entering the optimised state costs a review and an annotation, leaving it costs a config change that a job can make unattended, and there is no path from Recheck to anywhere except the two states that already exist. A design in which the failure branch went somewhere new — a hotfix prompt, a hand-edited variant, a third revision assembled under pressure — would put an unmeasured artefact in production at precisely the moment nobody has time to measure it.

Fallbacks that rot, and three things the rule is not#

The fallback has to be exercised for any of this to work, and this is where rules quietly rot. A baseline that has sat unserved for four months is a prompt written against a tool schema that has since changed a field name, or against a retriever that now returns a different passage format, and switching to it can produce a worse outage than the one that prompted the switch. Keeping it in the CI eval matrix, so it is scored on every eval run alongside the active prompt, catches most of that for the cost of a second scoring pass on the tuning set. Some teams go further and route a small slice of live traffic — one request in two hundred is enough — through the fallback continuously, which turns "does the revert work" from a belief into a graph.

It is not a circuit breaker. Circuit breakers act on live traffic in seconds, on operational signals such as error rate or timeout rate, and they belong to a different system with a different owner. The rule here acts on offline evaluation results at the timescale of a deploy or a migration, and its inputs are holdout scores rather than production telemetry. A team with a good rollback rule and no circuit breaker is still exposed to a prompt that starts producing malformed tool calls at 02:00.

It is also not a canary or a staged rollout. Those answer whether a change is safe to expose to more traffic; the rule answers whether a prompt has earned its continued place at all, which is a question you ask repeatedly over a prompt's whole life rather than once at release.

And it does not protect against anything the holdout cannot see. If the input distribution drifts — a new customer segment, a product launch that changes what people ask about — both prompts will be re-scored on a holdout that no longer represents production, the margin will look stable, and the rule will happily hold. That is a case for refreshing the holdout on a schedule rather than a case against the rule, and the holdout_refresh trigger is there so that a refresh forces a re-check rather than silently changing the baseline everything is compared against.

There is one honest tension in the design, which is that every trigger costs a holdout scoring and the holdout is a budgeted resource. A rule with seven triggers on a prompt that sits beside a pipeline under active development will fire more often than the quarterly floor suggests, and each firing spends a read. The mitigation is that a re-check scores both prompts in one pass and answers a question somebody was going to ask anyway, so the read is not wasted — but a team running several optimised prompts against one holdout should expect the trigger list, rather than the search itself, to be what depletes it, and should log the scorings per prompt rather than in one pool.

What a rollback rule buys is that the expensive judgement is made by people with time, and the cheap lookup is made by whoever is holding the ticket. Every axis in it exists because it was a live worry at adoption time, and writing the worry down as a number is what converts it from a thing people remember to a thing a job can check. The one-way design and the re-scored baseline are the two details that keep it from becoming noise.

All of this machinery — pinned revisions, a reviewed diff, a CI job that scores both prompts on a fixed holdout, an owner named in a file — was in place before any optimiser was involved. That is worth pausing on, because it settles what shape the search should take when it stops being an experiment and becomes something the team does every month.