One flag over a prompt and a model cannot be half-reverted

Three weeks after the support assistant's rollout flag assistant_v2 went to 20% of accounts, finance asked why inference spend had grown by roughly $360 a day against a plan that said $60. The engineering answer was straightforward once someone did the arithmetic. The flag switches two things at once: the prompt moves from assistant@57 to assistant@61, which adds a structured reasoning scaffold and six worked examples, and the model moves from the small tier to the mid tier. Input tokens per request go from about 1,900 to about 4,300 because of the prompt, and the price per token goes up 3.75× because of the model, so cost per request goes from $0.0028 to $0.021. On the 20,000 requests a day in the treatment arm that is $420 instead of $56, and at 100% of the 100,000 daily requests it projects to $767,000 a year against $102,000.

Quality had held. The resolution-without-escalation rate in the treatment arm was 76.2% against 71.4% in control, measured over 14 days, which is a real improvement on a metric the team had been trying to move for two quarters. Nobody wanted to give it up, and everybody had the same instinct about where the win was coming from: the reasoning scaffold and the examples, which are prompt work, not model capability. The obvious move was to keep assistant@61 and put it back on the small model, which would cost about $0.0056 a request — twice the original, not seven and a half times — and see how much of the 4.8-point gain survived.

That configuration could not be selected. There is one flag, it is boolean, and its two states are the pair of old things and the pair of new things.

The two cells you can reach and the two you cannot#

The two unknown cells are the ones that answer the question being asked. Without them the team has one comparison, between the cheapest configuration and the most expensive one, and the difference between them is the sum of two effects it cannot separate. The prompt might be carrying the entire 4.8 points, in which case there is a configuration on the table costing $0.0056 that is nearly as good as the one costing $0.021. The model might be carrying it, in which case the longer prompt is 2,400 wasted input tokens per request and should be reverted while the model stays. The effects might interact, with the scaffold only paying off on a model able to follow it, which is the case where the expensive cell genuinely is the only good one — and that is a finding, not an assumption to be adopted because the flag structure implied it.

Reaching those cells now requires a code change and a deploy, in the middle of a cost conversation with a deadline. That is the actual failure: the configuration you need under pressure is the one that was designed out, and it was designed out at the moment somebody named a flag after a release rather than after a decision.

Cost is the pressure in this example, and it is the friendliest version of the problem because nobody is being harmed while you work it out. The same structure produces much worse afternoons. A provider degrades the mid tier and p95 latency in the treatment arm, the figure 95% of requests come in under, goes from 2.1 s to 9.4 s; the only lever is a switch that also removes the reasoning scaffold, so the fix for a latency incident is a quality regression you did not ask for. Or the new prompt turns out to leak an internal field name into customer-visible output, and turning it off also drops you back to a model tier whose capacity you have already released. In each case the arm that needs reverting is one of two, and the switch operates on both.

Why one flag is tempting is worth naming, because the reasoning is not stupid. The flag was created to gate a piece of work — "assistant v2" — and the work was a prompt and a model together, developed together, evaluated together, and presented at review as one thing. Naming the flag after the project keeps the rollout plan legible to everyone outside the team, and it means there is one thing to turn off, which sounds like an operational virtue. It becomes a liability the moment the two halves have different problems, which is most of the time, because the flag was named after a unit of work rather than a unit of failure.

One flag per independently regressing decision#

The rule that avoids this is narrow enough to apply mechanically. Two changes belong behind separate flags when you can construct a plausible situation in which you would want to keep one and drop the other. Prompt and model pass that test immediately, because they fail for different reasons — a prompt regresses on instruction-following or format, a model regresses on cost, latency, provider availability or a capability you were relying on — and because their fixes are unrelated. A change to max_tokens from 800 to 1,200 that exists only because assistant@61 produces longer structured output fails the test, since reverting the prompt without reverting the limit is not a state anybody would want; that one belongs inside the prompt version.

Applied to this system the split looks like this:

[
  {
    "flag": "assistant_prompt",
    "unit": "account_id",
    "salt": "assistant_prompt.v61",
    "default": { "value": "assistant@57" },
    "rules": [{ "match": { "tier": ["pro", "enterprise"] }, "percentage": 20,
                "value": "assistant@61" }]
  },
  {
    "flag": "assistant_model",
    "unit": "account_id",
    "salt": "assistant_model.mid",
    "default": { "value": "small-2026-01-15" },
    "rules": [{ "match": { "region": ["us-east", "eu-west"] }, "percentage": 20,
                "value": "mid-2026-03-02" }]
  }
]

The targeting differs between the two, and that difference is not incidental. The prompt rollout is scoped by account tier, because the reasoning scaffold was written against the kinds of question pro and enterprise accounts ask. The model rollout is scoped by region, because the provider has not enabled the mid tier in eu-central and requests from there have to fall back. A single flag can carry one rule set, so the moment these two constraints exist together, the combined flag is either wrong for some accounts or restricted to the intersection of both conditions, which is a smaller and less representative rollout than either change deserves.

The service resolves both and carries both forward:

@dataclass(frozen=True)
class ResolvedConfig:
    prompt: str          # "assistant@61"
    model: str           # "mid-2026-03-02"
    reasons: dict[str, str]

def resolve(account: Account) -> ResolvedConfig:
    p = flags.evaluate("assistant_prompt", unit=account.id, ctx=account.context)
    m = flags.evaluate("assistant_model", unit=account.id, ctx=account.context)
    cfg = ResolvedConfig(
        prompt=p.value,
        model=m.value,
        reasons={"assistant_prompt": p.reason, "assistant_model": m.reason},
    )
    require_supported(cfg)
    return cfg

Different salts matter as much as different flags. Two flags bucketing on account_id with the same salt select the same 20% of accounts, which recreates the coupled rollout by accident while looking like two independent controls, and the dashboards will show four cells of which two are empty. Independent salts give you a genuine two-by-two: with both at 20%, roughly 4% of accounts land in the both-new cell, 16% in each mixed cell and 64% in the control. Whether 4% is enough traffic to conclude anything is the subject of chapter 4; the structural point is that the cell exists at all.

The combinatorics are real, and the answer is to declare rather than to test#

Two flags give four configurations, three give eight, and a service with a prompt flag, a model flag, a retrieval-index flag and a tool-schema flag has sixteen. You are not going to evaluate sixteen configurations on every change, and pretending otherwise is how teams end up collapsing everything back into one release flag to keep the matrix small.

What makes the matrix manageable is that most cells are never intended, and the useful thing to write down is the set that is. A supported-combination declaration turns "we never tested that" from a discovery at 03:00 into a rejection at config-write time:

supported:
  - { prompt: "assistant@57", model: "small-2026-01-15" }   # current default
  - { prompt: "assistant@61", model: "mid-2026-03-02" }     # candidate
  - { prompt: "assistant@61", model: "small-2026-01-15" }   # cost fallback
constraints:
  - reject: { prompt: "assistant@61", model: "small-2025-06-01" }
    because: "assistant@61 emits parallel tool calls; retired model returns one call per turn"

The three supported rows are the ones that go through the CI evaluation suite from topic 3.16 — including the cost fallback, which is not the configuration anybody plans to ship, and which is exactly the configuration you will want to reach in an incident. Evaluating the fallback costs one more suite run per change. Not evaluating it means the fallback is untested at the moment you need it, which is functionally the same as not having it.

The constraint block is the honest half. Prompt and model are separable most of the time and genuinely coupled some of the time, and the coupling is specific: a prompt written around a 200k context window on a model with 32k, a prompt that depends on parallel tool calls — several tool requests issued together in a single turn — which the older model does not emit, a prompt whose output format is only reliably produced by one model family. Those combinations should fail when someone tries to configure them, with the reason attached, rather than being unrepresentable because both changes ride the same switch. require_supported doing that check when configuration is written — not per request — is the difference between a validation error in a pull request and a fleet of pods serving a combination nobody has ever run.

What splitting the flags does not give you#

It does not give you attribution for free. Four reachable cells are four populations that each need enough traffic to say anything, and the mixed cells are usually the smallest, so a two-by-two rollout at 20%/20% has most of its statistical weight in the control. If the question is genuinely "which half is carrying the improvement", that has to be planned as a factorial rollout, meaning every cell is deliberately populated at once and sized for the effect you are trying to resolve, not read off a rollout that was sized for a single comparison.

It does not make a partial revert instant for work already in flight, either. A multi-turn conversation that started on assistant@61 and the mid model, and continues after assistant_model flips back, is now a conversation whose earlier turns were produced under one configuration and whose later turns are produced under another, with the earlier turns sitting in the context window as examples the model will imitate. Whether that matters depends on the product, and the way to find out is not to discover it during the revert. Resolving configuration once per conversation rather than once per request, and recording which conversations straddled a change, is the usual answer; it makes reverts slightly slower to take full effect and makes their effects interpretable.

It also does not make coupled changes safe to un-couple in the moment. Splitting the flags means the mixed cells are expressible; the supported list is what tells you whether one of them is a place you can actually go. A team that splits its flags and never evaluates a mixed configuration has bought optionality it cannot exercise, which is a better position than this chapter's opening but not the one it is aiming at.

And it does nothing about the mechanism. Both flags in this chapter are values in a configuration payload, and how quickly a change to that payload reaches a running request has not been discussed. Plenty of teams keep flag values in a YAML file that ships with the application, which makes flipping assistant_model back to the small tier a pull request, a build and a deploy. The next chapter takes that path apart, because the time it takes is the least interesting thing wrong with it.