Release and rollout

Read these first

• Prompt versioning, storage and review
• Evaluation in CI

What you will know

• Roll out a prompt change behind a flag

  The flag must select an immutable prompt artifact per request and support independent enablement, targeting, and disablement.

• Canary a model change on a fraction of traffic

  A small, representative cohort can reveal quality regressions, provider errors, and latency or cost shifts before they become fleet-wide.

• Roll back within minutes, without a deploy

  Recovery must change runtime configuration rather than depend on rebuilding or redeploying an application that may itself be unhealthy.

• Trace a production request to its exact prompt, model, flag, and evaluation versions

  Correlated release metadata is required to reproduce a bad answer and separate an artifact regression from user, retrieval, or provider effects.

• Run shadow traffic and compare candidate outputs before promotion

  Shadowing exposes behavior on realistic inputs without showing unproven responses to users, but requires careful handling of side effects and sensitive data.

Chapters

1. 1

   A flag that reads a mutable row cannot tell you what was served

   The flag is on for 10% of accounts, and the prompt text lives in a row the authoring tool updates in place, so an edit moves the treatment cohort underneath the experiment and the control has been drifting too. A flag has to select an immutable prompt artifact per request, with an edit publishing a new version rather than replacing the old one, or no regression can be pinned to anything.

   Read chapter
2. 2

   One flag over a prompt and a model cannot be half-reverted

   A single switch turns on a new prompt and a new model together for the same 20% of accounts. Quality holds, per-request cost jumps, and the cheap fix — old model with the new prompt — is the one configuration the flag cannot express. Give each independently regressing decision its own flag with its own targeting, and accept the combinatorial cost deliberately instead of meeting it mid-incident.

   Read chapter
3. 3

   Reverting a commit depends on the pipeline that may be broken

   Reverting and redeploying takes twelve minutes; writing a value the service reads per request takes thirty seconds. The gap is not ten minutes but a dependency chain — build, test suite, registry, healthy rollout — and the incidents where you most need to roll back are the ones likely to have broken part of it. Covers what a runtime configuration path requires first and the new failure mode it brings.

   Read chapter
4. 4

   A canary can be too small to see the failure it watches for

   Two percent of 100,000 daily requests over three days is 6,000 requests, which at a 1-in-500 base rate holds about twelve expected failures against twenty-four under a doubling — a gap Poisson noise routinely hides. Size is only half of it: a cohort taken as the first requests of each minute samples your schedulers, not your users. Includes the guardrail that should abort a canary without a human.

   Read chapter
5. 5

   A shadowed agent really does send the email

   A candidate agent with email, refund, and CRM tools genuinely refunds and genuinely emails unless something prevents it, and a staging CRM populated with real complaints is production data under weaker controls. Shadowing needs an inert tool layer that records intended calls, outputs that reach no user, and a comparison that examines decisions and tool calls rather than final wording — plus an owner for the week of output you just stored.

   Read chapter
6. 6

   Reproducing last week's answer needs what the alias resolved to

   A customer escalates a screenshot from the 14th. Since then the model alias has moved to a new snapshot, the prompt has been edited twice in place, and the flag controlling that cohort was switched off and reused. Learn what each request must carry — prompt version, resolved model identifier, sampling parameters, retrieval context, flag assignments — so 'how many of our users were affected' has an answer rather than an assertion.

   Read chapter