A tuned prompt is not a contract with the model

The first message came from a customer success manager on a Thursday: "is something different about the hero images? They look a bit… warmer?" The reply on the engineering side was that nothing had shipped in nine days. Error rate was flat at 0.4%, p95 latency was within its usual band, cost per accepted image had moved from $0.20 to $0.22, and the deploy log was empty. Everything that was being watched said the feature was healthy.

Reconstructing it took another two days. The provider had promoted a new default model version on the previous Wednesday, announced in a changelog entry nobody on the team subscribed to. The prompt, the reference image and the parameters were all unchanged. The output was warmer in the shadows, higher in contrast, and slightly more literal about compositional instructions, and none of those differences was large enough for any individual image to look wrong. In the eight days before anyone spoke up, 1,900 assets had been generated and published into customers' sites and campaigns, and there was no way to regenerate the earlier look, because a seed does not carry across versions.

Why the monitoring was quiet#

The instrumentation was watching the pipeline. A version change does not break the pipeline; it changes the distribution of a thing the pipeline cannot evaluate, and it presents exactly like the fourth failure class from chapter three — a valid response with a 200 on it — except at population scale rather than per request. The metric that did move, cost per accepted image going from $0.20 to $0.22, is a 10% shift that sits comfortably inside week-to-week variation and would take a month of data to distinguish from noise.

What makes this worse than an outage is the silence combined with the irreversibility. An outage is loud and produces no bad assets. A drift produces a week of subtly off-brand material that has already been published under a customer's name, and the only remedy after the fact is regenerating each asset with the new version and asking the customer to swap them, which costs their time rather than yours.

The instinct after such a week is to re-tune the prompt against the new version and subscribe to the changelog. Both are worth doing and neither is the missing piece, because the same event will happen again in six months against a prompt tuned for a model that no longer exists, and a changelog is a description of the changes a provider chose to describe. Silent behavioural shifts inside a version name — a quantisation change, meaning the model's numbers are stored at lower precision to make it cheaper to serve, different serving hardware, a safety-filter update — do occur and are not announced at all.

Two artefacts, one process#

The missing piece is two files and a schedule. The first file is the model version, pinned and treated as a dependency rather than as a constant embedded in a call:

# image/config.yaml — reviewed like any other dependency bump
generation:
  model: "image-gen-1@2026-03-11"      # NOT the floating alias "image-gen-1"
  size: "1536x1024"
  quality: "high"
  guidance: 5.5
  reference: "assets/brand-ref-04.png"
  reference_strength: 0.6
  negative_prompt: "text, watermark, lens flare, bokeh, oversaturated"

Most image APIs expose both a floating alias and dated or numbered version identifiers, and the alias is what appears in every quickstart, which is how it ends up in production. The alias means "whatever we think is best today", which is a reasonable default for a demo and an unreasonable one for a surface where six months of tuning is riding on the behaviour staying put.

The second file is a golden prompt set: a small, deliberately chosen collection of prompts with their parameters, checked into the repository beside the config. Twelve to twenty-four entries is the right size, and the selection matters more than the count. It needs one entry per distinct job the product performs — hero image, spot illustration, background replacement on a portrait, product-on-white — plus the cases you already know are fragile, because those are where a version change shows first.

[
  {"id": "hero-saas-01",  "prompt": "...", "seed": 8117, "n": 1,
   "tags": ["hero", "brand-palette"]},
  {"id": "portrait-bg-03", "prompt": "...", "source": "fixtures/phone-01.jpg",
   "mask": "fixtures/phone-01-mask.png", "n": 4, "tags": ["edit", "hair"]},
  {"id": "hands-hard-02", "prompt": "two hands holding a ceramic mug",
   "seed": 4402, "n": 4, "tags": ["known-hard"]},
  {"id": "signage-text-01", "prompt": "a shopfront with a plain awning",
   "seed": 991,  "n": 4, "tags": ["known-hard", "no-text"]}
]

Where a seed is available, one sample per entry is enough and the comparison is exact. Where it is not, four samples per entry make the comparison a distribution against a distribution, which is the point chapter five made about telling a real change apart from the dice.

Rendering the set, and what a machine can decide#

The runner renders every entry against both the pinned version and the candidate, stores the outputs as a pair, and computes the cheap metrics before any human is involved.

def render_golden(entries, model_version) -> dict[str, list[bytes]]:
    return {e["id"]: [generate(e, model_version) for _ in range(e.get("n", 1))]
            for e in entries}

def compare(pinned, candidate) -> list[dict]:
    rows = []
    for entry_id in pinned:
        a, b = pinned[entry_id], candidate[entry_id]
        rows.append({
            "id": entry_id,
            "delta_e_pinned":    mean(palette_delta_e(x) for x in a),
            "delta_e_candidate": mean(palette_delta_e(x) for x in b),
            "luma_shift":  mean(luma(x) for x in b) - mean(luma(x) for x in a),
            "sat_shift":   mean(sat(x) for x in b)  - mean(sat(x) for x in a),
            "text_pixels": mean(ocr_pixel_fraction(x) for x in b),
            "aspect_ok":   all(aspect(x) == entry_aspect(entry_id) for x in b),
        })
    return rows

The palette distance is the ΔE measure from the previous chapter, and it is the metric that would have caught the warmth shift in the opening incident: on that set, the mean ΔE against the brand palette moved from 4.9 to 8.6, which is well outside the run-to-run spread of about ±0.8 and is a number a threshold can act on. Luminance and saturation shifts catch the same family of drift. OCR pixel fraction — the share of the frame that optical character recognition, the standard technique for reading text out of a picture, believes is lettering — catches a model that has started rendering invented text on surfaces. Aspect compliance catches a parameter that silently changed meaning.

Automated metrics will find roughly the obvious half of what matters. They do not notice that the people in the images have started looking like stock photography, that the compositions have become more centred, or that the illustration style has moved from flat to gently three-dimensional — all real drifts that a brand team will reject and no scalar will flag. That is what the contact sheet is for, the term borrowed from film photography for a single sheet printed with every frame on the roll at thumbnail size, so a person can compare them in one glance: the pairs rendered side by side, old on the left, new on the right, in one page, reviewed by a named person with the authority to say no. Naming the person is part of the mechanism, because "the team reviews it" is a review that happens when there is time.

The weekly canary is the part that turns this from a reaction into a detector. A canary here is a scheduled run whose only product is a signal — the name comes from the caged bird carried into a mine to fail before the miners did — and it ships nothing itself. Once a week, render the golden set against the floating alias as well as the pin and diff the metrics. Nothing ships from it; its job is to tell you that the provider's default has moved, usually before the announcement reaches you and always before your pin is deprecated out from under you. At twenty entries with four samples, that is 160 generations a week, about $10 at the medium landscape price, which is less than the meeting it replaces.

Making the acceptance bar something a person can apply#

"A named person reviews the contact sheet" is only a mechanism if the person knows what they are deciding. Handing a designer forty image pairs and asking whether the new version is acceptable produces an answer that varies with their mood and with how the week has gone, and it produces no record of why. A rubric with three or four criteria, each answered per pair, fixes both problems at once and takes the reviewer about fifteen minutes for a twenty-entry set.

The criteria that work are the ones a brand team already argues about: palette fidelity, whether the composition respects the framing the prompt asked for, whether the rendering style matches the reference material, and whether there is anything present that would need removing before publication. Each is a yes or no per pair, and the gate is a threshold on the count rather than an overall impression — a candidate that fails style match on more than a fifth of the set does not ship, regardless of how good the best images look. The record of that decision, stored beside the config with the reviewer's name, the date and the counts, is what makes the next review a comparison rather than a fresh argument.

The golden outputs themselves belong in the repository as fixtures, stored as compressed thumbnails rather than full-resolution PNGs so that the directory stays a reasonable size. A twenty-entry set at 512 pixels on the long edge is around 4 MB, which is acceptable in version control and gives you something that a code review can actually show: a pull request bumping the pinned version arrives with the images that changed as part of the diff.

Rolling a version rather than switching to it#

Approving a candidate does not have to mean cutting over. The pin lives in configuration, which means it can be resolved per request, and routing a small share of live traffic to the new version for a week produces the one signal the golden set cannot: the acceptance rate real users give it. That is the multiplier from chapter four, measured on the population that matters, and it is the only measurement that distinguishes "different, and a reviewer was fine with it" from "better".

Ten per cent of traffic for a week on a feature doing 3,000 generations a week is 300 samples, enough to detect a shift in acceptance of about ten percentage points. If the new version's acceptance is materially worse, the pin never moves and the finding goes into the re-tune work; if it is better, that is a reason to move sooner than the deprecation date. The one thing to avoid is splitting traffic within a single user's session, since a user comparing four generations of the same prompt across two model versions is being shown an inconsistency they will report as a bug.

The three positions, and what each costs#

The third position is not free of the underlying problem, and it is worth being clear about the limit. Pins expire. Providers deprecate dated versions on windows that run from a few months to about a year, and once a version is withdrawn the images it produced cannot be reproduced at all — the exact model is gone. That is the second reason the first chapter's insistence on owning the bytes matters: the asset in your bucket is the only durable artefact, and the prompt, seed and version that made it are a record of provenance rather than a recipe you can re-run indefinitely.

Two further things this process does not do. It does not tell you whether the new version is better, only whether it is different; a candidate that shifts ΔE from 4.9 to 3.1 has drifted toward the brand and still needs a human to confirm the rest of the output is acceptable. And it says nothing about the failure modes that live entirely outside the model's aesthetics. Background replacement was in the golden set above with a mask fixture attached, and when that entry fails it usually fails for reasons no version pin addresses — the mask, its alpha channel, and the edges of the thing being cut out.