Pinning an approved revision so rollback becomes a configuration change

The previous chapter left a service whose prompt is a constant in its image, so the text the model reads and the text an author edits are the same object. Splitting those two apart is the whole move. Authoring stays where it was, slow and reviewed, and the serving path stops reading a file: it asks which approved artefact is designated for this prompt in this environment, and renders that.

The vocabulary is worth fixing before the machinery. A revision is one exact sequence of characters, named by an identifier derived from its own bytes, and it never changes. A pointer is a small record saying which revision is currently live for a given prompt and environment, and it changes whenever somebody decides it should. A control plane is whatever store holds those pointers — a configuration service, a feature-flag platform, a dedicated prompt platform, or a table behind an internal API — chosen so that a write to it takes effect in seconds without a build. Rollback, forward deployment and staged exposure are all the same operation on the third of those, which is why the split is worth the extra moving parts.

An id that can never mean two different things#

When a pull request touching prompts/ merges, CI publishes the prompt content as an artefact keyed by a hash of that content, together with the provenance of the merge:

# ci/publish_prompts.py
import hashlib, json, pathlib

def publish(path: pathlib.Path, meta: dict) -> str:
    body = path.read_bytes()
    digest = hashlib.sha256(body).hexdigest()[:8]
    revision = f"pr_{digest}"
    store.put(
        key=f"prompts/{meta['name']}/{revision}",
        body=body,
        immutable=True,
        metadata={
            "commit": meta["commit"],          # 7c1f9a2b
            "pull_request": meta["pr"],        # 2213
            "approved_by": meta["approvers"],  # ["t.beck"]
            "merged_at": meta["merged_at"],    # 2024-11-10T01:47:22Z
            "template_engine": "jinja2@3.1",
        },
    )
    return revision

The id is derived from the bytes, so pr_9d41c60e refers to one exact sequence of characters forever and a second attempt to publish the same content is a no-op. That property is what makes a rollback trustworthy: an engineer pinning pr_9d41c60e gets the identical text that was serving when it was last live, without needing to trust that nobody edited it in between. It is also what makes the artefact usable as a label on production traffic, which a later chapter in this topic, on the request-level record behind a response, leans on heavily.

Both of those properties come from one step, and the step is worth walking. The identifier is computed from the bytes rather than assigned to them, so two publishes of identical content produce identical digests and land on the same key — which is why re-publishing is a no-op rather than a second revision, and why nothing has to be deduplicated by hand. Run it in the other direction and the same arithmetic gives the guarantee an engineer cares about at two in the morning: any change to the text, including a trailing newline somebody's editor added, produces a different digest, so a revision id that resolves at all resolves to the bytes it was minted from. A name can drift away from its content. A digest cannot, because it is not attached to the content, it is derived from it.

Container images arrived at the same design a decade earlier and for the same reason. A tag like orders-agent:latest names whatever was pushed most recently, which makes it useless for saying what is running, so deployment manifests pin the digest — sha256:9d41c60e… — and the tag becomes a human-facing convenience that points at a digest. Prompts have exactly this structure: a name people use in conversation, a content-addressed id the system uses, and a mutable mapping between them that is itself a recorded decision.

That comparison is a precedent rather than a metaphor, and it breaks in one place worth knowing about. A container runtime verifies the digest: it pulls the bytes, hashes them, and refuses them if the hash does not match, so the digest is a check as well as a name. Nothing in the arrangement above does that by default. The revision store returns whatever it has under prompts/{name}/{revision}, and the guarantee that those bytes hash to that revision holds because the store was written to be immutable rather than because anybody re-checks. Closing the gap is one line in the resolver — hash the body you received and compare it to the id you asked for — and it is worth having in a system where the store is a vendor's, because it converts a promise in a contract into a property you verify.

Publishing on merge, rather than on demand from a web editor, is what carries the approval forward. The metadata says which pull request produced these bytes and who approved it, so a revision id in a config file is a claim that can be checked: given pr_7c1f9a2b, you can find PR #2213, read the diff, and see the review. A revision id minted by someone clicking save in a browser can carry the same metadata only if the platform enforces an approval before minting it.

A pointer that lives somewhere a deploy is not required to change#

The serving path does not ask for a file; it asks for whatever revision is currently designated for this prompt in this environment, and that designation is a small record:

{
  "prompt": "order_agent.system",
  "env": "prod",
  "revision": "pr_9d41c60e",
  "pinned_by": "s.okafor",
  "pinned_at": "2024-11-10T02:14:51Z",
  "reason": "incident 4471: tool-call errors 18% after pr_7c1f9a2b",
  "previous": "pr_7c1f9a2b"
}

What matters about the store holding that record is that a write to it takes effect in seconds, that every write is attributed and retained, and that it is a different system from the one that ships code — because the entire point is that the prompt's lifetime is no longer the release's lifetime.

Keeping the distinction between the two objects sharp is what keeps the design comprehensible. Revisions are content and never change. Pointers are decisions about which content is live and change often. Rollback is a pointer move, forward deployment of a new prompt is a pointer move, and staged exposure — several pointers for slices of traffic — is a later chapter in this topic.

Those three being the same operation is a consequence rather than a convenience, and it follows from where the mutability was put. In the previous arrangement the text and the decision about which text was live were the same object, so changing either meant rebuilding the object, which is why forward deployment and rollback both cost a release. Splitting them puts all the mutability in one small record and none of it in the content. Everything an operator ever wants to do at speed — put the old text back, put the new text out, give a tenth of traffic something different — is then a write to that record, and every one of those writes is the same size, takes the same time and leaves the same audit entry. A design where rollback is fast and rollout is slow has usually kept some of the mutability in the artefact.

Because pointer writes are the only thing standing between a person and production behaviour, the store holding them needs a short list of properties that are easy to skip when you build this yourself. Every write is retained with its author, timestamp and previous value, so the history of what was live is reconstructable months later rather than being a single mutable row. Write access is scoped per environment, so the set of people who can change staging is larger than the set who can change production. The record includes a reason, because "why did we pin this on the tenth" is the question the next investigation asks. And the pointer is validated at write time against the set of published revisions, since a pointer naming something that does not exist is a failure that surfaces at the worst possible moment, as the next chapter works through.

Environments follow naturally from the same structure. A revision published on merge is available everywhere at once, and promotion becomes a sequence of pointer moves rather than a sequence of builds: dev picks it up automatically, someone pins it in staging and runs the fuller eval there, and the production pin is a separate, attributed decision that may happen days later. The artefact is byte-identical across all three, which removes an entire family of "it worked in staging" investigations caused by a rebuild picking up a different template file.

The line between the two planes is where the fourteen minutes went. Authoring stays slow and reviewed, because that is where judgement about wording belongs. Choosing which approved artefact is live becomes fast, because that is an operational decision that has to be available at 02:14 on a Sunday.

What actually decides the repo-versus-platform argument#

Teams tend to frame this as a choice between two products: prompts as constants in the service repository, or a prompt platform with a web editor and an SDK — a software development kit, the vendor's client library that your service imports — that fetches the active version at runtime. Framed that way it turns into an argument about whether non-engineers should be allowed to touch production, which is the wrong question, because both arrangements can be built with or without the properties that matter.

Read down the first row and the decision rule falls out. What matters is which system is authoritative at serving time, and whether writes to that system carry review and produce an immutable id. A platform qualifies when the serving path pins an approved revision, and disqualifies when the editor can move live traffic the moment somebody clicks save. The web editor is not the hazard; unreviewed writes to the serving path are, and a text box that produces a draft revision requiring approval before it can be pinned is no more dangerous than a text editor on a laptop.

Authoritative at serving time is doing enough work in that paragraph to be worth an operational definition, because it is a phrase teams agree with while meaning different things by it. Change one system, change nothing else, and see whether production traffic behaves differently. Whichever system answers yes is the authoritative one, whatever the architecture diagram says. That test is worth actually running, because the common failure is a service that reads a platform at start-up and a repository at build time, so a change in either appears to work and the answer to what is production serving depends on when each instance last restarted. The belief a reader is likely to carry away from a vendor's marketing page is that adopting a prompt platform makes prompts versioned. What it makes them is saved, which is a different property: saving produces a record of what the text became, and versioning is only useful here when the serving path is required to name one of those records.

Vendors sell a third thing on top of the pointer, and it is the strongest argument for buying rather than building: linkage between revisions and production calls. A platform that sits in the request path can show you, for a given revision, the traffic it served, the outputs it produced, the cost and latency distribution, and the user feedback attached — which turns "we pinned pr_7c1f9a2b on Thursday" into a comparison you can read rather than a query you have to write. Assembling the same view from your own logs is entirely possible and is a week of work per view, so the honest way to make the build-or-buy decision is to price that work against the subscription rather than to argue about where text belongs.

Turning that into a purchasing conversation gives a short list of requirements. The platform must version every save with an immutable id you can log, gate promotion to production behind an approval by someone other than the author, expose role separation so that editing a draft and pinning production are different permissions, and offer an export API so the current text of every revision can be mirrored back into the repository on a schedule. A vendor that offers the first three and not the fourth is asking to be your only copy, which is a different risk and one the next chapter has to deal with anyway.

Where pinning stops helping#

Pinning solves the granularity problem and none of the others. It does not make a bad prompt good, and a revision id attached to sixty modified lines is exactly as unreviewable as the diff was in the first chapter of this topic. It does not tell you which revision to pin: the engineer above knew pr_9d41c60e was the last good one because the pointer record said so, and a team that has been flipping pointers casually all week has a history to read rather than an obvious answer.

It also does not cover the case where the prompt is fine and something else moved. A pointer flip is the right instrument when the prompt changed; when the provider silently updated a model snapshot behind the same alias, pinning the old prompt restores nothing, and the engineer who has just learned that prompt rollback is fast will reach for it first and lose ten minutes.

The arrangement is also not merely a worse repository or a better one. The repository gives you approval and a durable diff on the content itself, which a naive platform adoption throws away. The platform gives you something the repository structurally cannot, which is the pointer, and therefore rollback that is not a deploy. Holding both properties at once is what the third column of the table above describes, and the reason it is worth the extra moving parts is the entire content of the previous chapter.

The larger cost is structural, and it arrives immediately. The serving path now performs a lookup against a control plane on the way to answering a request, which means that store can be unavailable, slow or stale exactly when a request comes in, and there is a window between flipping the pointer at 02:14:51 and the last worker honouring it. That window, and what serves while the control plane is unreachable, is what the next chapter has to settle before any of this is safe to run.