Prompt versions and generation records

Quality metrics on a model-backed feature move for reasons that are usually invisible in the request body. The prompt was edited, or it was not; the provider moved the model behind the name you asked for; a retrieval index was rebuilt overnight; a tool started returning an extra field; a feature flag went from ten percent to fifty. Every one of those changes the output distribution, and none of them leaves a mark in a stored conversation unless somebody wrote a column for it. That is what a version pointer is for: a value stamped on each generated message that names one input precisely enough for a later GROUP BY to separate the traffic that got it from the traffic that did not.

The column has to name a thing, not a moment#

Here is a week of assistant messages grouped by a prompt_version column holding the git SHA of the service at request time, against the rate at which users pressed thumbs-down:

 prompt_version | messages | thumbs_down_pct | first_seen
----------------+----------+-----------------+------------------
 a3f19c2        |   18,402 |            3.9  | Mon 08:12 UTC
 7bd44e1        |   12,733 |            4.1  | Mon 16:44 UTC
 c02e8aa        |   21,908 |           10.6  | Tue 09:31 UTC
 5f1b930        |   15,254 |           11.2  | Wed 11:02 UTC

The boundary is unambiguous: everything from c02e8aa onward is bad and everything before it is fine. The table is also incapable of telling you anything, and the reason is structural rather than unlucky. Sorting messages by a deploy identifier and sorting them by created_at produce the same partition, because a new deploy identifier appears exactly when a deploy happens, so grouping by it can never distinguish a cause from anything else that occurred at the same time. It is a timestamp with extra steps.

That equivalence is worth walking, because the table looks like evidence and the argument against it is not obvious. Each message is stamped with whichever SHA was live when it was generated. A SHA becomes live at one instant and stops being live at the next deploy, and no message is generated under a SHA outside that window. The set of messages carrying c02e8aa is therefore exactly the set created between Tuesday 09:31 and Wednesday 11:02 — the same set a WHERE created_at BETWEEN clause would return, reached by a different route. Grouping by the column is grouping by a time interval whose boundaries happen to be deploy times.

Everything else that changed inside that interval is consequently inside the same group, contributing to the same rate, with nothing in the data to distinguish it. The clean boundary is not a sign that the cause was found; it is guaranteed by construction, because the column changes only at deploys and so can only ever produce boundaries at deploys. A column that can only draw its lines in one place will draw them there whether or not that is where the cause is.

A version pointer is useful when two conditions hold: it changes when and only when the thing it names changes, and the thing it names can be fetched and read afterwards. A service SHA fails both. It changes on every deploy, including the several a day that touch nothing relevant, which fragments your traffic into groups that differ by nothing. And it fails to change when the effective behaviour changes outside the repository, which is most of the interesting cases — including a repointed model alias, an alias being a model name that stands for whichever build the provider currently serves under it, as against a pinned identifier that names one build for as long as it exists.

Both halves of that failure come from one property, and it is easier to see by following the prompt text itself into the request. A deploy identifier is computed from the state of a source tree at build time, so it can only be a faithful name for things that were in the tree at build time. The string the model actually received arrived by a different route: a template fetched from a row somebody edited in an admin screen, or a file in the image, or a value read from a config service at boot, and in every case with variables bound into it per request. Only the middle option is inside the artefact the SHA names, and even then the SHA covers the whole tree rather than that file.

Naming the artefact rather than any input is what produces both symptoms at once. A probe timeout and a prompt edit are the same event to a SHA, and an alias repointed on the provider's side is no event at all, because the tree it was computed from never saw either kind of change arrive.

The practice of stamping a release identifier on every record and grouping metrics by it comes from ordinary web services, where it works well, and it works there for a reason that does not carry over. In a conventional service the deployable artefact really does contain the behaviour: the code that renders the page, the query that fills it, the business rules that decide what it says. Behaviour outside the artefact is limited to data, and data changes slowly and visibly. A model-backed service inverts that ratio, because the largest single determinant of the output is a set of weights hosted by someone else, the second largest is a block of text that may or may not live in the repository, and the third is a corpus that is rebuilt on a schedule nobody watches. The release identifier still names the artefact faithfully; the artefact is simply no longer where the behaviour is.

Six of those seven rows are cases where the SHA is either noise or silence. Reading the second column down: one row where it moves and something real changed, one where it moves and nothing did, and five where the behaviour of the service changed and the column sat still. That is why the fix is not a better single column: no one value can name seven independent things.

A team whose prompts live in the repository will object that the top row is their whole world, and the objection is half right. Where the text is a file in the tree, the SHA does determine which text ran, in the sense that the file can be fetched back out of that commit. It still fails the two conditions. It fragments, because it also moves for the deploys that touched no prompt, splitting one prompt's traffic across groups that differ in nothing and shrinking each one until the rates are noise. And it stays silent on the five rows underneath, which are changes the repository never saw. Prompts in git make the SHA a precise answer to a question the investigation is not asking: not which text, but which build.

Write what you used, not what the config says#

Two tables carry it. The first is a content-addressed store of prompt text, keyed by the hash of what was rendered:

CREATE TABLE prompt_versions (
  id          bytea PRIMARY KEY,        -- sha256 over template + role + params schema
  name        text NOT NULL,            -- 'summarize'
  label       text,                     -- 'v3', set by humans, not load-bearing
  template    text NOT NULL,
  variables   text[] NOT NULL DEFAULT '{}',
  created_at  timestamptz NOT NULL DEFAULT now()
);

The primary key being a hash of the content is what makes the pointer honest. Two deploys that ship identical prompt text land on the same id, so the traffic does not fragment; an edit of one word produces a different id the moment it is first used, with no release process to trust and no human remembering to bump a label. The label column exists because people need to talk about "v3", and it is deliberately not the key, since labels get reused and hashes do not.

Both of those properties come from the key being a function of the content and of nothing else. Equal text gives an equal key, which is what keeps two deploys of the same prompt inside one group instead of splitting it. Different text gives a different key, so an edit cannot ship without announcing itself, and a prompt somebody pasted into a config screen at seven on a Friday gets a distinct id the first time it renders even though nobody registered anything. Neither direction depends on a person remembering a step, which is the only kind of versioning that survives the week it is most needed.

What goes into the hash needs saying plainly, because the obvious reading produces a table with one row per message. The digest covers the template as assembled — its text, any shared preamble folded into it, the role it is sent under, the schema of the parameters it expects — and stops before the user's values are bound in. Those values are recorded separately, on the generation row. Hashing after binding would give every request its own id, and a dimension with one row per fact is not a dimension. Hashing only a fragment of what was assembled would do the opposite damage, letting two prompts that differ in their preamble collide under one id and report as one population.

The second table records what one message generation actually resolved to:

CREATE TABLE generations (
  id                 uuid PRIMARY KEY,
  message_id         uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
  prompt_version_id  bytea NOT NULL REFERENCES prompt_versions(id),
  bound_variables    jsonb NOT NULL DEFAULT '{}',
  model_requested    text NOT NULL,      -- 'claude-sonnet-4-5'
  model_resolved     text NOT NULL,      -- echoed back by the API
  sampling           jsonb NOT NULL,     -- temperature, top_p, max_tokens, thinking budget
  tool_schema_hash   bytea,
  retrieval_snapshot text,
  flags              jsonb NOT NULL DEFAULT '{}',
  service_sha        text NOT NULL,
  provider_request_id text,
  input_tokens       int,
  output_tokens      int,
  created_at         timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX generations_prompt_idx ON generations (prompt_version_id, created_at);

model_resolved takes its value from the response body rather than from the request, because the request only carries what you asked for. When an alias is repointed, model_requested stays constant and model_resolved steps to a new string on a specific request at a specific second, and a group-by on it separates the two populations directly.

tool_schema_hash covers the case where a tool's own team adds a field to its response and every conversation that calls it starts behaving differently, with nothing in your repository having changed. retrieval_snapshot names the index build, which is the difference between "the prompt got worse" and "the documents got re-chunked at 02:00". flags captures the ramp state at request time, so a 10%-to-50% rollout is analysable after the fact rather than reconstructable from a dashboard's audit log. service_sha stays, because it is still the fastest way to get from a suspicious generation to a working tree, and it is now one field among several rather than the field that carries the whole argument.

The direction model_resolved travels is the rule for every column above. Each one holds what was resolved at the moment of the call, taken from whatever did the resolving — the response body for the model, the flag evaluator for the ramp, the retrieval client for the snapshot — rather than looked up afterwards from the system that decided it. Those systems are mutable and mostly keep no history you can join against: a flag service answers for today, a config dashboard shows the temperature in force now, and an alias resolves to whichever build is current. Writing the row once the response comes back is fine, since the values are already in hand by then. Re-deriving them a week later is not, and no column in the schema can tell the two apart, because both leave a populated field.

Both tables are append-only, and that has to be enforced rather than intended. A row in prompt_versions can never be updated, because the primary key is derived from template, so an in-place edit would leave a row whose key no longer describes its contents and would silently retag every historical message that pointed at it.

"Silently" is the word to slow down on there. A generation names its prompt by id and carries no copy of the text, so the id is where the whole meaning of the pointer sits. Edit the template in place and every historical row still points at that id, still resolves, and now resolves to a string that was never sent to a model — a report on how that version performed becomes a report on text nobody ran. Nothing fires, since the foreign key is as satisfied as it ever was. The check that would catch it exists only because the key is derived: recompute the digest of template, compare it against id, and an edited row fails to match its own key.

The cheapest enforcement is a rule that revokes UPDATE on the table for the application role and a trigger that raises on any attempt, which costs nothing and removes an entire category of "we fixed a typo in the prompt" incident. The write path is an upsert at render time: compute the hash of the rendered template, INSERT … ON CONFLICT DO NOTHING, and use the hash. Nobody registers a version, which matters because the versions you most want recorded are the ones somebody shipped in a hurry.

Once the pointers exist, the analysis they support is the one a regression investigation wanted in the first place:

SELECT g.prompt_version_id, g.model_resolved,
       count(*)                                              AS n,
       round(100.0 * avg((f.rating = 'down')::int), 1)       AS down_pct
FROM generations g
JOIN feedback f ON f.message_id = g.message_id
WHERE g.created_at >= now() - interval '7 days'
GROUP BY 1, 2
HAVING count(*) > 500
ORDER BY down_pct DESC;

That query partitions a week by two independent axes at once, which is what turns a boundary in time into a statement about which input moved.

What the second axis buys is a comparison in which one candidate is held still. If a single prompt version's traffic splits across two model_resolved strings and the down-rate differs between them, the prompt was constant across that comparison and cannot be what moved. If one model string spans two prompt versions and the rate differs there, the same reasoning runs the other way. One axis can only report that something changed at a boundary, because every candidate changed at that boundary together. The HAVING count(*) > 500 floor does an unglamorous version of the same job: a group of a dozen messages produces a spectacular rate out of two complaints and would otherwise lead the sort.

Storage, and why not simply keep the whole prompt#

The tempting simplification is to store the fully rendered prompt on every message and skip the version table entirely, which is unambiguous and needs no discipline. The cost is real but not prohibitive: a 3,000-token system prompt is roughly 12 KB of UTF-8, so at 3 million assistant messages a month that is about 36 GB a month before compression, against a few hundred rows in prompt_versions holding the same information. TOAST compression on Postgres — the transparent compression it applies to oversized values before moving them out of line — will take a large bite out of that, and object storage would make it cheaper still.

The multiplication is allowed because nothing amortises: storing the prompt on the message means every message carries its own copy of it, so the monthly volume is one prompt's size times the number of messages. Twelve kilobytes on each of three million messages is thirty-six million kilobytes, which is the 36 GB. The other side of the comparison is the striking one. The same text on a few hundred rows is a few megabytes however many messages point at it, because a pointer costs the width of a hash rather than the width of the thing it names.

The stronger objection is analytical rather than economic. A rendered prompt has the user's variables baked into it, so every message has a unique string and grouping by prompt becomes grouping by message. Separating the template from the bound variables is what makes the template a dimension you can partition on, and bound_variables is where the per-request part goes when it is small enough to keep. Where the bound variables are large — a retrieved document set, a long conversation summary — store a hash of them plus a pointer to the retrieval snapshot, and accept that reconstructing the exact rendered text needs the same replay path as everything else.

Version and time, confounded#

A version pointer also answers a narrower question than it appears to. It says which prompt text produced a message; it does not establish that the prompt caused a metric to move, and the reason is a property of how most teams deploy rather than of the schema. If exactly one prompt version is live at a time, then version and time are perfectly confounded — every message on version B is also a message from after the cutover, and anything else that changed at the cutover, or in the hours around it, is indistinguishable from the prompt. The pointer improves on the SHA because the other axes are now separately recorded, so you can check whether model_resolved or retrieval_snapshot also moved at that boundary. It does not become an experiment.

Getting causation needs the versions to overlap in time, which means ramping rather than cutting over: run version A and version B concurrently against randomly assigned traffic, record the assignment in flags, and the comparison stops depending on the clock.

Why randomisation dissolves the confound is one step, and it is worth having. Under a cutover, group membership is decided by when a message arrived, so every difference between the groups carries every other difference between those two stretches of time. Under a ramp, membership is decided by something with no relationship to the clock, and both arms therefore draw from the same hours, the same index build, the same alias and the same population of users. Anything that changes for everybody at once now changes both arms together and cancels out of the comparison. What is left differing systematically between them is the assignment, which is the thing being measured.

Ramping is a deployment practice the schema enables and does not provide. Recording the assignment at request time rather than looking it up later is the part that belongs here, because a flag service will happily tell you today's assignment for a user whose message was generated three weeks ago under a different one.

Two further limits are worth stating plainly. The pointers say nothing about the provider's own changes below the model string — serving-stack updates, quantisation, safety-filter tuning — which move behaviour under a pinned model id and leave every column in generations unchanged. And model_resolved is not a lineage record for the model itself; it names a product, not a set of weights you can inspect. Where those matter, the only instrument that works is a fixed evaluation set replayed on a schedule against a pinned model, so drift shows up as a movement in your own numbers rather than as a customer complaint.

The record a message needs, then, is a vector rather than a value: which prompt text, which model the provider actually served, which index build, which sampling parameters, which flag values, and — still useful, now demoted — which build of your service. Each entry is written at request time from what was resolved rather than from what the configuration said, and each points at something immutable so it can be read months later. That is the difference between an investigation that ends in a GROUP BY and one that ends in an argument about the deploy log.

All of this assumes each message has one generation and one history behind it. The moment a user hits regenerate, that assumption fails: one parent turn now has two assistant replies, both real, both attributable, and the schema so far has no way to say which one the conversation continued from.