Auditing a console-edited prompt before you freeze it

Every technique in this topic assumes a prompt that already has an identity: a file with a history, a revision with an id, a pointer with a record of who moved it. Plenty of running services have none of that. The production prompt is a text area on a vendor's web page, one field, current contents only, no versions, and several people hold the password. Getting from there to the arrangement this topic describes is a specific piece of work, and doing it in the obvious order destroys the evidence you need.

The obvious order is to revoke edit access this afternoon, copy the current text into the repository, and require a pull request from now on. That plan has the right destination and takes two wrong steps to get there. Both are expensive enough to be worth refusing, and both are easier to see with a concrete case in front of them: a ticket-classification service routing inbound support messages into eleven categories, handling about 9,000 messages a day, whose weekly spot-check shows macro-F1 — the score that averages each category's balance of precision and recall, so a rare category counts as much as a common one — sliding from 0.86 in mid-October to 0.79 by late November.

Freezing today's text freezes whatever is in the box#

The first problem is that nobody knows whether the text currently in the console is the good one. A quality slide covering six weeks and an unknown number of edits says nothing about which edit did the damage, and the current contents are simply the most recent of them. Copying that into version control and declaring it the baseline canonises the drifted version, and it does so in a way that is very hard to undo later, because from that moment the repository's history begins with a commit that says "import existing prompt" and contains no information about what came before. The audit trail you are creating starts by erasing the only evidence still in the system.

The sentence written at the end of that afternoon is the prompt is in version control now, so it is under control, and the first half of it is true. What the second half assumes is that a history beginning today is a history. A repository whose first commit reads import existing prompt records everything from now on and nothing at all before, which is adequate for a prompt nobody has changed and useless in the only situation anybody does this in — a service already behaving worse than it used to. The evidence for what changed sits in three places, and all three are draining. The console holds one value. The request logs hold the rest, behind a retention window that is running down. One person's memory holds the intent, and that runs down faster. Copying the box into a file touches none of them, while making everybody believe the problem is closed.

The second problem is the person who has been editing. Console editing is rarely vandalism; it is usually the mechanism by which a classifier's categories keep up with a product that ships new features monthly, and some of those edits are the reason the metric was ever at 0.86. Taking away the loop and offering a pull request in its place gives a non-engineer a workflow that requires a laptop setup they do not have and a review cycle measured in days for a change they used to make in ninety seconds. Workflows that are strictly worse for the person doing the work get routed around, and the routing is creative: a second console account, a "temporary" override in a config the platform team does not watch, a request to an engineer that turns into a rubber-stamped pull request. The version of this failure that gets discovered in March is worse than the one you have now, because by then everyone believes the prompt is under control.

Reconstructing a history nobody wrote down#

The audit's goal is a dated timeline of distinct prompt texts, and it is usually recoverable even when the console offers nothing, because the prompt was transmitted on every request.

Two things are worth doing before any of the archaeology. Turn on request payload logging today, if the vendor offers it and it is off, so that whatever happens next is recoverable and the next person inheriting the service has an easier afternoon. Then talk to whoever has been editing, early and as a source rather than a suspect, because they are the only living record of intent: which edits were experiments they abandoned, which coincided with a taxonomy change the product shipped, and roughly when they stopped paying attention. An audit framed as an investigation into a person gets you a defensive account and no dates; one framed as reconstructing a history the tooling failed to keep gets you their calendar and their chat history.

The first row does most of the work when it is available. If request logging captures the payload, the timeline is a group-by:

SELECT date_trunc('day', ts)          AS day,
       md5(system_prompt)             AS prompt_hash,
       count(*)                       AS calls,
       min(ts) AS first_seen, max(ts) AS last_seen
FROM   vendor_request_log
WHERE  endpoint = 'ticket-classifier'
  AND  ts > now() - interval '60 days'
GROUP  BY 1, 2
ORDER  BY 1;

 day        | prompt_hash | calls | first_seen           | last_seen
 2024-10-07 | 1a4f...     |  8912 | 2024-10-07T00:00:11Z | 2024-10-07T23:58:40Z
 2024-10-11 | 1a4f...     |  9104 | ...
 2024-10-11 | 7c02...     |  1877 | 2024-10-11T15:41:03Z | 2024-10-11T23:59:12Z
 2024-10-14 | 7c02...     |  8801 | ...
 2024-10-14 | b93d...     |  2210 | 2024-10-14T14:02:55Z | 2024-10-14T23:57:02Z
 2024-11-02 | e550...     |  6033 | 2024-11-02T09:19:44Z | ...

Reading that output is a two-column job. Grouping by day and by the hash of the prompt yields one row per distinct text seen on each day, so a day carrying two rows is a day the text changed, and the first_seen on the second row is when. The hash is doing equality here and nothing else — it is shorter than the prompt and it compares in one operation — and it is emphatically not protecting anything, since the prompts are yours and identifying them is the entire object of the query. A day with one row is a day the text held still, and a hash that reappears after an absence is somebody putting an old version back, which is a fact worth having and one the console would never have told you.

Four distinct texts, and the days on which each first appeared, which is a change history reconstructed from traffic. Where payload logging is off, the second row gives a coarser version of the same thing: for a fixed-shape classification request the input token count is nearly constant, so a step in the daily mean from 741 to 812 tokens on 14 October localises an edit to that day without recovering a single word of it. That is enough to correlate with the quality series and enough to ask what somebody remembers about the fourteenth.

That inference has three links, and it is worth checking each against your own service before leaning on it. The input to a request is the prompt plus one ticket, so the input token count is the prompt's length plus the ticket's length. Individual ticket lengths vary enormously and their daily mean does not, because thousands of them are averaged and the population writing them changes slowly. So the prompt's length is the only term in that sum capable of moving the daily mean by seventy tokens overnight, and a step in the mean is a step in the prompt. The size of the step is informative too — about seventy tokens is a couple of sentences rather than a rewrite — which is worth knowing before asking anybody what they remember. The link to check is the middle one. A service whose input mix shifts on the same day, because a new ticket source came online or a form field began truncating, breaks the inference completely, and that is exactly why the fifth row of the table belongs in the audit.

The fourth and fifth rows exist because unattributed console edits do more than defeat rollback; they make it impossible to attribute the drift at all. During six weeks the provider may have moved the alias you call to a newer snapshot, the category taxonomy may have gained a class, and an upstream ticket form may have changed its free-text field length. A 0.07 drop in macro-F1 has several candidate causes, and the audit is only finished when each candidate has a date attached and can be lined up against the metric. Finding that the quality step is on 14 October and the only dated event on 14 October is a prompt hash change is what turns a suspicion into a finding.

Deciding which recovered text was the good one#

With four candidate texts in hand, the question of which to freeze is answerable by measurement rather than by seniority. A support analyst's spot-checks are a labelled set already; extend them into something worth running — 1,200 tickets with agreed labels is a weekend of work for two people — and run every recovered revision against it:

$ promptctl bench --set labelled/tickets_1200.jsonl \
    --candidates 1a4f7e02 7c02b915 b93d40cc e550a731

  revision   first seen    macro-F1   worst class            cost
  1a4f7e02   2024-10-07      0.861    refund_request 0.72    $0.38
  7c02b915   2024-10-11      0.858    refund_request 0.71    $0.38
  b93d40cc   2024-10-14      0.812    billing_dispute 0.44   $0.41
  e550a731   2024-11-02      0.793    billing_dispute 0.39   $0.41
  84 s wall, 4 revisions x 1200 examples, $1.58 total

Three readings of that table matter, and only one of them is the headline. The first is where the step is: the earliest two revisions score within a few thousandths of each other and the third sits nearly five hundredths below, so the damage is one edit rather than a slide. The second is where it landed. Macro-F1 averages the classes, so a large fall in the average across eleven classes has to come from a large fall in one or two of them, and the worst-class column names which — one category collapsing while the rest hold. The third is what the final revision added, which is very little: it scores a shade worse again and inherits the whole of the earlier loss, which is what editing on top of a broken clause looks like when the person editing cannot see the clause.

The drop is at b93d40cc, it is concentrated in one class, and the two later texts inherited it. Diffing 7c02b915 against b93d40cc — now possible, because both are recovered artefacts — showed that a clause distinguishing billing disputes from refund requests had been shortened while adding two new categories, which is a change nobody would have defended if anyone had read it.

The audit can also come back empty, and knowing what to do then is part of running it. If every recovered text scores within noise of the others on the labelled set, the prompt is not the cause, and the timeline you built is still the instrument that finds the real one: line the quality series up against the dated events from the other rows of the table, and look for a model alias change, an index rebuild, or a shift in the input distribution such as a new ticket source or a form field that started truncating. The third of those is the most common and the least suspected, because it presents exactly like a model regression while being a change in what the model is asked about. Freezing the prompt in that scenario is still worth doing for every reason in this topic, and it will not move the metric, and saying so before you do it keeps the credibility you will need next week.

Give the loop back before you take the console away#

What Mira gets in exchange has to be faster than what she lost, which is achievable because her old loop was not actually fast. Editing the console and waiting for the weekly spot-check meant a feedback cycle of up to seven days, with production traffic as the test set. The replacement is a draft that runs against the labelled set on demand:

$ promptctl draft edit ticket_classifier.system      # opens the current revision
$ promptctl draft bench --set labelled/tickets_1200.jsonl

  baseline  7c02b915   macro-F1 0.858
  draft     (local)    macro-F1 0.871   (+0.013)
    billing_dispute  0.71 -> 0.79
    refund_request   0.71 -> 0.70
    9 examples changed prediction; see drafts/diff-2024-11-21.html
  71 s wall, $0.39

$ promptctl draft propose --title "Sharpen billing vs refund distinction"
  opened PR #2341 with bench results attached; reviewers: @ai-platform @support-ops

Ninety seconds to an answer, against seven days, with per-class numbers she never had. The pull request is opened by the tool with the benchmark output already in the description, so the review is a review of a change with evidence attached, and the reviewer's job under this topic's one-decision-per-revision rule is to check that the diff contains one decision. Nothing about this asks Mira to learn Git; it asks her to run two commands, and the artefact that reaches production carries approval and an immutable id.

The one thing to be careful about is the labelled set becoming the target. A drafting loop that scores every idea against 1,200 examples will, over months, produce prompts tuned to those 1,200 examples, so the set needs refreshing from recent traffic on a cadence and a held-out slice that drafts are not benchmarked against. That is the same discipline any evaluation suite needs and it does not go away because the author is a product manager.

The mechanism behind that warning deserves a sentence more than it usually gets, because from inside the loop nothing looks wrong. Each iteration keeps the variant that scored best on the set, and a score on a finite sample is the true quality plus whatever that particular sample happens to reward. Selecting repeatedly against the same sample climbs both terms, and the second one does not transfer to traffic. The number keeps rising the whole time, which is why nobody notices. A held-out slice is what makes the split visible: a draft that improves on the tuned set and not on the held-out one has learned the sample rather than the job, and that is a finding to act on rather than a nuisance.

The four questions to ask of a service you inherit#

The audit above generalises into something you can run against any service in an afternoon, and it is a better first move than reading its prompts. Ask what the serving path treats as authoritative at request time, and follow the answer to an actual store rather than to a diagram. Ask who can write to that store and whether a write requires anyone's approval. Ask what identifier production logs alongside each response, and whether that identifier still resolves to the exact bytes that served. Ask how long it takes to put back the previous known-good text, and time it rather than estimating.

Most services answer one or two of those well. A service that answers all four has the properties this topic has been building toward, and the gaps in the other cases are specific enough to be turned into work: a bundled fallback that does not exist, a pointer nobody validates, a log line carrying an alias instead of a snapshot, a rollback that is a deploy. Start with the one whose absence would hurt most at 02:10 on a Sunday.

The order those four questions are asked in is the order this chapter argues for, and it is the reverse of the order a governance instinct suggests. Access control is the last step rather than the first, because revoking writes before you know which text is good freezes a drifted version with a signature on it, and because the person you are revoking is usually the only source for what the missing history meant. Reconstruct, measure, choose, publish, and only then take the console away.