Regression testing a prompt patch

Porting a prompt to a second model leaves a residue: a handful of cases that passed on the old model and fail on the new one. The obvious loop is to read them, write wording that fixes them, confirm they now pass, and ship. That loop produces a number that always improves and a system that sometimes does not, and the reason is structural rather than careless.

The instinct behind it is that a prompt is a collection of independent rules, so adding one that addresses a specific failure leaves the rest untouched. Prompts do not behave that way. An added sentence shifts the model's behaviour on every input, and on the 388 of a 400-case set that were already passing, most of that shift is invisible because the output happens to stay correct. Some of it is not invisible, and finding out which requires re-scoring cases you were not working on.

A patch is a change to the whole distribution#

The reason prompts do not behave that way has one step in it. A prompt is not a set of rules the model consults. It is tokens in the context, and every next-token distribution the model produces is conditioned on all of them at once. There is no scoping construct available — nothing in the format says "this sentence governs documents with no discount and no others" — so a sentence written for eleven cases is present, and conditioning the output, on all 400. Whether it changes the answer on any of the other 389 depends only on whether that document was near enough to a margin for the shift to tip it, which is a fact about the document rather than about what the sentence was for.

The belief being displaced here is not a silly one. Prompts look like rule lists: they are written in bullets, they read like configuration, and configuration composes — a line added to a config file does not change what the lines above it mean. A sentence added to a prompt does, because both are input to one function and the function has no notion of which sentence a token came from.

Consider a patch written to fix a missing field: "If the invoice does not state a discount, output "discount_pct": 0." It fixes the cases it was written for. It also gives the model a cheap default on every other document, and finding a discount stated in a footnote requires reading past the line-item table, so the sentence lowers the effort required to produce an answer that satisfies the instruction. The original prompt, which said nothing about defaults, left the model with no easy out and it went looking.

Effort is a picture, and what sits under it is not. Before the patch, a document whose line items mention no discount offers the model no output the prompt licenses except one it has to find support for, so every continuation that satisfies the instruction runs through carrying on reading. After the patch there is a short, specific, explicitly licensed continuation available immediately — the literal string the sentence names — and probability mass moves onto it. Nothing in the model decided to stop looking. A cheap valid answer was added to the set of valid answers, and a distribution redistributed.

That is the sense in which an original phrasing can be protecting cases: not deliberately, and nobody knows it is doing so until the protection is removed. The severity of the trade is the part a pass rate cannot express. A missing field is a loud failure that a validator rejects at the boundary, sending the document to a human. A defaulted zero is schema-valid, plausible, and lands in the database as a discount of nought on an invoice that had one.

Score the whole set, per error type, as a paired comparison#

The comparison that catches this has three properties. It covers the entire frozen set rather than the failures you were working on. It is paired, meaning the same case is scored under both prompts so you can see individual flips rather than only aggregates. And it is broken out by error type, because a headline number sums categories that are not interchangeable.

def paired_compare(cases, run_a, run_b):
    """run_a / run_b: {case_id: {"pass": bool, "error_type": str | None}}"""
    flips = {"fixed": [], "broken": [], "still_failing": [], "unchanged_pass": []}
    by_type = Counter(), Counter()          # (before, after)
    for c in cases:
        a, b = run_a[c["id"]], run_b[c["id"]]
        if not a["pass"]:
            by_type[0][a["error_type"]] += 1
        if not b["pass"]:
            by_type[1][b["error_type"]] += 1
        key = ("fixed" if b["pass"] else "still_failing") if not a["pass"] \
              else ("unchanged_pass" if b["pass"] else "broken")
        flips[key].append(c["id"])
    return flips, by_type

The two things it returns answer different questions. flips tells you what your patch did to individual cases, and its broken list is the one nobody produces by accident. by_type tells you whether the composition of the remaining failures got better or worse, which is where severity lives.

Twelve to three is a 75% reduction and it is the wrong summary. The patch converted twelve failures that the system already handled safely into three failures the system cannot detect, and the correct decision on that table is to keep the enum half of the patch, drop the default-to-zero half, and handle a missing discount_pct in code by treating absence as unknown rather than as zero.

Running the comparison also gives you a check against fooling yourself with noise. With 12 fixed and 3 broken, a McNemar exact test — which ignores the cases both prompts got right and asks only whether the fixes outnumber the breakages by more than chance would produce — gives p ≈ 0.035, so the direction of the change is probably real. Had the split been 12 fixed and 9 broken, p ≈ 0.66 and the honest statement would be that the patch made no measurable difference to the pass rate while shifting failures from a loud category to a silent one — which is a decision you would then be making on severity alone, and could make correctly, as long as you knew that was what you were doing.

That test earns its place by throwing most of the set away. It ignores every case both prompts got right and every case both got wrong, because neither carries information about which prompt is better — those cases would look the same whichever way the patch had gone. What remains is the discordant cases: fifteen of them here, twelve in the patch's favour and three against. If the patch had no real effect, each discordant case would fall one way or the other on something close to a coin flip, and twelve out of fifteen landing on the same side is not what coins usually do. That is the whole of what the small p is claiming, and it is a claim about direction and nothing else. Change the split to twelve fixed and nine broken and there are twenty-one discordant cases with twelve on one side, which is what coins do constantly — hence a p near two-thirds, and no statement about direction available at all.

One patch per failure shape, and each in the cheapest place it can live#

Failures cluster by shape, and a patch addressing one cluster is a separate change from a patch addressing another: they have different evidence behind them, different blast radii, and they can be accepted and rejected independently. Landing two clusters in a single diff means the good half has to be recovered by argument after the re-score, when splitting them would have cost a second commit.

The second discipline is choosing where the fix lives before choosing what it says. Wording is the most visible option and the least reliable, and three of the four alternatives are cheaper to verify.

A missing discount_pct sits in the last row. Absence is decidable, so treating it in code — None for unknown, distinguished from 0 for a stated zero discount — fixes eleven cases with a test rather than with a sentence, leaves the model's behaviour on the other 389 documents untouched, and preserves the distinction the prompt patch destroyed. The rule of thumb that falls out of the table is to try the lowest row that can express the fix, because the guarantee gets stronger and the surface it perturbs gets smaller as you descend.

Both halves of that follow from what the rows are. The top two change an input to a probabilistic function, so all they can do is move a distribution, and they move it over every case that function sees. The bottom two sit outside the model: a decoding constraint removes outputs from the sampler's reach, and post-processing is a decidable function of a response that has already been produced. Neither can be tipped by a document nobody thought about while writing it. That is the entire ordering — descending the table trades a control whose effect has to be measured for one whose effect can be stated.

The two middle rows are where most real patches end up, and the choice between them is the one people get wrong in a hurry. A failure that reproduces on every model is a gap in the task description and belongs in the shared core, where it costs a re-score on every model and buys a fix everywhere. A failure that reproduces on one family is a divergence and belongs in that family's adapter, where it costs one re-score and cannot silently change what the other models see. Putting a per-family patch in the core is the cheaper edit and the more expensive mistake, because the wording that fixes one model is arbitrary text to the others and its effect on them is unmeasured until somebody re-runs their sets.

Whatever survives that choice gets committed with the case identifiers it was written for. A prompt line whose commit message reads "fixes INV-0117, INV-0233, INV-0244 — currency emitted as a word" can be re-tested in isolation two model versions later, and dropped when the divergence that motivated it no longer reproduces. A prompt line with no such attribution is permanent by default, because nobody who inherits the file can tell whether removing it is safe.

The set wears out#

There is a second failure mode in this loop that a single paired comparison will not catch. Each round of patch-and-re-score uses the same cases to select wording, and after five or six rounds the prompt has been fitted to that specific set in the same way a model overfits to training data. The pass rate keeps climbing and the number stops predicting production, because the remaining headroom on the set is idiosyncrasies of those documents rather than properties of invoices.

The defence is a slice you do not look at. Split the frozen set once, before any patching, into a development portion you iterate against and a held-out portion you score exactly once per candidate prompt — not once per iteration, and never as a diagnostic. On a 400-case set, 280 development and 120 held-out is a workable split, keeping the class balance the same in both. The split is the ratio applied to the set — seven tenths of 400 is 280, and what is left over is 120 — and keeping the class balance is the part that matters more than the proportion. The interesting categories are small; one bad enum value existed in the whole set. A random split has a real chance of putting every instance of a rare class on one side, and a held-out slice containing none of a category cannot report on it, while a development slice containing all of it will cheerfully be patched until it is gone. The held-out slice is the number that goes in the migration decision; the development slice is the number that guides the work.

dev, held_out = stratified_split(FROZEN_400, ratio=0.7, key="error_class", seed=11)
# dev is scored freely during patching.
# held_out is scored once per candidate, and the count of scorings is logged.
assert holdout_scorings[candidate_id] <= 1, "held-out slice re-used; results are now dev results"

That assertion is not decoration. The way a held-out slice dies is that someone scores it, sees a failure, fixes the failure, and scores it again — at which point it is a development set with an official-sounding name.

What that costs is information rather than honesty, and the mechanism is worth stating precisely. Every time a set is scored and something is changed in response, the change is selected partly for its real effect and partly for the accidents of those particular documents — one supplier's odd date format, one scan with a smudge across the total. The real effect carries over to documents nobody has seen. The accidents do not. Repeat the loop enough times and the score keeps climbing on the strength of the accidents alone, which is the same thing a model does when it overfits, and it happens to a prompt for the same reason: the selection is being made against one sample, over and over. A slice scored once has been selected against zero times, and that is the only condition under which its number means what it appears to mean.

When the held-out slice is used up, replace it rather than reusing it: sampling 120 fresh documents from recent traffic and labelling them costs a day of somebody's time, which is cheap relative to the cost of not knowing whether the port is safe.

Frozen sets and the traffic they do not contain#

Per-error-type scoring on a frozen set is not a claim about production. The set has the composition it had on the day it was frozen, and the traffic moves: a supplier who starts sending credit notes in a new format is a category the set does not contain in any proportion. The comparison tells you what your patch did to the distribution you measured, and it is silent about the rest, which is the reason a standing production comparison exists as a separate thing at the end of this topic rather than as a re-run of this one.

And it is not an argument against patching. The alternative — porting without adjusting the prompt — is what the triage team in the first chapter of this topic did, and it produced a refusal regression nobody caught for nine days. Patching per model is correct and necessary. What comes with it is that every patch is a change to behaviour on the whole distribution, and the only way to know its sign is to measure the whole distribution before and after, with the failures kept separate by kind.

The output of that measurement is a quality delta with error classes attached: model B, patched, at 3 failures per 400 with all three silent, or at 11 per 400 with all of them loud if the default-to-zero half is dropped. Neither number is a decision on its own, because the migration was proposed for a price. What that delta is worth depends on what the repairs, retries and extra round trips cost against the saving that started the exercise, and that arithmetic is worth fixing in writing before the review meeting rather than during it.