Promoting a trace into a change

An escalated trace arrives with a strong implicit argument attached to it — a real customer was given a wrong answer, the cause has been localised, and the fix is one line in a file somebody owns. What that argument leaves out is that the line goes into a system prompt, which is present on every request the service handles, and that the evidence for it is one input out of a weekly population nobody sampled. Those two omissions point in opposite directions. The change is wide and the evidence is narrow, and neither of those facts is visible in the diff a reviewer approves.

The gate between them turns on a single measurement. The class rate is the failure rate over inputs constructed to satisfy the same predicate as the failing one — not repeated runs of the same input, which is a different quantity, but different inputs sharing the property the failure is being blamed on. A tail sample and a trace standing for an entire input class look identical until that number exists. The mirror-image habit, that one trace may never justify a change to anything shared, is worse than the habit it corrects, and two situations settle the question on a single trace.

Why a verified fix regresses everything else#

Start from what a system prompt is at the moment a request is served. The instructions are numbered and they read like a list of rules, but nothing downstream treats them as a list. They are concatenated into one string, that string is placed ahead of the retrieved blocks and the user turn, and the model conditions its continuation on the whole of it at once. No dispatcher selects the applicable rule for this input and holds the others back, because nothing in the request marks which inputs a given line was written for. Every line participates in every request, and the only scoping an instruction has is whatever conditions its own English states — conditions the model applies probabilistically, the way it does everything else.

The verification run is therefore close to a tautology. The instruction was authored while looking at that specific input, by a person who kept editing until that input produced the right answer. Testing it on the same input asks whether the author succeeded at the thing they were doing, and the answer is yes by construction. It is fitting on a single point and then reporting training error, and it carries no information about any other point.

The belief that gets teams here is that a fix verified against the trace which prompted it is verified, and three things make it tempting. The run flipped: the same request that produced the wrong answer now produces the right one, which is the strongest evidence anyone produces about anything in this work. The change is small, one sentence added to a dozen, and small edits feel like they have small blast radii. And the reasoning is legible — a reviewer can read the instruction, read the trace, and see exactly why the one addresses the other. What the belief assumes without saying so is that the instruction's effect is confined to inputs resembling the one it was written for, which is the author's own scoping being attributed to machinery that has none. The line shifts the distribution over continuations for every conversation the service handles that week, including all the ones nobody read.

A system prompt is also global state, in the sense the phrase carries in ordinary code. The retrieved blocks change per request and the user turn changes per request, but an instruction added on Monday is present on all 40,000 conversations a week. Adding a line to it is a deployment that touches every code path in the system, evaluated on one of them. A change with that reach would normally attract a regression suite; this one attracts a single re-run, because the re-run looks like a test and the edit looks like a wording change.

Then the lines interact, in the way instruction bisection made concrete (chapter 6, where two individually harmless instructions jointly produced a refusal nobody had written). Here is what that costs, measured: three instructions added to one support assistant's prompt over a single week, each verified against the trace that motivated it, run afterwards one at a time against the full 420-conversation evaluation set.

Instruction 12 tells the model to withhold claims that the retrieved documentation does not support, and instruction 14 tells it to ask rather than act when things are unclear. Individually each is mild. Together they produce a model that responds to any question about a feature it cannot verify by asking a clarifying question, which is why the abstention and the extra-turn effects are larger in combination than the rows suggest when added naively. Instruction 13's effect runs through a completely different channel: longer answers against an unchanged max_tokens cap, which turns a quality instruction into a truncation problem.

The delta column is worth adding up, and the total says less than it appears to. One and four fifths plus nine tenths is two and seven tenths; adding one and a half gives four and a fifth points, which is exactly the drop from 87.4 to 83.2 that the release recorded. Taken at face value that says the three changes composed additively. Taken carefully it says almost nothing. A run with one instruction added cannot exhibit an effect that requires two instructions to be present, so the pair term is missing from every row by construction. The only run in which all three were present at once was the release itself. The additivity is an observation about this set on this occasion rather than a property of prompts, and the joint behaviour described above shows up in which conversations moved rather than in the total.

Four points is a small-looking number attached to a large one, and converting it into conversations is what makes it legible. The score is an average over the 420 labelled conversations, so a point of it is one per cent of the set, and one per cent of four hundred and twenty conversations is four and a fifth of them. The drop is itself four and a fifth per cent of the score, and those two multiply to a little under eighteen conversations' worth. That becomes eighteen whole conversations which used to be right and are now wrong if the rubric marks a conversation right or wrong. If it marks degrees instead, the same loss spreads as partial marks across a larger number of them. Either reading sits far outside what the set does on its own: run-to-run noise on a rubric-scored set this size is often around half a point, and four and a fifth points is more than eight times that.

The last part of the mechanism is about which inputs you ever look at. Twelve traces reach an escalation channel of this kind in a week, out of 40,000 conversations, which is 0.03% of traffic. Twelve in forty thousand is three in ten thousand, or three hundredths of one per cent, which is also one complaint for every three thousand-odd conversations the service handled. That sample is not random. It over-represents customers who are angry enough to write in, accounts with a named contact who has somewhere to complain, failures that are visible to a user rather than merely wrong, and English-language conversations during business hours. Optimising a global prompt against that channel, week after week, moves the system toward that slice and away from the median conversation — and because the median conversation has nobody to complain on its behalf, the movement is invisible until an evaluation run.

The rule for promoting a trace into a change#

The rule that prevents the four-point drop is a gate with a small number of conditions, most of which are already satisfied by the work in the earlier chapters of this topic.

Reproduction and localisation come from the replay, ablation and bisection chapters, and they are gates rather than steps: a trace that will not reproduce under a pinned replay does not license a prompt change, it licenses a fix to whatever makes replay impossible. The genuinely new condition is the class rate.

Characterising the class means writing down, in one sentence, the predicate over inputs that this failure is claimed to hold for. "Questions about features that exist in the changelog but not in the indexed docs." "Refund questions where the documents contain a promotional page and a policy page with different windows." "Tool-calling turns where the user gives an order id and an ambiguous verb." Then you construct or retrieve fifteen more inputs satisfying that predicate and measure the failure rate on them.

What makes fifteen inputs siblings is one property they share and a long list they do not. Every one of them has to satisfy the predicate, since an input that fails it is measuring some other class. Everything the original trace happened to carry alongside the predicate has to vary across the set: a different customer phrasing the question a different way, a different account with a different history, a different set of documents coming back from the retriever in a different order, different lengths, different times of day, different languages where the service takes them. The variation is what turns one trace into a measurement, and the reason it is required is the reason the original verification run was worthless. Fifteen near-copies of the failing input run the same tautology fifteen times at fifteen times the cost, and the number they produce is a fact about one input wearing the clothes of a fact about a class. Retrieving siblings from logs is fine when the log is production traffic and misleading when it is the escalation queue, which is biased in precisely the direction under test.

siblings = find_or_construct(predicate="feature question, changelog-only feature", n=15)
rates = {s.id: measure_rate(s, n=30) for s in siblings}   # per-input rate machinery
print(sum(1 for r in rates.values() if r > 0.05), "of 15 fail above 5%")

Two different rates live in those three lines, and conflating them is the commonest way this measurement goes wrong. measure_rate replays one input thirty times at production sampling settings and reports how often that single input fails, which is the per-input rate from the replay chapter (chapter 4, where thirty replays with their seeds recorded turn one bad output into a frequency with an interval around it). The class rate is the level above: the share of the fifteen inputs whose per-input rate is high enough to count as failing. A class can be badly broken in either of two shapes, and the class rate alone does not tell them apart. Every sibling might fail some of the time, or a few siblings might fail all of the time, and only the per-input rates behind the count distinguish those — which is why the dictionary is worth keeping rather than only its summary.

The five-per-cent comparison in the last line is doing more work than it looks. With thirty replays per sibling, one failure is a little over three per cent and does not clear the bar, while two failures are nearly seven per cent and do. The predicate is in effect "at least two failures in thirty", and it sits there because a single failure in thirty runs leaves an interval wide enough to be compatible with a rate nobody would act on.

The number that comes back decides everything downstream. Fourteen of fifteen — over nine in ten — means the escalated trace was the first observation of a defect that affects an entire input class, the population impact is far larger than one complaint, and a change is clearly warranted. One of fifteen, under one in ten, means the trace was a tail sample of a system that is mostly fine on that class, and a global instruction is an expensive response to a rare event. That second reading needs its condition attached, because fifteen is a small sample and one failure in fifteen leaves a wide interval around the true class rate. What the measurement establishes is that the class rate is not high, not that it is low. The decision it licenses is to keep global state out of the prompt today, and to log the trace into the failure taxonomy so that a class quietly failing one time in fifteen accumulates evidence rather than evaporating.

The step from a class rate to an admissible change runs through two quantities rather than one. On the harm side, the class rate multiplied by the share of traffic satisfying the predicate estimates how many conversations a week the defect is spoiling. That is the quantity the complaint count stands in for badly: one complaint arrived out of a class whose size nobody had counted. On the cost side sits the effect of the fix on everything outside the class, which for a prompt instruction is unmeasured by definition, since the instruction reaches every conversation while the evidence so far covers one predicate. A change is admissible when the first quantity is large, the second has been measured rather than assumed, and the measurement of it comes back inside the noise band. A high class rate on its own licenses the investigation and not the deployment.

The rest of the gate follows from ordinary engineering discipline. Verify the fix on siblings that were held out while the fix was being written, because the ones you looked at while writing it are contaminated in the same way the original trace was. Run the full evaluation set before and after, and decide the threshold in advance — a class rate that drops below some target while the aggregate stays within the noise band of the set, which for a 420-item rubric-scored set is often around half a point. Record the trace id and an owner next to the instruction, and a date to revisit it, because prompts accumulate instructions the way legacy code accumulates flags and nobody ever deletes one whose provenance is unknown.

One condition is easy to skip and does more good than the rest combined: prefer a fix that is not a prompt instruction, wherever one exists. The duplicated side effect of the replay chapter (chapter 4, the same order id refunded twice inside one turn) wants an idempotency key. The invented refund window of the coverage chapter (chapter 3, where the figure in the answer appeared in none of the retrieved passages) wants an ingestion fix and a validator that rejects answers containing a policy number absent from the retrieved blocks. A schema constraint, an allowlist, a retrieval filter scoped to one query type — all of these have a blast radius you can state, and a sentence in a system prompt does not.

Adding the trace to the golden set is not the gate#

The standard response to this whole problem is to add every escalated trace to the golden set and keep shipping. That is necessary and nowhere near sufficient, and the reasons are worth being precise about.

A golden set is a fixed list of inputs kept with their expected behaviour and re-run at each release, and the question it answers is one per case: does this input still behave. It is silent about every input it does not contain, and an edit to a shared prompt reaches all of them. Run against the release described above, a golden set holding traces 4471, 4602 and 4711 comes back green on all three while the 420-conversation set drops four points. Passing the case that prompted a change is the one outcome that change cannot fail to produce.

Traces fixed this way become test cases that the prompt is guaranteed to pass, because the prompt was written against them. They stay green through every future release, contributing nothing, while looking like coverage. Meanwhile the set grows in exactly the direction the escalation channel is biased toward, so after a year the golden set is a portrait of the complaint queue rather than of the traffic. And none of it addresses the actual failure: two instructions that interact damage conversations nobody has ever complained about, which therefore have no representative in the set. The blind spot the verification run had is now written down and inherited by every release that follows.

What repairs it is holding out siblings rather than the trace itself, and periodically comparing the composition of the evaluation set against the composition of production traffic — by language, by query type, by whether retrieval returned anything, by conversation length. Where the set is thin relative to traffic, sample from traffic and label, which is the aggregate work of topic 3.6 — coding a sample of production conversations and projecting the counts through a base rate into a share of traffic — pointed at one specific gap.

When one trace is enough on its own#

The mirror-image position, that a single trace may never change the prompt, is worse than the habit it corrects. Two situations make chasing one trace exactly the right call.

The first is severity. Where a single occurrence is unacceptable — money moved without authorisation, personal data disclosed to the wrong account, a regulated claim made in writing — the rate is not the deciding variable and waiting for a second instance is not a research strategy. Notice, though, that severity argues for a hard control rather than for an instruction. If one occurrence is genuinely unacceptable, a probabilistic mitigation with an unmeasured effect size is not a response to it; a deterministic check in code is. That substitution has a precondition worth stating out loud, because it is not always available: a control can only be written where the unacceptable thing is expressible as a property of the request or the response, such as a repeated order id, a figure absent from the retrieved blocks, or a recipient outside the account. Where the unacceptable thing is a matter of judgement, no validator exists to write, and severity has bought an investigation and a human in the loop rather than a control.

The second is the trace that is a class representative. Some failures are one-in-one on an entire slice of input: every customer whose surname contains an apostrophe, because a name is interpolated into the prompt through a formatter that escapes it into an HTML entity, so that O'Brien reaches the model as O'Brien; every order with a null shipping address, because the block renders as the literal string "None" and the model reads it as a place; every conversation in Portuguese, because the language-detection step routes to a system prompt that was never translated. These arrive through an escalation channel as one complaint, and they are 100% failures over thousands of daily requests that nobody reports because the customers simply leave. The rate is total in those three cases because each mechanism is deterministic and unconditional: the formatter escapes every apostrophe, the renderer prints the same literal for every null address, the router sends every Portuguese conversation to the same untranslated prompt. Where the mechanism has a branch in it — a formatter that escapes on one code path only, a detector with a confidence threshold — the slice fails at some rate instead of always, and it needs the same fifteen siblings as anything else.

That condition also supplies the cheap version of the measurement. When a class representative is suspected, the class rate can sometimes be established by reading code rather than by sampling: find the transform, confirm it has no branch, and the rate over the slice is one by inspection rather than by replay. Constructing a few siblings is still worth the time, because the code cannot tell you how much traffic satisfies the predicate, and a deterministic failure over a slice almost nobody occupies is a smaller problem than an occasional failure over a large one.

The class rate is what decides which of the two you are holding, and it is the reason that step is in the gate rather than being left to judgement. A tail sample gives you one failure in fifteen siblings; a class representative gives you fifteen in fifteen. Both started as one screenshot in a support channel, and until the siblings exist they look identical — same length, same tone, one customer each, and the same conviction from whoever escalated it.

The sequence this topic has built is short enough to work from directly. Freeze the request as it left your service, with a known-good trace of the same shape. Read the envelope — finish reason, token counts, rendered wire format — before reading the prose. Check that the evidence was present, in the source documents, in the retrieval, and in the rendered prompt. Establish whether your stack reproduces the trace at all, and get a rate at production settings so you know what a single run is worth. Substitute context blocks one at a time with matched filler; bisect the instructions and expect a pair rather than a culprit. Then measure the class before changing anything global.

The honest limitation is that this procedure costs a few hours per trace, and no team can afford it for every complaint. What it is for is the trace that would otherwise send a week of prompt edits in the wrong direction, and the way you recognise that one in advance is imperfect: it tends to be the failure that contradicts something you believe about your own system. Those are worth the hours, and the twelfth instruction someone was about to add on Monday morning is worth the twenty minutes it takes to find out whether it is fixing anything.