Restructuring a prompt

Restructuring a prompt means moving sentences without rewriting them: prose becomes named sections, sections get reordered, the payload moves to the end. The verification people reach for is a diff of the wording, and it is easy to make that diff provably empty — split both versions into sentences, normalise whitespace, and assert that the two multisets are identical.

That assertion is true and it establishes nothing. The intuition that a layout change cannot alter behaviour comes from code, where it holds — a compiler reads a function the same way whether it sits above or below its caller. A prompt has no compiler. It is a token sequence consumed by a system whose output depends on the whole sequence, including the order, so an empty wording diff tells you nothing about an empty diff of effects.

Three things a reordering does#

Three effects show up, and all three are general.

Reordering moves which instructions sit near the end. Take a contract-clause classifier whose prose prompt ended with a sentence somebody had appended in April: "When a clause could reasonably fall into two categories, prefer the broader one." In the restructure it went into <edge_cases>, roughly two-thirds of the way through. On the 340 clauses in the holdout set that the legal team had marked ambiguous, the classifier chose the broader category 84% of the time before and 61% after. That single shift accounted for most of the drop. Position is not decoration; content near the end of a long prompt is attended to differently from content in the middle, and a restructure is an operation that changes position for almost every sentence in the file.

Convert those two rates into clauses and the size of the effect stops being abstract. The ambiguous holdout is three hundred and forty clauses. Eighty-four percent of it is a bit over two hundred and eighty-five; sixty-one percent is a bit over two hundred and seven. So roughly seventy-eight clauses came back with a different category than they had before, on a change that reworded nothing. Set that against the whole-holdout figures in the scenario below: twelve hundred clauses, agreement falling from 91.3% to 87.8%, which is about forty-two clauses moving from agreeing with the legal team to disagreeing. Around half of the switches landed on the wrong side of a label and the rest happened to land on the right side, and that is what "accounted for most of the drop" means when it is written out. It also shows why the aggregate is such a poor instrument here. Seventy-eight clauses changed behaviour and the headline number moved by three and a half points.

Be honest about the mechanism underneath, because it is the one place in this topic where the honest answer is partly empirical. That later positions are weighted differently from middle ones is a measured property of these models rather than something derivable from the architecture, it varies by model and by prompt length, and a team that treats it as a law will be wrong about the direction on some model at some point. What is safe to carry is the weaker statement, and it is enough for the engineering: moving a sentence changes what it is worth, the size and sign of the change are not predictable from the file, and therefore position is one of the things a paired run exists to measure rather than one of the things you reason about in review.

Reordering changes the token prefix that is being cached, and here the interesting part is what the investigation found rather than what the team expected. The old prompt interpolated the clause text about a third of the way in, which meant the static prefix ahead of the payload was around 900 tokens. There was a cache breakpoint on it and there had been for months. There had also never been a single cache read, because the minimum cacheable prefix on the model they were using is 1,024 tokens and a shorter prefix silently does not cache — no error, no warning, cache_creation_input_tokens reporting zero on every response. The restructure moved the payload to the end, taking the static prefix to about 2,850 tokens, which crossed the threshold. For a few hours the service paid the 1.25× write premium on every request while the old and new prompts alternated across a staged rollout, and then it settled into cache reads at roughly a tenth of base input price. The cost spike that alarmed the on-call engineer was the transition; the steady state after it was cheaper than before.

Two silences in that story are worth explaining rather than noting. The first is why a prefix below the minimum caches nothing and says nothing. A minimum exists because a cache entry is not free to keep — the provider stores intermediate state, indexes it and holds it for a window — and below some size that bookkeeping costs more than recomputing the prefix would. So the provider's sensible behaviour on a short prefix is to skip the cache, and skipping an optimisation is not an error condition. Nothing failed. A request that could have been cheaper was merely not cheaper, which is the class of defect that survives longest, because the only symptom is a number that never appears.

The second is why a staged rollout costs more than either prompt does on its own. A cache read requires the current request's prefix to match bytes that are already stored. During the rollout two different prompts are in flight, so consecutive requests carry two different prefixes, and each one is a miss followed by a write. Compare the multipliers the previous chapters used: a read is charged at about a tenth of base and a write at a quarter above it, so the transition is not paying the ordinary price, it is paying more than twelve times the price it would settle at. Doubling the daily spend for a few hours is exactly the shape that produces. Knowing this in advance turns the on-call engineer's alarm into a prediction, which is the difference between an incident and a rollout plan.

Reordering promotes an aside into a rule. In the old prompt one sentence read: "We generally don't flag boilerplate indemnity clauses, since counsel reviews those separately." Mid-paragraph, hedged with "generally", trailing a justification — it read as background. In the restructure it became the entire content of <exclusions>, alone under a heading. Flagging on indemnity clauses fell from 47% to 12%. Nobody decided to make that rule stronger. Giving a sentence its own heading is a claim about its importance, and the model reads that claim.

That last sentence is short enough to slide past, and it is the one carrying the mechanism. A heading is not metadata; the earlier chapters established that there is no metadata. It is more tokens in the same sequence, and what it does is what any tokens do — it changes what the sequence resembles. Text that appears alone under its own heading, in the enormous quantity of documents these models were trained on, is overwhelmingly text that governs: policy clauses, specification requirements, rules in a manual. Text hedged with "generally" in the middle of a paragraph, trailing a justification, is overwhelmingly commentary. So moving the sentence did not add emphasis in the way a capitalised word adds emphasis. It moved the sentence from one distribution to another, and the model's reading followed.

Read the two rates as a proportion and the size of that becomes clear. Flagging fell by thirty-five points from a starting point of forty-seven, which is roughly three quarters of all the flagging that used to happen. A word like "generally" was doing three quarters of the work of that rule, and nobody who approved the restructure was aware that it had a strength for the layout to change.

None of the three is visible in a wording diff, which is why the wording diff being empty is mistaken for evidence.

The paired run, and what it has to enumerate#

"We restructured it and the evals passed" is not evidence of neutrality, because most eval suites report an aggregate and the aggregate hides exactly the kind of movement described above — 11 improvements and 3 regressions net out to something that looks like noise. What establishes neutrality is a paired run with the differences enumerated.

Put the arithmetic under that, because "looks like noise" is the sort of phrase a reader accepts and cannot reproduce. Eleven cases got better and three got worse, so the net is eight cases. On a run of five hundred, eight cases is a little over one and a half percentage points of aggregate agreement — and in the wrong direction for the person reading the dashboard, since the aggregate went up. A reviewer looking at that number sees a restructure that was neutral or mildly beneficial. What actually happened is that fourteen cases changed behaviour, three of them into disagreement with the legal team, all on one rule, on a change whose wording diff was empty. The aggregate did not hide the regression by being imprecise. It hid it by adding a positive number to a negative one, which is the operation an average is for.

def paired_run(inputs, old_prompt, new_prompt, model):
    rows = []
    for item in inputs:
        a = call(model, old_prompt, item)
        b = call(model, new_prompt, item)
        rows.append({
            "id": item.id,
            "rule_exercised": item.rule_tag,   # which prompt rule this case tests
            "old": a.output, "new": b.output,
            "identical": a.output == b.output,
            "old_cache_read": a.usage.cache_read_input_tokens,
            "new_cache_read": b.usage.cache_read_input_tokens,
        })
    return rows

Three properties of that harness matter more than the code. Both calls go to the same model version, within the same run, because comparing a new prompt this week against last week's logged outputs confounds the restructure with anything the provider changed. The inputs are real production traffic, sampled rather than curated, because a curated set was written by the same people who wrote the prompt and exercises the rules they were thinking about. And every case carries a tag naming which rule it exercises, which is what turns "18 of 500 differ" into "of the 18, eleven are ambiguous-category cases and three are indemnity clauses".

That triage is the deliverable. A restructure is behaviour-neutral when you can name every case that changed and say why, and it is not neutral when you can only say that the aggregate held.

Be honest about what 500 samples can establish. At that size, a shift in a behaviour that occurs on 3% of traffic is barely detectable — you would expect fifteen such cases and a real 20% relative change moves three of them, which is inside sampling noise. Both of those figures are one step of arithmetic and both are worth doing, because the conclusion is counter-intuitive enough that people re-derive it every time. Three in a hundred, out of five hundred, is fifteen. A fifth of fifteen is three. Now ask how much a count of fifteen wobbles on its own: about the square root of it, which is close to four. The effect you are trying to see is smaller than the noise you are seeing it through, so a run that shows no change and a run that shows the full change look the same, and neither is informative. The paired design helps a great deal, since identical inputs remove most of the variance you would otherwise be fighting, but it does not turn 500 into 5,000. For the rules that matter most, sample stratified: pull 200 inputs specifically tagged as exercising the redaction rule rather than hoping enough of them appear in a random 500. And accept that some effects will surface in production rather than in the run, which is an argument for staged rollout and per-rule monitoring rather than for a larger offline suite.

Per-rule monitoring in production is the complement to the offline run and it is cheaper than it sounds. A classifier that already emits a category per clause can also emit a tag for which rule branch produced it — broader-category tiebreak, exclusion, governing-law override, default path — which turns the output stream into a set of rates you can chart. A restructure then rolls out to five percent of traffic and you watch the four rates rather than the aggregate agreement number, which moves too slowly and too little to catch a shift on 3% of inputs. Both of the regressions above would have been visible within an hour on that chart, and the indemnity one within twenty minutes, because a rate dropping from 47% to 12% is not subtle when you are looking at that rate specifically rather than at a weighted average that contains it.

The constraint hiding in the examples block#

Halfway through a restructure, one sentence usually has nowhere to go. On this classifier it was sitting between the third and fourth few-shot examples, in the middle of what would become <examples>: "Note that clauses referencing a governing law outside the EU are always category 7, regardless of subject matter."

That is not an example. It is an absolute rule, phrased as an aside, in the one part of the prompt nobody reads carefully because examples are skimmed for shape rather than read for content. Finding it there tells you three things.

It tells you the rule was probably being carried by demonstration as well as by statement. Somebody added it while editing the examples, most likely right after adding an example that exercised it, and the pair worked together — the sentence stated the rule and the neighbouring example showed it. Moving the sentence out may therefore remove half of a mechanism, which means the check is whether the examples still demonstrate the behaviour on their own, and whether the extracted rule still produces it without them.

It tells you the prompt had no structure at the time it was written. A rule about category assignment belongs wherever category rules live, and it went into the examples because the author was in the examples and there was no obvious better place. That is the failure mode of a prose prompt seen from the inside, and it is an argument for the section lint rather than an argument about this one sentence.

And it tells you the rule is untested, because nobody who reviewed the prompt in nine months knew it was there. When a rule surfaces this way, the first action is to write a case for it before touching it, so that the paired run has something to compare.

What you do with it is a three-step move rather than a cut-and-paste. Write the rule explicitly in the section where it belongs, in unhedged language, since it was already absolute. Leave the examples alone on this change — pulling the sentence and rewriting the examples in one commit makes the paired run uninterpretable. Then run the pair specifically on inputs that exercise the rule, stratified, not on a random sample where non-EU governing law might appear four times. On this classifier that meant 150 tagged clauses; the extracted rule held on 148 of them, and the two failures turned out to be a case the legal team also disagreed about internally, which is a different and better problem to have.

So the standard to hold a restructure to is a low bar that almost nobody currently clears: name every case whose output changed, and say why each one changed. Anything less is a claim that the aggregate did not move, which is a claim about a weighted average and not about behaviour.

The pieces of this topic compose into a short discipline that fits on an index card and that you apply per prompt rather than per incident. Every region that carries text from outside the system gets an unambiguous marker, chosen against a measured collision rate on your own corpus, plus a sentence nearby stating what the region is for. The render path scans for the marker, does something explicit and counted when it finds it, and the boundary cases live in the test suite with expected outcomes written down before the fix was made. Anything the model can trigger is scoped so that a fully successful injection is a bounded, survivable event, and that bound is written down in a form a reviewer can check. The prompt itself is laid out in the same named sections as every other prompt in the repository, with a lint that fails when a section is missing, empty, or has grown an appendix. And any change to that layout — including one that changes no wording at all — gets a paired run whose differences are enumerated by hand.

Most of that is cheap. The scan is twenty lines, the framing sentence is forty tokens, the section lint is a single test, and the paired harness is written once and reused. The expensive parts are the two judgment calls: deciding what your tools are allowed to name, and reading eighteen diffs carefully instead of looking at an aggregate. Those are also the two that keep working when the model underneath you changes, which over the life of a system is the only property that matters.