Bisecting a prompt by deletion

Bisecting by deletion is the prompt-level version of git bisect: instead of looking for the instruction that is breaking a case, you locate it by removing half the instructions, re-running the case, and recursing into whichever half still misbehaves. Five or six rounds take a thirty-four-rule prompt down to one or two candidates. It is the dependable move when a specific case has become unreliable, nothing else in the system changed, and careful reading has turned up nothing.

Reading is a bad tool for this and the reason is combinatorial. Thirty-four rules make 561 pairs, and a conflict does not have to be a pair — rules 6, 19 and 27 can be individually consistent and jointly unsatisfiable on one input class. A human reading top to bottom holds maybe five or six rules in working memory at once, so the pairs that get checked are the ones that are close together on the page, and the pair that actually collides is usually far apart, because rules written at the same time about the same thing tend to have been written by someone who was thinking about both.

Both numbers in that argument are worth deriving rather than quoting, because the contrast between them is the reason the procedure exists. An unordered pair drawn from thirty-four rules is a choice of two: thirty-four candidates for the first, thirty-three left for the second, and every pair arrived at twice that way, so the count is thirty-four times thirty-three, halved. That is the five hundred and sixty-one. The bisection's step count comes from the opposite operation. Each step halves the candidate set, so the number of steps is the number of times thirty-four can be halved before one or two rules remain — thirty-four, seventeen, nine, five, three, two — which is where five or six rounds comes from. Reading grows with the square of the rule count. Halving grows with its logarithm. That difference, and not any claim about human attention, is why one scales and the other does not.

Halving the rule set instead of reading it#

A run over a support agent's system prompt, hunting a chargeback case that had dropped to eighty-percent escalation, looks like this. Each node is a rule subset and the number under it is the rate at which the case behaved correctly.

The last step is the one that matters and it is easy to skip. Rules 33 and 34 each behave perfectly on their own and misbehave together, which is the signature of a conflict rather than of a single bad rule. Splitting a colliding pair and stopping at the first suspicious rule gets you a deletion that fixes nothing, because either rule alone is innocent.

Read those three bottom measurements as a small truth table and the conclusion is forced rather than inferred. Remove rule 33 and the behaviour comes back. Remove rule 34 and the behaviour comes back. Remove neither and it stays broken. So neither rule is sufficient on its own to break the case, and each is necessary for the break — which is what "these two collide" means, stated as three measurements instead of as an interpretation of two sentences. It also explains why no investigation aimed at a single guilty rule could have arrived here: rule 33 examined alone is innocent, rule 34 examined alone is innocent, and a procedure that stops the moment one arm comes back green stops on an innocent rule with a clean result in hand.

This is git bisect with one important difference. In a bisect over commits, the test either passes or fails, and one run tells you which half to take. Here the test returns a rate, the rate is the thing that moved, and a single run tells you almost nothing. If you run each arm once and the guilty half happens to succeed, you walk down the wrong branch and spend the rest of the afternoon proving that rules 18 to 26 are innocent.

How many runs each arm needs#

The two states you are trying to distinguish are roughly 0.80 and 0.95. That gap is not large in sampling terms. At twenty runs per arm, an eighty-percent arm produces 16 successes with a standard deviation of about 1.8, and a ninety-five-percent arm produces 19 with a standard deviation of about 1.0; the distributions overlap enough that you will misread a step roughly one time in six, and over six steps that is close to a coin flip on whether the whole bisection lands anywhere useful.

Those two standard deviations are computed rather than estimated, and the formula is short enough to carry around. Twenty runs of an arm that succeeds with some fixed probability is a count of successes out of twenty independent tries, and the spread of such a count is the square root of three things multiplied together: the number of tries, the success probability, and the failure probability. For the eighty-percent arm that is twenty times four fifths times one fifth, and the square root of the result is the first figure quoted. For the ninety-five-percent arm it is twenty times nineteen twentieths times one twentieth, whose square root is the second. Notice which arm comes out tighter, and why: the product of the two probabilities is at its largest when they are equal and shrinks as either approaches certainty, so an arm near ninety-five percent is intrinsically steadier than one near eighty. The two distributions are therefore not symmetric about the gap between them, and most of the overlapping is done by the noisier arm.

Running the numbers properly: to separate 0.80 from 0.95 at the five-percent significance level with eighty-percent power — a one-in-twenty chance of calling a difference that is not there, and an eighty-percent chance of seeing the one that is — takes about 35 observations per arm. Fifty is a reasonable working number if calls are cheap, because it also gives you room to detect the intermediate cases — an arm sitting at 0.88 is telling you something, and you want to be able to see 0.88 as distinct from both endpoints. With two arms per step, fifty runs each, and six steps, the whole bisection is about 600 calls. At roughly 4,300 tokens in and 200 out per call, that is around 2.6 million input tokens; on a mid-tier model at $3 per million input tokens it lands under ten dollars, and at eight concurrent requests it finishes in about twenty minutes. That is a cheap afternoon compared to three engineers reading a prompt.

The 35 deserves a footnote, because two standard methods disagree about it and you will meet both. The arcsine transformation, which is what most proportion calculators use, gives 35 per arm for 0.80 against 0.95; the unpooled normal approximation on the same two numbers asks for about 75. Neither is wrong — the arcsine transform stabilises the variance and is generous near the ends of the range, while the normal approximation is more conservative there — and the practical consequence is that any n between 35 and 75 is defensible. Fifty sits in the middle of that band. Where calls are expensive enough for the difference to matter, run a sequential rule rather than a fixed n: sample in blocks of twenty per arm, stop as soon as the two arms' ninety-percent intervals no longer overlap, and give up at eighty. On this prompt that ended the average step at 34 samples per arm and brought the whole bisection to about 390 calls, at the cost of roughly one extra misread step per ten complete runs.

The harness is small enough to write once and keep.

import itertools, statistics
from concurrent.futures import ThreadPoolExecutor

def render(rule_ids: list[int]) -> str:
    body = "\n".join(f"{i}. {RULES[i]}" for i in sorted(rule_ids))
    return f"{TASK_PREAMBLE}\n\n{body}\n\n{OUTPUT_SCHEMA}"

def measure(rule_ids: list[int], n: int = 50) -> float:
    prompt = render(rule_ids)
    def one(_):
        out = client.messages.create(
            model=MODEL, system=prompt, temperature=0.7,
            max_tokens=400, messages=[{"role": "user", "content": CHARGEBACK_CASE}],
        )
        return escalated(out)          # the case's own assertion
    with ThreadPoolExecutor(8) as pool:
        hits = list(pool.map(one, range(n)))
    return sum(hits) / n

def bisect(rule_ids: list[int], baseline: float, floor: float) -> list[int]:
    if len(rule_ids) <= 2:
        return rule_ids
    mid = len(rule_ids) // 2
    left, right = rule_ids[:mid], rule_ids[mid:]
    l, r = measure(left), measure(right)
    if l < floor and r < floor:
        raise RuntimeError(f"both arms low: {l:.2f}/{r:.2f} — resample before splitting")
    return bisect(left if l < r else right, baseline, floor)

Three details in that harness are load-bearing. TASK_PREAMBLE and OUTPUT_SCHEMA are never removed, because a prompt with no task statement fails for reasons that have nothing to do with the conflict, and every arm would look broken. Temperature stays at whatever production uses — bisecting at temperature 0 collapses the rate to 0 or 1 and destroys the signal you are measuring. And floor is a threshold you set from the baseline rather than a fixed number, typically the midpoint between the broken rate and the healthy rate, so a step where an arm lands at 0.88 asks you to look rather than silently branching.

When both halves look broken#

The RuntimeError in that code is there because both-arms-low is the most common way a bisection goes wrong, and the most common cause is not two independent conflicts. It is undersampling. Two arms measured at 0.82 and 0.84 with n=20 are, with high probability, one arm at 0.95 and one at 0.80 that happened to land close together. Before concluding anything, re-run both arms with three or four times the samples. In practice that resolves it most of the time.

The second cause is that you split a rule away from the behaviour it produces. If rule 19 is the rule that says to escalate on chargebacks, the half that does not contain rule 19 will fail every single time, and it is failing for a boring reason. This is why you run a control arm at every step: the full prompt, to confirm the baseline has not drifted, and the arm you are about to discard, to confirm it fails for the right reason. An arm at 0.02 is not "still misbehaving", it is missing the instruction that defines the case, and the bisection should treat it as invalid rather than as a signal.

Genuine multi-conflict prompts do exist, and they announce themselves differently: both halves sit meaningfully below baseline but well above zero, the effect persists at n=200, and each half bisects down to its own pair. That happens, and when it does you have found two bugs rather than one, which is worth the extra hour. It is just much rarer than the arithmetic of small samples makes it look.

Leave-one-out is worth knowing about because it is the first thing people build, and it is not useless — it just answers a different question. It tells you which single rules are load-bearing for a case. On an interacting pair it returns two rules that both look guilty, with no way to rank them, and it costs nearly three times as much. Reordering is the cheapest diagnostic of all: shuffle the rule order, re-measure, and if the rate moves substantially you have confirmed the behaviour is decided by a precedence contest rather than by any single instruction, which is enough to justify the bisection budget to whoever is asking why you are spending an afternoon on this. The step-by-step isolation procedure in the Prompt Debugging chapter of the Prompt Engineering Playbook covers the same ground with a different bookkeeping scheme, and is worth reading if you want a second template for the harness.

What halving changes besides the rule set#

Cutting seventeen rules also removes about 2,000 tokens, and the instructions that survive do not behave identically in a 2,100-token prompt and a 4,100-token one. Adherence to any given instruction tends to rise as the set around it shrinks, so part of every healthy arm's score is an artefact of length rather than evidence about the rules you took out. Two side measurements on this prompt put a size on the effect. Moving rules 33 and 34 to the top of the untouched 34-rule prompt, changing nothing else, took the escalation rate from 80 percent to 71, so position alone is worth nine points here. Re-running the healthy 17-rule left arm with fourteen paragraphs of inert product background pasted back in to restore the original token count moved it from 96 percent to 92, so length alone is worth about four.

Both of those side measurements are one subtraction each, and both were designed so that only one thing could account for the difference. Moving rules 33 and 34 to the top changed position and nothing else — the same rules, the same count, the same token total — and the escalation rate went from eighty percent to seventy-one, so the nine points between them can only be position. Restoring the token count of the healthy left arm with inert background changed length and nothing else — the same seventeen rules, in the same order — and the rate went from ninety-six to ninety-two, so the four points between them can only be length. Each is the one-variable rule applied to a nuisance instead of to the thing under test, which is the only way to put a size on a nuisance rather than worrying about it.

Neither number invalidates the procedure, and both change how the numbers should be read. Compare each arm against its sibling at the same step rather than against the full-prompt baseline, because the two arms are the same size and the baseline is not. Treat an arm at 92 percent as healthy when the other arm is at 76, and do not spend a round chasing the four points. Then confirm the final pair inside the full prompt rather than inside the four-rule fragment you ended on: the 95, 94 and 78 at the bottom of the tree were measured by taking the complete 34-rule prompt and removing rule 33, then rule 34, then both. Those are the numbers a pull request can quote, and they are the ones a reviewer can reproduce against production.

What bisection is not is a general evaluation. It localises one failing case, and the rule pair it finds is guilty for that case; the same pair may be entirely benign on the rest of your traffic, or may be causing three other failures nobody has reported. Running it a second time against a different failing case is cheap, and on a prompt that has produced one conflict it is usually worth doing.

Halving works because it replaces a question about meaning, which a human answers badly across thirty-four rules, with a question about a measured rate, which a harness answers well six times in a row. Everything expensive about the procedure is in the sampling rather than in the logic: the tree is four lines of recursion, and the reason a bisection goes wrong is almost always that an arm was measured with twenty runs when it needed fifty.

What it hands you at the end is a pair and a demonstration, not a decision. Deciding which of two clearly written, currently defended rules should be removed takes a different kind of evidence, and it is not in the prompt text at all.