Score each job separately before you split the prompt doing three

Prompt decomposition is the move of taking one prompt that asks a model for several things at once and breaking it into several prompts that each ask for one thing, joined by ordinary application code that passes results between them. What it replaces has a name too. A mega-prompt is what a single prompt turns into as a feature grows: sixty-odd lines, three unrelated outputs, several authors over a year, and no way to change one part of it without putting the other parts at risk. When the resulting stages run in order and each one reads the output of the one before it, the arrangement is called prompt chaining; when they run side by side on the same input, it is a fan-out, and the difference between those two shapes decides what a split costs.

The advice to decompose exists because the decomposed version is addressable in ways a mega-prompt is not. Each job gets a pass rate of its own, so a quality regression has an owner and a number attached to it; each prompt can be edited without re-validating the others; and each call can run on whichever model is cheap enough for the work it does. You meet the pattern wherever a feature does more than one thing to a piece of text — extract the fields from an invoice and then validate them, pick the sections of a policy document that bear on a question and then answer from them, classify an incoming request and then route it to the handler for its type. What the advice leaves out is that a split is an experiment you only get to run once, and running it before anything has been measured spends the evidence that would have justified it.

Two explanations, and a split makes them indistinguishable#

Take the common case: one prompt does three jobs and exactly one of the three looks erratic. There are two live stories for that, and they call for different work. The first is interference — three unrelated jobs competing inside one request, so the instructions for one dilute the attention available to another, the output format for one bleeds into another, and the model spends its budget where the prompt spends its words. The second is that the rules for the failing job were never written down. One line of instruction saying "based on how urgent the ticket is" leaves the model to invent a rubric on every call, which is a sampled decision — the model draws its answer from a probability distribution rather than following a written procedure — and varies like one. Topic 4.1, on the anatomy of a prompt, covers which slot in a prompt that hole lives in.

"Competing for attention" is a phrase that can mean almost anything, so it is worth separating into the two mechanisms actually underneath it. The first is that three sets of instructions share one window. Every line written to govern the reply is a line the priority instructions are contending with for influence over what gets produced, in the same way a retrieved document contends with a rule. The second is that three outputs share one completion, and a completion is produced left to right with every token conditioned on everything already written. The summary is not merely generated alongside the reply; it is generated first, and then read by the model while the reply is being produced. A constraint that clips the summary leaves a clipped prefix, and the reply inherits its register from what precedes it. Both mechanisms are checkable, both disappear the moment the jobs are in separate calls, and neither has any counterpart in the second story — which is exactly what makes the two stories separable by measurement rather than by argument.

Both stories predict exactly the symptom. Both are consistent with the other two jobs looking fine, because summarisation needs almost no policy and the model has seen a hundred million examples of it. If you split now and the failing job improves, you will not know whether it improved because the job got its own call or because whoever wrote the new prompt inevitably wrote down the missing rules while they were at it. You will have shipped a change, absorbed the added latency and token cost of two extra calls, and learned nothing you can apply to the next mega-prompt. The measurement that separates the two stories has to happen before the split, on the prompt as it stands.

Three outputs are three evals#

The first thing to notice about a mega-prompt is usually that nobody has ever scored it. There is a pass rate, and it is a single number a reviewer produces by reading the whole output and marking it acceptable or not. A composite score over three unrelated outputs tells you the proportion of items where all three happened to be fine, and it moves when any of them moves, so it never points at anything. Worse, it compounds rather than averages. A prompt whose three jobs pass independently at 91%, 62% and 74% is fully correct on 0.91 × 0.62 × 0.74 = 42% of items, so a reviewer applying a strict all-three standard reports 42% and a reviewer who waves through anything with two good outputs reports something near 80%. Both numbers are about the same prompt, neither is wrong, and neither tells anyone which of the three jobs to open.

Multiplication is the right operation there for a reason worth stating, because it is what makes the composite number behave so badly. A strict reviewer marks an item acceptable only when all three outputs are acceptable, so the item passes when the summary passes and the priority passes and the reply passes. Where the three verdicts are independent of one another, the chance of all three holding is the chance of each holding multiplied together, which is why the three rates go into one product rather than into an average. An average of 91, 62 and 74 would be in the mid-seventies and would describe nothing that happens to any item. The product describes the item, and it is lower than every one of its factors, because each additional requirement can only remove items from the passing set.

Independence is an assumption in that sentence and not a fact, which is worth flagging because the rest of this chapter is about testing it. If the three jobs interfere, their verdicts are correlated and the true joint rate will not equal the product. That is not a reason to distrust the arithmetic; it is a reason to notice that the composite number depends on something nobody has measured, on top of already being unable to say which factor moved.

Take 300 items with known-good labels and score each output against its own criteria. Priority against a human-adjudicated correct level. Summary against a rubric — does it name the product area, does it state what the customer wants, is it under 40 words. Reply against whether an agent would send it after light editing. That gives three independent numbers on the same 300 rows, which is what an eval is here: a scored set of examples whose correct answers are known in advance.

def score_run(rows):
    return {
        "summary": mean(rubric_summary(r.out.summary, r.gold) for r in rows),
        "priority": mean(r.out.priority == r.gold.priority for r in rows),
        "reply":    mean(rubric_reply(r.out.reply, r.gold) for r in rows),
    }

# mega-prompt, 300 tickets
# {'summary': 0.91, 'priority': 0.62, 'reply': 0.74}

Sixty-two percent on a four-level classification is worth holding against the floor. Guessing uniformly gets 25%, and always answering P3 — the modal level in this queue at 44% of tickets — gets 44%. So the prompt is doing real work and doing it badly, and the confusion is not uniform: 71 of the 114 errors are one level too low, and 38 of those are P1 tickets marked P2 or P3. The model is systematically under-escalating rather than answering at random, which is what an under-specified rubric looks like when the training prior is doing the deciding. That is a diagnosis pointing at the second story, and it is still only a suggestion.

Two floors and one count are doing the work in that paragraph, and each is a line of arithmetic rather than a judgement. The uniform-guess floor is one over the number of levels: four levels, so a coin-flipping classifier lands a quarter of the time. The majority-class floor is the share of the most common label, because a classifier that always answers P3 is right exactly when P3 is right, and P3 is the answer on 44% of this queue's tickets. That second floor is the one that matters, and it reframes the headline: 62% is not thirty-seven points above chance, it is eighteen points above answering P3 to everything. The error count follows the same way: take the 62% of 300 items that came back correct away from the 300, and what remains is the 114 that did not.

What makes those 114 errors a diagnosis rather than a complaint is their shape. A classifier that had no rubric and was answering at random would scatter its errors across the three levels that are not correct, so a ticket that should be P1 would land on P2, P3 and P4 in roughly equal numbers. These errors do not scatter. Nearly two-thirds of them are one level too low, and the single largest group is P1 tickets marked P2 or P3, which is a consistent direction rather than a spread. A consistent direction means something is deciding, and the something is not the prompt, because the prompt says only "based on how urgent the ticket is". What is deciding is whatever the model's prior associates with urgency in ticket text, which is a rubric nobody in the room wrote or can read.

The ablation that settles it#

An ablation removes one element of a system, holds everything else fixed, and re-measures to find out what the removed element was contributing. Here that means taking the priority instructions out of the mega-prompt verbatim — the same single line, the same context, the same ticket — and running them as a standalone call that returns nothing but a level. Nothing about the instructions changes. The only thing that changes is whether the model is also producing a summary and a reply in the same completion.

That table is a two-by-two, and seeing it as one is what makes the conclusion safe to draw. Two things could have changed — whether the job runs on its own, and whether the rules are written down — and the four rows are the four combinations of those two. Read first the pairs that differ in only one factor. With the rules left vague, isolating the job moves 62% to 64%. With the rules written out, isolating the job moves 87% to 89%. Both differences are two points, and two points is below what three hundred rows can resolve. Now read the other factor. With the job left in the mega-prompt, writing the rules down moves 62% to 87%. With the job isolated, writing them down moves 64% to 89%. Both of those are twenty-five points. Each factor's effect is the same size whichever level the other factor sits at, which is the reassuring outcome: the two changes are not entangled, so the conclusion does not depend on which one was tried first.

The shape of that table is the common case rather than a curiosity. Isolating the job bought two points; writing the rules down bought twenty-five. Run the ablation in the other direction as well, because it is cheap and it protects you from the reverse error. Score the summary alone and the reply alone. If summary-alone came back at 91% and summary-in-mega at 78%, you would have direct evidence that adding two more jobs to the request costs the summary thirteen points, and the split would be doing something a rubric cannot. Interference is real and it does happen, most visibly when one of the jobs has a strict output format and another produces free prose, or when one job's constraints are the kind a model tends to over-apply — a "be concise" for the summary quietly shortening the reply. The point is that interference is a measurable claim.

What 300 rows can and cannot resolve#

Three hundred is a deliberate number rather than a round one. The standard error — how far a measured proportion drifts from one sample of that size to the next — is 2.8 percentage points on a proportion near 0.6 at n=300, so the exercise can resolve differences of about six points and cannot resolve differences of two.

That figure comes out of one expression and it is worth being able to reproduce, because it is the number that decides how many rows to label. Take the proportion, multiply it by one minus itself, divide by the number of items, and take the square root. At six in ten over three hundred items, the multiplication gives a little under a quarter, dividing by three hundred gives eight in ten thousand, and the square root of that is the 2.8 points. Two of those is where the six comes from: a difference has to clear roughly two standard errors before it stops being the kind of thing one resampling would produce on its own. That is exactly the resolution this decision needs: the question is whether isolating the job buys a large amount or a small one, and 62 against 64 answering "we cannot tell them apart, and the ceiling on the difference is small" is a conclusive result for the purpose. If the ablation had come back at 62 against 78, nobody would have needed a second sample to know what to do.

The objection raised against doing this first is always time, so it is worth being exact about how much. Adjudicating a few hundred correct labels is a morning of one domain expert's attention, and the arguments it provokes are not overhead — they are the tie-break rules the prompt was missing, arriving in written form. The summary and reply rubrics can be graded by a model on a three-point scale and spot-checked by a human on 40 rows; where the human and the grader agree on 37 of them, that is good enough for a decision of this size and nowhere near good enough to publish. Inference is negligible: four runs of 300 tickets, each about 900 tokens of input, is roughly 1.1 million input tokens plus a few hundred thousand output tokens, which lands in the low tens of dollars on any provider.