Eleven prompts, 68% accurate, and a twelfth after every incident

A long chain of small focused prompts is a defensible design and it is usually arrived at one incident at a time. Every step has an owner, an eval set and a pass rate; every step was added by someone solving a real problem in a review where the alternative was to leave a known failure in production. Each individual decision was locally correct, and the arrangement they add up to can still be losing eighteen points and nine seconds without anyone able to name where.

The obvious response — that this is over-engineered and should be collapsed — is the wrong first move, because it is exactly as evidence-free as the last step somebody added. Where nobody can say which of eleven steps carries the losses, a collapse is as likely to destroy the two steps doing real work as to fix anything, and the team will correctly observe that you replaced their untested design with your untested one.

What small focused prompts genuinely buy#

Take the design seriously first, because most of it is defensible and you will need to have said so before anyone listens to the rest.

When a step misses something, the review has an address: it is step nine, it has a prompt of 40 lines, one person owns it, and the fix is reviewable in isolation. Compare that to a 600-line mega-prompt where the same fix is an edit that could plausibly affect three other jobs, and where the only way to know is to re-validate everything. Small prompts are individually auditable, which matters more than usual in a regulated line of business where a compliance reviewer may need to read the sanctions logic without reading anything else.

Per-step model selection is real money. Document classification runs on a small fast model at a fraction of the cost of the extraction steps, and that option exists only because the steps are separate. Steps whose output depends only on the document can be cached on document hash, so a resubmission skips them entirely.

What is missing is the second half of each of those decisions, and its absence is the actual defect in how such a team works.

The ratchet, and the arithmetic under it#

The asymmetry that turns that loop into a ratchet is in the evidence rather than in anybody's judgement. The case for adding a step is an incident: a specific submission, a specific customer, a date, and somebody who had to apologise. The case for removing one would be a counterfactual — how much this step has caught since it shipped, set against how much it has broken — and no such number is collected anywhere, because nothing in the loop asks anyone to collect it. One direction always has evidence and the other never does. A process built that way accumulates steps whatever the people running it believe about decomposition, which is exactly why this is a defect in the process and not a position anybody holds and could be argued out of.

Nothing in that loop ever removes a step, and nothing measures whether an added step catches more than it breaks. That second question has an answer you can compute. Say a proposed step is meant to catch a failure occurring on about 0.4% of submissions, and a 40-line extraction prompt on this stack has an own-fault rate of roughly 1% on inputs it should handle. It also fails open, which here means passing a malformed upstream input through untouched rather than stopping on it, so its error rate applies to everything the target failure does not cover:

caught  = 0.004          # failures the new step prevents
broken  = 0.010 * 0.996  # its own error rate, on everything else
print(round(caught - broken, 4))   # -0.0060

Both lines of that calculation are worth reading slowly, because the second one is the line teams leave out. caught is easy: the step exists to stop a failure occurring on 0.4% of submissions, so at best it prevents 0.4% of submissions from going wrong. broken is where the argument lives. The new step is a prompt, and a prompt of this kind is wrong about one input in a hundred; the question is which inputs it is loose on. Because it fails open — it passes a malformed input through rather than halting on it — it runs on every submission, including the 0.996 of them that never had the target failure. So its own error rate applies to that share, which is why the second line multiplies rather than standing alone. Subtract, and the step removes four failures in a thousand while introducing very nearly ten.

Six tenths of a percentage point of net harm, from a step that would be logged as a fix.

Two properties of that arithmetic decide the answer and neither is about the quality of the prompt anybody writes. The first is the ratio between the two rates: a step is only worth adding when the failure it catches is commoner than the errors it makes, and a rare failure is a hard thing to be net-positive against. The second is which direction the step fails in. A step that failed closed would stop on inputs it could not handle, and its errors would be escalations rather than wrong answers — a different currency, and one that shows up in a queue instead of in a customer's hands. Fail-open is what puts the step's error rate on the whole population, and it is chosen by default because it is the behaviour that does not break anything visibly on the day it ships. That calculation takes two minutes and it is rarely run, which is the single highest-leverage thing to change about a team's process regardless of what happens to the design.

The same formula says when an added step is worth having, which is the half that makes it something a team can adopt rather than a stick to beat proposals with. A step catching a failure that occurs on 3% of traffic, with an own-fault rate of 0.5%, nets 0.03 − 0.005 × 0.97, or about 2.5 points, and no amount of dislike for long chains argues against that. The three quantities to demand from a proposal are therefore the incidence of the failure being caught, the own-fault rate of a prompt of the kind proposed, and whether the new step fails open or closed. Only the first is specific to the incident; the other two are properties of the stack that a team can measure once and reuse in every review afterwards. That is what turns the arithmetic from a special request into a line on a template.

Compounding is the other half. Eleven steps at an average own-fault pass rate of 96.8% multiply to 0.968¹¹ = 0.699, which is within a point of a measured 68% — so a pipeline in that state is losing roughly what eleven steps at that quality would lose, and no single step is obviously broken. That is why the team cannot find the problem: there is no problem, in the sense of a defect with a location. There are eleven small taxes and a design that levies all of them.

The exponent in that expression is doing the work and it is worth saying where it comes from. A submission is processed correctly only if every one of the eleven steps handles it correctly, so the chance of an end-to-end success is each step's pass rate multiplied together — and where those rates are all about the same, multiplying eleven of them is raising one to the eleventh power. What makes the result surprising is how gentle each factor looks. A single step at 96.8% loses three points and change, which no reviewer would call a problem. Eleven of them lose thirty between them. The losses compound rather than add — each step works on the survivors of the step before, so each takes its cut of a slightly smaller pile, and the total lands a little under eleven times three — but that correction is nothing beside the headline. A defect too small to open a ticket about, levied eleven times, is a third of the product.

Run the same relationship backwards and it prices the design rather than merely describing it. For a chain of eleven steps to deliver nine cases in ten, each step has to be right on a bit over ninety-nine in a hundred — because the per-step rate is the eleventh root of the target rather than anything near it. That is a much harder standard than the one anybody was holding these prompts to, and nothing in an incident review ever states it, which is how a team ends up with eleven steps each of which passed its own review.

Latency has the same structure, and the latencies here are as unremarkable as the pass rates. Eleven sequential round trips at 700–900 ms each is 8 to 9 seconds, and there is nothing pathological in any individual call. Some of those steps do not depend on each other — sanctions screening and loss-history extraction both read the raw document and neither reads the other's output — so they are a chain only because they were written as one.

Ask for measurement and nothing else#

Per-step own-fault rates on clean inputs, inherited and amplified counts, and downstream evals rebuilt on real upstream outputs wherever a step's eval uses hand-written gold: that is the whole proposal for the first fortnight. Ask for nothing else and change nothing. It is a request no reasonable team refuses, and it converts every subsequent argument from opinion into arithmetic.

What you give up when a step goes away#

Every step you remove costs you attribution. After the first change, an address defect no longer has a prompt to point at; it has a library version and a lookup table, and the next incident review of that kind will be harder rather than easier. That is a real loss and the honest way to present it is as a trade — three failure surfaces and 2.4 seconds against the convenience of a prompt to blame. Teams that pretend the trade is free lose the argument the first time the collapsed step fails.

The process change matters more than any of the three. Propose that every future added step carries the two-minute arithmetic above, and that steps added after an incident get a review date at which they must show they caught more than they broke. A doctrine of small focused prompts is not the thing to attack, because it is broadly right and it is why anyone can debug such a system at all. The ratchet is the thing to attack, and it is a process defect rather than a design philosophy, which makes it something a team can agree to without conceding they were wrong.

The order that holds up is the same order to run on your own pipeline. Score each output against its own criteria before proposing any split, and ablate to find out whether the jobs are interfering or whether an instruction was never written. Price the change in duplicated input tokens and added sequential latency at your real volume. Ask what each proposed stage receives that its predecessor did not have, and refuse the ones that receive nothing. Check whether a rule, a retrieval or a restructured single prompt does the stage's work for milliseconds instead of a round trip. Instrument per-step pass rates with a fault label on every failure. Rebuild every downstream eval on the real output distribution of the step above it. Write the handoff contract down — what it carries, whether it validates, whether the source stays available, what happens when it arrives wrong. Then, and only then, argue about the architecture, with numbers on your side of the table.

What that order protects against is the failure this chapter is named for, which is not over-decomposition and not under-decomposition. It is a loop in which every change is a locally correct addition, no change is ever a removal, and nobody has computed whether the additions have been paying. Eleven steps at 96.8% is not a mistake anybody made; it is a mistake nobody made, eleven times.