Failures at a shared origin arrive clustered, not spread evenly

The brief service runs a three-level tree — orchestrator, two sector supervisors, six segment supervisors, eighteen workers — and each boundary in it completes cleanly about 95% of the time on the current instrumentation. Going from two levels to three moved the naive end-to-end figure from 90% to 86%, a loss of four points that sits comfortably inside the error budget the team had agreed with product. An error budget is the share of runs a service is allowed to get wrong over a period before new work stops and reliability work starts. The proposal on the table is a fourth level, and the arithmetic supporting it is not wrong.

The arithmetic is not wrong and the conclusion does not follow, because the model underneath the arithmetic assumes something a delegation tree does not provide. Two customer-visible failures in one week made that visible before the fourth level was built, and an incident channel that treated them as unrelated is where this chapter starts.

The multiplication is correct and the model behind it is not#

Compounding across a path of d sequential handoffs at per-hop success p is p^d, and there is no arithmetic dispute to be had about it. The rule comes from one assumption and nothing else: if knowing that one hop was clean tells you nothing about whether another was, then the chance that all of them are clean is the chance for one, multiplied by itself once per hop.

Working the naive column of the table below is that rule applied four times. At one boundary the figure is just p, 95.0%. At two, multiply 95.0% by 0.95 and the result is 90.3%. At three, multiply that by 0.95 again for 85.7%. At four, once more, for 81.5%. Each step is the same multiplication, and each step is licensed only by the independence assumption.

The two right-hand columns are the point. p^d assumes that whether one boundary completes cleanly tells you nothing about whether another does — the same assumption that lets you multiply the availability of independent services along a request path. In a delegation tree that assumption is false by construction, and the reason is a chain worth walking rather than asserting.

Every delegate below a hop works from a restatement of what that hop received; it has no other source for what its task is. So a defect in what the top hop received is present in every restatement below it. A delegate satisfying that restatement exactly is behaving correctly, and its boundary check confirms as much. Therefore a defect at the top produces clean boundaries all the way down, and one event has determined the outcome of every hop in the run at once, which is the exact opposite of the hops being independent.

A brief at the top that says "the storage and compute segments" when the task was storage and compute security is not a fault at any boundary; it is a premise. Every layer beneath inherits it, satisfies it faithfully, and reports a clean handoff while doing so. That is exactly what happened in the run this failure mode was named for in chapter 3, where a sector's boundaries all scored clean and the run answered the wrong question.

Modelling it as a separate term makes the shape visible. If a fraction q of runs carry a defect at the top that no amount of downstream diligence can repair, and the remaining runs face independent per-hop risk, then

P(run succeeds) = (1 - q) · p_hop^d

Read that from the left, it says a run has to clear two hurdles that are not the same kind of thing. It has to be one of the runs whose premise was sound, which happens with probability 1 - q, and then it has to get its findings across d hops without one of them going wrong, which happens with probability p_hop^d. The two factors multiply because the second only applies to runs that cleared the first.

The measured numbers slot straight in. Over the 200-task regression suite the tree fails 9% of the time. Eleven of those eighteen failures — 5.5% of all runs — have the shared-origin signature: every boundary clean, zero constraint violations, wrong question answered. The other seven, 3.5%, are attributable to a specific boundary that failed and stayed failed after its retry. With q = 0.055 and a per-boundary (not per-attempt) rate of 99%, the model predicts 0.945 × 0.970 = 91.2%, which is the 91% on the dashboard.

Both factors in that product are worth checking rather than accepting. The first, 0.945, is 1 - q with q = 0.055. The second, 0.970, is the per-hop rate raised to the depth: 99% multiplied by itself once per hop along a three-hop path. The two multiply to 91.2%, and the dashboard says 91%.

The corrected column of the table is the same product computed at each depth. Take the naive figure for that depth and multiply it by the same 0.945: 95.0% becomes 89.8%, 90.3% becomes 85.3%, 85.7% becomes 81.0%, 81.5% becomes 77.0%. The marginal-loss column is then just the differences down the corrected column — 89.8 minus 85.3 is 4.5 points, 85.3 minus 81.0 is 4.3, 81.0 minus 77.0 is 4.0 — which is what a colleague means when they say a level costs four points.

Two different kinds of thing are in play in the review that produced this table, and they are worth separating before the table is argued from. The error budget is a requirement: product agreed a share of runs the service may get wrong, and that share is not the team's to move. Choosing marginal loss per level as the quantity to check the proposal against is a decision — one somebody made, probably without noticing they were making it, by reaching for the column that the naive model happens to produce. A requirement constrains what you may ship. A decision about which column to read constrains what you can see, and the second is the one that went wrong here.

Notice what the q term does to the depth argument, and read the table by the gap between its two middle columns rather than down either one. That gap stays near five points at every depth. It does not widen. q appears once as a single factor rather than once per hop, so it does not compound: a defective premise ruins the run whether it is restated once or four times. The colleague computing marginal cost per level is measuring the smaller of the two effects, and measuring it in the metric that already excludes the failures customers actually see.

There is one more reading of the corrected column that changes what q is usually taken to be. Look at the depth-1 row: with a single boundary on the critical path, the corrected figure is 89.8% rather than 95.0%, so the shared-origin term is still there. A flat design has one agent restating the task before the work starts, and a premise defect at that restatement ruins the run exactly as it does in a tree. q is therefore not the price of hierarchy. It is the price of restating a premise at all, which every design pays once, and which depth then multiplies — 2.5% at two levels against 5.5% at three on this workload. Reading q as a hierarchy tax leads to the conclusion that flattening removes it, and the table's own first row says otherwise.

The diagram is two failure shapes drawn to the same scale. The solid path starts at one event and reaches every leaf; the dashed path starts at one boundary and reaches one segment. Both are drawn as single defects, and the difference between them is entirely in how far the arrows go.

Two on one afternoon is a measurement, not a coincidence#

On a Tuesday in March the brief service produced two customer-visible failures four hours apart. Both went out to enterprise accounts, both were well-formed, well-sourced documents, and both answered a question adjacent to the one that had been asked — one substituted market share for revenue growth throughout, the other profiled the storage segment when the request had been about storage security. Between them the service completed 38 other runs that day without incident. The on-call engineer's first note was that the two failures were unrelated, four hours apart, different customers, different sectors.

The instinct to dismiss same-day failures as the tail is worth taking seriously enough to do the arithmetic, because sometimes it is right. At a 9% run-failure rate over roughly 40 production runs a day, the expected number of failures per day is the rate times the count: 3.6. Seeing two is not merely unremarkable, it is below average. The clustering that matters here is not clustering in time, and an argument that starts by counting failures per afternoon has already picked the wrong axis.

What makes the Tuesday pair informative is that both failures had the same symptom — a confidently-answered adjacent question — out of a symptom taxonomy with six populated classes in the last quarter's incident log. If the two failures were independent draws from that distribution, matching symptoms is a coincidence of about one in six even before weighting by class frequency. Pair that with the second observation, which nobody made until the traces were pulled: both runs used the same brief-template revision, deployed the previous Friday, in which a phrase in the task-framing section had been shortened. Two failures, same symptom, same upstream artefact, on the first two runs where that artefact met a security-scoped request. That is not the tail of an independent process; that is a defect with a blast radius, meaning a single change whose damage lands across a whole set of runs at once rather than on one of them, and it had produced zero failing boundaries in either run.

Retrying is the response this misreading recommends, and it is the tempting defence in this chapter. It is tempting for a good reason: a retry is the correct response to almost every other kind of failure in a distributed system, and it costs nothing to reach for. What makes it useless here is that the quantity a retry varies is the attempt, while the quantity that decides the outcome — the brief — is held constant across every attempt. A retry at every boundary re-sends the same brief through the same path to the same model; the boundary was never the thing that failed, and the second attempt satisfies the same misstated premise as faithfully as the first.

It is worth pricing so it can be rejected on its own terms rather than on principle. Three attempts per boundary across 26 boundaries triples the work: a run that costs about 640,000 input tokens becomes one that costs close to 1.9 million, and the invoice triples with it, from roughly $3.20 to $9.60 per run at Opus rates. That buys a movement in the run success rate of nothing measurable, because the 5.5% it needs to fix is not in the population retries operate on.

Instrumenting for correlation before the next incident#

Telling a correlated failure from an independent one is a data-collection problem, and all four of the signals it needs are cheap and none of them are collected by default.

The first is a fingerprint of the premise, propagated unchanged down every hop. Hash the task framing, the constraint list and the template revision at the root, and carry the digest in every DelegationBrief and every BoundaryVerdict. When two failures share a fingerprint prefix, they share an origin by definition, and the question stops being a judgment call.

import hashlib

def premise_fingerprint(task_framing: str, constraints: list[str], template_rev: str) -> str:
    h = hashlib.sha256()
    h.update(template_rev.encode())
    h.update(task_framing.strip().lower().encode())
    for c in sorted(constraints):
        h.update(c.encode())
    return h.hexdigest()[:12]

Two details in that function are doing work. The constraints are sorted before hashing, so two runs whose constraint lists differ only in order produce the same digest — the order is not part of the premise, and treating it as part would split one origin into several. The task framing is stripped and lowercased for the same reason. What is deliberately not normalised is the template revision, which is prepended raw, because a template change is exactly the kind of upstream event the fingerprint exists to separate.

The second is a sibling-outcome field on every failed run: how many of the run's 26 boundaries scored not-clean. Independent failures produce runs with one bad boundary and twenty-five good ones. Shared-origin failures produce runs with zero bad boundaries and a bad answer, which is the most diagnostic single number in the whole system and shows up as a spike at zero in a histogram nobody had drawn.

The reason one field separates the two populations follows from what each kind of failure is. An independent failure is a hop going wrong, so by definition at least one boundary in that run scored not-clean, and the count is one or a small number. A shared-origin failure is a premise going wrong, and every hop below a bad premise satisfies it, so the count is zero. The two populations therefore land on opposite sides of a single integer, and a histogram of that integer over failed runs separates them without anybody having to classify anything. The mean count of not-clean boundaries across the shared-origin runs is that spike, reported as an average.

What the field cannot do is tell you which premise defect you have, or whether a run with zero bad boundaries failed for some third reason nobody has enumerated. It sorts failures into two bins reliably and says nothing about what is inside either one.

The third is symptom classification at the run level, applied to failures by whoever triages them, using a fixed vocabulary rather than free text. Six classes is enough — wrong question answered, missing company, stale figure, unsourced claim, contradictory figures, timeout. The value is entirely in the fixed vocabulary: free-text incident notes cannot be grouped, and two failures with the same cause described in two engineers' words look unrelated in exactly the way the Tuesday pair did.

The fourth is a depth control: replay the failed run's top-level brief through a flat, single-orchestrator version of the same task. The logic of that control is a clean separation. If the flat run reproduces the failure, the defect travelled with the premise and no depth policy addresses it. If the flat run succeeds, the only thing that differed was the restatement, so depth is implicated.

Restatement is what depth actually adds to the risk#

The q term is not a constant of the universe; it is a property of how many times the premise gets paraphrased before it reaches the work. When the same 200 tasks were run through the two-level tree — orchestrator, six segment supervisors, eighteen workers — the shared-origin failure rate was 2.5%, five runs rather than eleven.

The mechanism behind that doubling is worth naming, because it is not random degradation. Every level added is another agent that reads the brief in its own words and writes a new one. The failure modes of paraphrase are systematic: qualifiers get dropped before nouns do, and the qualifier is usually the part that made the task specific. Storage and compute security becomes storage and compute far more often than it becomes something unrecognisable. A noun that vanishes leaves a sentence that reads wrong; a qualifier that vanishes leaves a sentence that reads fine and means something else.

That is the honest form of the compounding claim, and it is stronger than p^d rather than weaker. Depth raises the independent per-hop risk a little — four points per level on the table above — and raises the correlated risk a lot, from 2.5% to 5.5% for one additional level on this workload, with a failure mode that produces confident wrong answers rather than obvious breakage. Neither effect is visible in a per-boundary dashboard, and neither is addressed by a retry.

The q term is a fitted parameter#

The model has one free parameter and it is fitted to the data it explains. q was not predicted from anything; it was measured by counting the failures with the shared-origin signature and dividing by the suite size. So 0.945 × 0.970 = 91.2% matching the dashboard's 91% is a consistency check rather than a prediction, and a model with a fitted parameter that reproduces its own input is doing less work than it appears to. What it earns is the separation of two populations that were previously one number, and that separation is the useful part.

The rest of the model still assumes independence, and only for the hops. Inside the (1 - q) fraction, p_hop^d is doing exactly what the naive column did, with all of its assumptions intact. If two boundaries fail together for some reason unrelated to the premise — the same rate limit, the same malformed fixture — that correlation is in neither term.

The depth control is a strong instrument with a narrow reading. It tells you whether a given failure survives flattening; it does not tell you what fraction of a future workload will. Nine of eleven is a share of one quarter's failures on one task distribution, and the two that did not reproduce are enough to say restatement contributes something rather than nothing.

None of which is an argument that the three-level tree is wrong. It is an argument that the three-level tree costs about five points of run success and about $0.40 of coordination per run relative to the flatter version, and that a cost of that size has to be paid for by something. What it is supposed to be paid for by is better output — briefs that are more complete, more consistent, or more recoverable than the flat design produces. The team has been asserting that gain for two quarters without measuring it against a baseline that anyone had kept current, which turns out to be where most of the evidence for hierarchy quietly goes.