A clean boundary and a successful run are different events

The weekly brief service now covers eighteen companies rather than six, and the tree grew a level to hold them: the orchestrator delegates to two sector supervisors, each sector supervisor delegates to three segment supervisors, and each segment supervisor runs three research workers. A finding about Corveil now crosses three delegation boundaries on its way to the top.

Counting the boundaries uses the rule from chapter 1 — every agent below the root is reached by exactly one delegation, so the boundary count is the number of agents below the root. Two sector supervisors, three segment supervisors under each of those, and three workers under each segment: two, then six, then eighteen. Twenty-six boundaries execute per run. The dashboard the team built after chapter 2 reports two numbers above the fold: 96.3% of boundaries clean and 91% of runs successful over the 200-task regression suite, which is the fixed set of tasks with agreed-good answers that the team re-runs after every change so that a drop shows up before customers find it. In the review where those numbers were first shown, someone asked what a clean boundary was, and the answer took four minutes and changed twice.

That is not a trivia question, because both numbers are about to be multiplied, budgeted against and argued from. Two runs from last week's regression set show why the identity everyone assumes — clean boundaries mean a good run — does not hold in either direction.

Three things you might mean by "clean", and they catch different failures#

In run 174, the orchestrator's brief for the infrastructure sector said "companies in the storage and compute segments" when the task actually asked for storage and compute security. The sector supervisor read that brief, decomposed it faithfully, dispatched three segment supervisors with correct sub-scopes, received three schema-valid summaries, and returned one schema-valid sector summary in which every numeric claim carried a source URL and an in-window as_of date. All eight boundaries under that sector were clean by any check the system performed. The run produced a confident, well-sourced brief about the wrong question, and a customer noticed.

In run 181, the segment supervisor for the applications-tooling segment returned a summary whose companies array had four entries when its scope had three. The contract check rejected it, the supervisor was re-invoked with the violation attached, and its second return was correct. The run finished on time and the brief was right. If your boundary counter increments on attempts, that boundary is a failure sitting in the numerator of the 96.3%, in a run that succeeded.

The four minutes in the review were spent discovering that three definitions were in play at once, held by different people, and that the dashboard was quietly reporting the cheapest one.

The three definitions get strictly stronger as you go down the table. The first checks only that the bytes are the right shape. The second adds the scope and the constraints, so it can catch a return that parses and is about the wrong companies. The third adds the parent's own experience of using it, so it can catch a return that is well-formed, on-scope, and too thin to write from.

Now read the third column, which is the one that carries the argument. It does not change. Three definitions, each strictly stronger than the last, and every one of them scores run 174 as clean. A column that stays constant while the thing beside it varies is telling you that the varying thing is not what the column measures, and that is exactly the situation here: strengthening the per-hop check does not move the verdict on run 174 at all, because the defect that ruined it never crossed a hop.

The reason is worth stating as a rule rather than as an observation about one run. A delegate can only be measured against the brief it received. A brief that misstates the task produces a delegate that satisfies it perfectly. So no per-boundary metric, at any of these three definitions, can see a defect that entered above the boundary being measured. That is the structural limit of boundary instrumentation, and it is worth stating plainly now, because the temptation for the next hundred pages of any incident review is to believe that a sufficiently good per-hop metric would have caught this.

What the three definitions do buy is the ability to tell the shapes of failure apart. Schema-valid alone catches malformed output and nothing else, which on this tree is about 40% of boundary failures — usually an extra company, a missing as_of, or a source_url that is a bare domain. Brief-satisfied additionally catches scope drift and dropped constraints, which is the other 60% and the more dangerous half, because a scope-drifted return is well-formed and therefore invisible to a schema check while being wrong in a way that propagates. Usable-downstream catches the residue: returns that are well-formed and on-scope but so thin that the parent has to re-invoke to get anything to write with.

Those two shares are worth reading against each other rather than separately. The cheapest definition covers the smaller share, and it covers the half whose failures announce themselves anyway — a payload that will not parse stops the parent immediately. The definition that costs a set comparison covers the larger share, and it covers the half that produces a well-formed document about the wrong companies. A team that instruments only the first definition has therefore bought the least useful 40% for almost all of its measurement effort, which is the opposite of the trade the cost column appears to offer.

Instrument all three, and record which attempt you are on#

Computing all three per boundary is cheap enough that choosing between them is a false economy. The important design decision is not which to record but what the denominator is: a record per attempt answers "how often does an invocation go wrong", a record per boundary answers "how often does this hop end up correct", and those are different rates the moment you retry anything.

from dataclasses import dataclass, asdict
import jsonschema, datetime, json

WINDOW_START = datetime.date(2026, 1, 1)

@dataclass
class BoundaryVerdict:
    run_id: str
    boundary_id: str
    parent_id: str | None
    attempt: int
    schema_valid: bool
    brief_satisfied: bool
    constraint_failures: list[str]
    usable_downstream: bool | None  # filled in once the parent's step completes

def judge_boundary(brief: DelegationBrief, payload: dict, attempt: int) -> BoundaryVerdict:
    failures: list[str] = []
    try:
        jsonschema.validate(payload, brief.return_schema)
        schema_valid = True
    except jsonschema.ValidationError as e:
        schema_valid = False
        failures.append(f"schema:{e.json_path}")

    returned = {c["name"] for c in payload.get("companies", [])}
    if returned != set(brief.scope):
        failures.append(f"scope:{sorted(returned ^ set(brief.scope))}")

    for company in payload.get("companies", []):
        for claim in company.get("claims", []):
            if not claim.get("source_url", "").startswith("https://"):
                failures.append(f"unsourced:{company['name']}")
            if datetime.date.fromisoformat(claim["as_of"]) < WINDOW_START:
                failures.append(f"stale:{company['name']}:{claim['as_of']}")

    return BoundaryVerdict(
        run_id=brief.run_id, boundary_id=brief.boundary_id,
        parent_id=brief.parent_id, attempt=attempt,
        schema_valid=schema_valid,
        brief_satisfied=schema_valid and not failures,
        constraint_failures=failures,
        usable_downstream=None,
    )

The scope check in the middle is the one worth reading closely, because it does the work of the second definition in one line. returned ^ set(brief.scope) is the symmetric difference of two sets: the names that are in one and not the other, in either direction. A supervisor that invented a company and a supervisor that dropped one both land in that set, so a single comparison catches an extra entry and a missing entry without needing to know which failure it is looking for.

The last line of the constructor is where the first two definitions are wired together, and the wiring is a decision worth seeing. brief_satisfied is computed as schema_valid and not failures, so a payload that fails the schema can never be brief-satisfied, whatever else is true of it. That makes the two definitions nested rather than independent: every brief-satisfied return is also schema-valid, and the second definition is the first plus more. The alternative — scoring the scope and constraint checks on a payload that did not parse cleanly — is available and was rejected, because the checks below read fields the schema is what guarantees. A number pulled out of a structure that failed validation is a number about nothing.

Every predicate in that function is deterministic, which is deliberate: a boundary check that itself calls a model adds a boundary. The checks cover the constraints that were written down in BRIEF_CONSTRAINTS, and they cover them exactly — which is another way of saying that a constraint you did not write in a checkable form is not being measured at any hop, no matter what the dashboard says.

Emitting one of these per attempt and one roll-up per boundary is what makes the two rates in the header separable:

def boundary_rate(verdicts, *, per: str) -> float:
    if per == "attempt":
        return mean(v.brief_satisfied for v in verdicts)
    by_boundary = defaultdict(list)
    for v in verdicts:
        by_boundary[(v.run_id, v.boundary_id)].append(v)
    return mean(any(v.brief_satisfied for v in vs) for vs in by_boundary.values())

The difference between the two branches is one word. The attempt branch averages brief_satisfied over every record. The boundary branch groups the records by hop and averages any(...) over each group. any is what encodes "this hop ended up correct": a hop counts as clean if at least one of its attempts was. Since a group with one clean attempt scores 1 under any and somewhere below 1 under a plain average whenever it also contains a failure, the per-boundary rate is greater than or equal to the per-attempt rate, always, and strictly greater as soon as any retry succeeds.

A hop has more states than pass and fail

The reason the two denominators diverge is that a boundary passes through several states between dispatch and the moment anyone can score it, and two of those states are only resolvable after the parent has done its own work.

Usable and Repaired are the states that cannot be decided at the boundary itself, which is why usable_downstream starts as None and is filled in by the parent emitting a consumed_without_repair event once its own step completes. Roughly 3% of Satisfied returns end up in Repaired on this tree, almost all of them summaries that are on-scope and well-formed but carry two claims where the parent needed six, and they are worth separating from failures because the fix is a richer brief rather than a stricter contract.

The type of that field is bool | None rather than bool, and the third value is carrying a real distinction. False means the parent tried to use the return and had to repair it. None means nobody has tried yet. Collapsing the two — defaulting the field to False on construction, which is the obvious thing to do — would file every boundary that has not yet been consumed alongside the ones that were consumed badly, and since a boundary is scored the moment it returns, that is nearly all of them. The consequence for the pipeline is that a boundary's verdict cannot be written once and left alone. The record is emitted incomplete and updated later, which is a small amount of extra machinery bought to keep "not known" from being reported as "known bad".

The Abandoned state is the other one worth pausing on, because of what it licenses. Every other terminal state leaves the run able to continue in some form. Abandoned is the state in which a hop has spent its budget and produced nothing usable, so the run fails and the failure has a boundary id attached to it. That makes it the only boundary outcome that reliably answers "which hop did this" — and the residual is exactly the failures that never reach it, which are the runs where every hop returned something the contract accepted.

The retry budget is one attempt, and that number is a policy rather than a discovery. It is worth separating the two kinds of thing in that sentence. That a hop can be retried at all is a requirement of the arrangement: the delegate returned something the contract rejects, so the parent either retries, gives up, or accepts a violation. How many times it retries is a decision somebody made and could unmake. Two retries would lift the per-boundary rate by about a point and a half on the observed data and add roughly 9 seconds at p95 — the 95th percentile, the figure that 95 runs in 100 come in under, which is where a latency promise is usually written — to any run that uses them, which is a trade the interactive surface cannot make. The Abandoned state is what the run sees when the budget is spent, and it is the only boundary outcome that reliably produces a run failure attributable to a specific hop.

Every predicate in judge_boundary is deterministic for a reason beyond speed. A model-based judge at each boundary would catch things the predicates cannot — a summary that is on-scope and well-sourced but subtly misstates a company's positioning — at a cost of about 3.4 seconds and 8,000 input tokens per boundary. Both of those are per-boundary quantities and the tree runs 26 boundaries, so the run-level cost is each of them multiplied by that count: the seconds come to 88 and the tokens, priced at the model's input rate, come to $1.04 per run. It would also be a delegation: another agent, another brief, another return that can be wrong in the same correlated way as the thing it is judging. Deterministic checks cost about 2 milliseconds and cannot introduce a new failure mode, so they run everywhere, and the model-based judge runs once, offline, on a sample of the regression suite, where its own errors can be inspected rather than compounded.

Why 96.3% per boundary and 91% per run are both true#

The naive expectation is that a run succeeds when every boundary on its critical path is clean, the critical path being the chain of hops a single finding has to cross to reach the final answer. That expectation has an arithmetic behind it worth writing out, because the arithmetic is where the assumption hides. If each hop is clean with probability p, and whether one hop is clean tells you nothing about whether another is, then the chance that all d hops on a path are clean is p multiplied by itself d times. So a three-hop path at 96.3% should succeed 0.963³ ≈ 89.3% of the time, and the observed 91% ought to be a measurement error.

It is not. Two effects push the run rate off the product in opposite directions, and on this tree they nearly cancel, which is the most dangerous arrangement because it makes the arithmetic look like it works.

The first effect pushes upward. The 96.3% is an attempt-level number, and the segment supervisors retry once on a contract violation. Over the 200-run suite, 61 boundary attempts failed brief_satisfied; 44 of them were retried and 38 of those retries came back clean. Recomputed per boundary rather than per attempt, the rate is 99.1%, and 0.991³ ≈ 97.3%. A boundary that hiccups and recovers is a failed event inside a successful hop, and treating those as the same thing understates the tree by eight points.

The second effect pushes downward, and it is the one that explains why 97.3% of paths clean does not produce 97.3% of runs clean. The mismatch is between two different counts that the word "path" hides. A critical path is one finding's route to the top, and on this tree that is three hops. A run is the whole tree, and the whole tree is twenty-six hops, because every company's finding takes its own three-hop route and there are eighteen companies with supervisors stacked above them. Raising a per-hop rate to the third power answers a question about one finding. Nobody ships one finding.

A run succeeds only if the whole tree does, and 26 boundaries execute, not 3 — a defect anywhere in any of the six segments reaches the final brief. More consequentially, the failures are not spread evenly across boundaries. Run 174 is the pattern: one bad brief at the top, boundaries all scoring clean, and a ruined run that no boundary metric attributed to anything. Eleven of the 18 failed runs in the suite have that shape, and in those runs the mean number of boundaries scoring not-clean is 0.4.

So the reconciliation is not that one number is wrong. It is that the per-boundary rate and the end-to-end rate have different denominators and different failure populations, and the product of the first is a prediction of the second only under an assumption nobody checked: that boundaries fail independently, and that failure at a boundary is the only way a run goes wrong.

What that leaves the compounding arithmetic licensed to say is narrower than it looks. p^d licenses a prediction about the runs whose only way of going wrong is a hop going wrong, and it licenses nothing else. Eleven runs out of 200 falsify the second assumption on their own, and they falsify it with a signature: well-formed output, zero constraint violations, wrong question answered. That signature is now visible in the verdict stream precisely because constraint_failures came back empty on every hop of a run that failed. The residual is the eleven, and no refinement of p reaches them.

A per-boundary rate is not a service level indicator#

A per-boundary success rate is not a service level indicator, or SLI — the headline number a team publishes for how well a service is doing its job, such as the share of requests it serves successfully — and it should not be presented next to one.

The reason is an analogy that fails, and the analogy is worth naming because it is where the habit comes from. In a distributed system, availability composes across a request path: if each service on the path is up independently with some probability, the chance the whole path serves the request is the product of those probabilities. That is the same p^d shape, and importing it here is tempting because a delegation tree also looks like a chain of dependencies.

The place it breaks is specific. An HTTP dependency either serves the request or does not, and there is no third option in which it serves the request perfectly while corrupting it. A delegation boundary has exactly that third option: it can serve a request perfectly and still be handing down a corrupted premise, which has no analogue in an HTTP dependency graph. The habit of reading p per hop and reporting p^d end-to-end is imported from a world where the third option does not exist, and the import is where the error enters.

Nor is brief_satisfied a measure of correctness. It is a measure of fidelity to the parent, and a tree that scores 100% on it is a tree in which every delegate did exactly what it was told. The belief that a high fidelity score is a defence is the tempting one here, and it is tempting because fidelity is genuinely the thing you want from a delegate — a delegate that improvises is worse than one that obeys. What makes it a trap is that the score stays pinned at the top while the property that decides the run, whether what the delegate was told was right, moves freely underneath it.

Whether what a delegate was told was right is a property of the top of the tree, and it is inherited, unchanged and unexamined, by every layer beneath. That inheritance is what makes the independence assumption false, and it is why the failures it produces do not arrive as an even trickle of one-per-hop attrition. They arrive together.