A failed contract check needs a policy you wrote in advance

The gates went in on the Thursday after an incident in which a provider's model update broke a nightly document pipeline, and the run promptly went red at 00:04 with a ValidationError on document 41 of 12,000. Whoever was on call did the obvious thing, which was to wrap the stage in a try and continue past anything that failed, and the run went green again. Eleven weeks later the claims team asked why 2,847 policies from the previous quarter had no record at all.

The gate had been doing its job the entire time, and that is the sentence to sit with, because it is where the false comfort lives. Every night the check fired on exactly the records whose payloads were wrong, and every night the run finished green. What was being held constant was the correctness of the check. What varied underneath it, silently, was the number of policies that never reached the database. A green run is a statement about the process reaching its end, not about the records arriving, and the two came apart the moment a continue was inserted between them. There was no decision about what a failed gate should cause, so the decision got made at 00:11 by a tired person optimising for the alert going away.

There are only four things a run can do when a stage's output fails its contract, and every real pipeline is some assignment of those four to its stages. The stage can be re-run, with the validation error fed back so the model can correct itself. A substitute output can be put in its place, accepting a known reduction in quality. The whole run can stop. Or the record can be set aside — dead-lettered, meaning parked in a queue of records the pipeline could not finish, for a person to deal with later — while the rest of the batch proceeds. None of these is a default. They differ in cost, in what they leave behind, and in what they assume about the stage that just failed.

The four exits, and what each one assumes#

The two columns that decide a choice are the last two, and they are the ones a table like this is usually read past. "Requires" is a precondition: pick a policy whose requirement your stage does not meet and the policy does not work, however well it fits the failure. "Leaves behind" is the residue the rest of the system has to cope with, and it is where the four differ most — nothing, a marked record, a half-written batch, a queue.

The last column of the dead-letter row is the one that fails in practice. A dead-letter queue with no owner is a continue statement with extra infrastructure, and the 2,847 missing policies would have been 2,847 rows nobody looked at instead of 2,847 silent skips. What makes the difference is an alarm on queue depth and age, not the queue itself. That is the same false comfort as the green run, one layer up: the queue makes the skipped records visible in principle, and visibility in principle is not a person noticing.

Retry deserves a budget rather than a constant, because its returns fall off sharply. On the policy pipeline, instrumenting a month of gate failures on stage two gave the following: of 100 contract failures, appending the ValidationError message and re-running recovered 71 on the first retry, a further 9 on the second, and 1 on the third. The third attempt cost 12% of the retry token spend to recover 1% of the failures, so the budget was set at two and everything still failing went to the dead-letter queue.

The rule that produced that budget is worth stating in general, because the numbers will be different on your stages and the shape of the decision will not. Each attempt costs roughly the same as the last, since it re-runs the same stage on the same record, so the cost of the n-th attempt is close to constant while its yield is whatever the previous attempts left unfixed. Set the budget at the point where one more attempt is spending a similar share of the retry bill for a much smaller share of the recoveries, which on this stage was the third attempt: a tenth or so of the money for a hundredth of the failures.

That distribution is worth measuring on your own stages rather than assuming, but its shape is typical: models fix their own format errors readily and fix their own comprehension errors almost never, so the tail is not made of things another attempt will help with. The asymmetry has a mechanism. A format failure gives the validator something exact to say — a missing field, a value outside an enumeration, a string that does not match a pattern — and that message, appended to the prompt, is a precise instruction the model can act on. A comprehension failure produces a payload that validates, so the validator has nothing to report at all, and a record only reaches a third retry when the first two attempts failed on a problem the error message never described. Re-running with the same input and the same instruction samples again from much the same distribution, which is why the third attempt recovers almost nothing.

class GateFailure(Exception):
    def __init__(self, stage: str, payload: dict, error: str):
        self.stage, self.payload, self.error = stage, payload, error

def run_stage(stage: Stage, record: Record, run_id: str) -> StageResult:
    attempts, feedback = 0, None
    while True:
        try:
            out = stage.invoke(record, correction=feedback)
            return StageResult(ok=True, value=stage.contract.validate(out))
        except ValidationError as e:
            attempts += 1
            ledger.note(run_id, record.id, stage.name, "contract_failed",
                        attempt=attempts, error=str(e))
            if attempts <= stage.retry_budget and stage.side_effects_free:
                feedback = (
                    "Your previous output failed validation with:\n"
                    f"{e}\nReturn a corrected object. Change nothing else."
                )
                continue
            return stage.on_exhausted(record, e)   # degrade | dead_letter | abort

The condition stage.side_effects_free is doing more work than its length suggests. Re-running a stage is safe when the stage is a pure function of its input, and stages stop being pure the moment somebody adds a write. A stage that files a claim and then fails its own output contract has already filed the claim; retrying it files a second one. This is ordinary idempotency work, and the usual remedy applies — derive a key from the record and the stage (policy_id + ":" + stage_name), pass it to whatever the stage writes to, and let the second write be a no-op — but it has to be decided per stage rather than for the chain, because in most chains only one or two stages touch anything outside the process.

The reason the key has to be derived from the record rather than generated per attempt is the whole of the mechanism. The receiving system cannot tell two attempts apart by looking at their contents, since a retry sends much the same thing; what it can do is recognise a value it has already accepted. A key computed from the record identifier and the stage name has the property that every attempt at that stage on that record produces the same value, and no other work in the pipeline produces it, so the second write is recognisably a repeat rather than a new claim.

Retry-on-contract-failure also sits on top of a layer that is not the same thing and should not share a budget. Transport failures — HTTP 429, a socket timeout, a 529 from an overloaded endpoint — are handled with exponential backoff at the client, they carry no correction message, and their budgets are usually larger. The two differ in what a repeat is expected to change: a transport retry is waiting for a condition on someone else's side to pass and needs no new information, while a contract retry is asking the model to produce something different and is useless without the error message that says what was wrong. Merging the two produces a stage that responds to rate limiting by rewriting its prompt, which wastes tokens and confuses the log. Keep the backoff inside stage.invoke and the contract retry outside it.

What the batch leaves behind when it stops halfway#

The abort row in the table has the widest blast radius, because the interesting question is not stopping, it is the state of the world at the moment you stopped. A run that dies at document 7,400 of 12,000 has written 7,400 claim records. Whether that is fine or a disaster depends on a property you chose earlier, possibly without noticing: whether the unit of atomicity is the record or the batch.

Per-record atomicity is the easier design and the right default for batch document work. Each record runs the full chain, writes once at the end, and is marked complete in a ledger keyed by run and record. A restart reads the ledger, skips the 7,400 completed records, and costs nothing but the time already spent. Batch atomicity — stage all 12,000 results and commit them together — is what you need when downstream consumers require a consistent snapshot, such as a nightly export where a half-loaded table is worse than yesterday's table. It costs more, because you hold the results somewhere until the end and you throw away the whole run on a late failure.

Degradation deserves a warning of its own, because it is the policy most likely to be regretted. Substituting a fallback value keeps the batch moving and produces a record that is indistinguishable from a good one unless somebody marked it, and the marking is the entire safety of the mechanism. A degraded record needs a field saying which stage degraded and why, downstream consumers need to be able to filter on it, and something needs to watch the degradation rate the way it watches the error rate. Without that, "risk band unknown" quietly becomes 14% of the table over six months, a dashboard somewhere starts averaging over it, and the pipeline has been lying at a slowly increasing rate with nothing in the logs to show for it.

The ledger is the cheap part and the part teams skip. Three columns — run_id, record_id, stage, plus a status and a timestamp — turn "the run failed, start again" into "the run failed at stage three on 12 records, here they are". It is also what makes the retry budget auditable after the fact, since a stage whose retry rate drifts from 4% to 19% over a month is telling you something about the model or the input distribution long before the end-to-end score moves. That early warning is a property of where the measurement sits: a retry happens at one boundary and is attributable to one stage, while an end-to-end score is a single number over the whole chain, so a stage degrading from a twenty-fifth to a fifth of its records needing a second attempt has to get considerably worse before it moves the composed figure at all.

Once these choices are made, they belong in the stage definition rather than scattered through the runner, so that reading the pipeline tells you its failure behaviour:

PIPELINE = [
    Stage("extract",  contract=EntitySet,   retry_budget=2,
          side_effects_free=True,  on_exhausted=dead_letter),
    Stage("facts",    contract=PolicyFacts, retry_budget=2,
          side_effects_free=True,  on_exhausted=dead_letter),
    Stage("classify", contract=RiskBand,    retry_budget=1,
          side_effects_free=True,  on_exhausted=degrade(RiskBand.unknown)),
    Stage("file",     contract=FiledClaim,  retry_budget=0,
          side_effects_free=False, on_exhausted=abort_run),
]

Reading down that list gives you the pipeline's whole failure story. Extraction and fact-building are retried and then set aside; classification degrades to unknown rather than blocking a record, because a claim with an unknown risk band is still useful and gets picked up by a downstream review; filing is never retried automatically and stops the run, because it writes and because a systematic filing failure at 3 a.m. is not something to spend 12,000 attempts discovering. Notice that the two settings move together down the list: as side_effects_free goes from true to false, the retry budget goes from two to zero, which is the precondition column of the table showing up as code.

The recovery a chain cannot express#

Everything above is anticipated recovery. Someone sat down, enumerated the ways each stage can fail its contract, and wrote a policy for each. That covers a great deal, and it is worth being precise that it covers retrying, because there is a widespread claim that a fixed chain cannot recover from a failure and needs an agent loop for that. Retrying a link is a control-flow decision made in code at a boundary. The while loop in run_stage is thirteen lines and it re-runs a stage with the error appended, which is exactly the recovery that gets attributed to agency.

The honest edge of the pattern is elsewhere. A chain's response set is finite and fixed at design time, so when a failure arrives that nobody enumerated, the chain picks the closest policy it has, which is usually the dead-letter queue. A tool-using loop can read the content of the failure and choose a next action that was never enumerated — fetching the original PDF page when the extracted text looks truncated, querying a reference table to disambiguate two policies with the same number, asking for the document in a different format. That is a real capability and it is the thing loops add over chains, and the price is that you no longer know in advance what the system will do, which is exactly the property that made the four-row table above writable.

Whether that trade is worth taking depends on the task, and there is a class of work where it plainly is not, because the operations and their order are fixed by the data rather than by anything the model could reason about. Recognising that shape is what lets you stop having this argument on every design review.