A retry cap is only as good as its exhaustion path

A repair budget is the ceiling on how much work a single request may spend trying to turn an invalid response into a valid one. Nearly every pipeline acquires one after an incident, and nearly every one expresses it the same way:

MAX_REPAIRS = 3

def extract(doc: str) -> LineItems | None:
    messages = build_request(doc)
    for attempt in range(MAX_REPAIRS + 1):
        text = call_model(messages)
        try:
            return LineItems.model_validate_json(text)
        except ValidationError as err:
            messages = repair_messages(messages, text, err)
    return None

That does the job it was written for. What it does not do is say what a None means to the caller, and the answer to that question is where the cost of a cap actually lands. This chapter is about the two halves that the two-line version leaves out: what unit the budget should be counted in, and what the request that exhausts it becomes.

Attempts are not the only ceiling that matters#

Three attempts on an 8,000-token document is not "three times the cost", because each repair carries the full conversation forward. Attempt one sends about 8,300 input tokens. Attempt two sends those plus the invalid output plus the errors, roughly 9,900. Attempt three is near 11,500. A request that exhausts a three-repair cap has consumed about 44,000 input tokens against the 8,300 of a clean run, so the worst case is five and a half times the best case rather than four.

Each increment there has a name, and knowing the name is what lets you predict the figure for a schema you have not measured. The step from the first attempt to the second is the invalid output plus the serialised Pydantic errors appended beside it, which on this schema came to roughly a fifth of the original request. The step from the second to the third is another copy of that pair, because repair_messages appends to the message list rather than replacing what is in it, so attempt three carries both failed exchanges and attempt four carries all three. The sequence is therefore not four copies of 8,300 but 8,300, then that plus one repair turn, then plus two, then plus three, and a run of four calls whose sizes rise by a roughly constant step sums to well over four times the first of them. The instinct that a cap of three costs at most four times a success is the arithmetic of a stateless retry, where every attempt is a copy of the last. A repair loop is not stateless by construction — the accumulated state is the entire reason it beats a resample — so the multiplier always exceeds the attempt count, and by how much depends on the size of a failed attempt relative to the request that produced it.

Latency compounds the same way and usually binds first. In one extraction service the first attempt ran at a p50 of 2.1 seconds and each repair added between 2.5 and 3.2 seconds, so a fully exhausted request landed somewhere near 11 seconds — against an API gateway timeout of 10. The effective cap was two repairs, and the third one was pure spend on a connection nobody was listening to any more. Discovering that from a p99 graph took a week; reading it off the arithmetic would have taken a minute.

The minute in question is one addition. Start at the first attempt's 2.1 seconds and add repairs at up to 3.2 each. One repair lands a little over five seconds, two land a little over eight, and three land above eleven, which is past the gateway's ten. That is where "the effective cap was two" comes from: the third repair begins after the deadline that governs the request has already passed, so whatever it returns arrives at a socket the gateway has closed. It still costs tokens, it still occupies a worker, and it cannot change the outcome. A budget expressed in attempts has no way to notice this, because attempts and seconds are different units and nothing in range(MAX_REPAIRS + 1) knows what time it is.

The consequence is that the ceiling wants to be expressed in whichever unit actually runs out, and usually in more than one:

@dataclass(frozen=True)
class RepairBudget:
    max_attempts: int = 3
    max_total_tokens: int = 40_000
    deadline: float = 8.0          # seconds, wall clock, from request start

    def exhausted(self, attempts: int, tokens: int, elapsed: float) -> str | None:
        if attempts >= self.max_attempts:
            return "attempts"
        if tokens >= self.max_total_tokens:
            return "tokens"
        if elapsed >= self.deadline:
            return "deadline"
        return None

Returning the reason rather than a boolean costs nothing and is worth a great deal later, because "exhausted on deadline" and "exhausted on attempts" are different failures with different fixes, and a single exhausted counter merges them the same way the previous chapter's single validation_failures counter merged two unrelated populations.

Where the number three comes from#

The cap is usually three because three is what people write. The recovery curve argues for a smaller number. Replaying 5,000 logged first-attempt failures through the structured-feedback repair of chapter 1, and letting them run to four attempts rather than stopping:

The first two percentage columns are counted against different denominators, and reading them as though they shared one is the commonest way to misread a recovery curve. The first column is conditional: 41% at attempt three means 41% of the requests still failing after attempt two, not 41% of the original pool. Walk a hundred original failures through it. Attempt two recovers 89 of them and leaves eleven. Attempt three recovers 41% of those eleven, which is between four and five, and that is what takes the cumulative figure to 93.5%. Attempt four recovers 9% of the six and a half still left, which is a little over half of one request in a hundred, and that is the step to 94.1%. Written out that way the fourth attempt's headline 9% stops looking like a share of anything worth spending on.

The marginal column falls out of the same walk, and it is the one that decides the cap. Everybody still in the pool pays for the attempt; only the ones who recover get anything back for it, so the cost per recovery is the whole pool's spend divided by the survivors of that round. At attempt two that is a hundred requests at the second attempt's 9,900 input tokens each, shared among 89 successes, which is the 11,100 in the table. At attempt four the pool has shrunk to six and a half requests but each is now carrying three failed exchanges in its context, and the division is by half a success, which is how a per-recovery cost reaches six figures. The curve does not flatten because the model tires. It flattens because the denominator collapses faster than the numerator does.

The first repair does nearly all the work. The second buys four and a half points at more than twice the marginal token cost, which is often still worth it. The fourth attempt recovered 0.6% of the original failures at roughly 190,000 input tokens per success, because by then every request in the pool is carrying three failed attempts in its context.

The reason the curve flattens so hard is that the failures surviving two repairs are usually a different kind of failure. They are not formatting accidents; they are requests where the document does not contain what the schema demands, or the constraint contradicts itself, or the OCR text is unreadable at the relevant line. More attempts do not help, because the model is being asked for something that is not there, and it will either keep refusing or start inventing — the failure mode Pithy Cyborg's The LLM Retry Loop That Looks Like Progress and Does Nothing is written about. Setting the cap where the curve flattens is partly a cost decision and partly a way of stopping before the loop starts producing confident fiction.

There is a second-order reason to keep it low. Each repair appends the previous invalid output to the context, so by attempt four the model is conditioning on three examples of the wrong answer, written in its own voice. That is not neutral. On the replay, attempt-four responses repeated a structural mistake from attempt two roughly twice as often as attempt-two responses repeated a mistake from attempt one. If you want more than two repairs, it is generally better to discard the failed turns and start clean with a modified prompt than to keep stacking them.

There is a mechanism under that observation and it is the same one that makes the repair loop work at all. A model predicts each token from everything above it, and the repair message deliberately places the previous attempts above it as assistant turns, because that shape is what makes a correction legible as a correction. The price of the shape is that the failed answers become part of the pattern being continued, written in the model's own voice, and a context in which a field has been rendered the same wrong way three times is a context whose most likely continuation renders it that way once more. The error feedback argues against the pattern while the pattern argues for itself, and each round adds another vote to one side only. It also explains the character of late failures: by attempt four the model is being coherent with what it was handed, and coherence with three wrong answers is exactly what next-token prediction rewards.

The three ways a request can end without a valid response#

Three of those terminal states are ways of not succeeding, and they are not interchangeable.

Rejection is the default worth defending, because it is the only one of the three that cannot be misread. A 422 — the HTTP status for a request that was understood and could not be processed — carrying {"error": "extraction_failed", "reason": "attempts", "attempts": 3} is unambiguous to every consumer that has ever existed, and a caller that ignores it is making a choice you can point at in a review.

Escalation is the right answer whenever the failure is one more model call cannot fix, and the clearest case is the one from the previous chapter: a refund decision that parses cleanly, validates against the schema, and then fails the check because the amount exceeds that customer's remaining balance. There is no repair to attempt, because the model was never given the balance and a second attempt draws from the same distribution over the same context. The loop should not spend an attempt on it at all. It should stop on the first violation and hand the decision to a person, and the fact that the request still had two attempts in the budget is irrelevant — a state-dependent rule failure short-circuits the whole state machine rather than consuming it.

The reason the third outcome has to be decided rather than defaulted is that the exhausted state acquires one either way, and the only open question is who wrote it. A function typed LineItems | None has already answered: exhaustion means None, and None means whatever the first caller to meet it decided under time pressure. The same answer arrives through an except: pass in a worker, through an or 0 in a template, through a nullable column with a default, and through a client library that reads an empty body as an empty object. Not one of those is a position anyone would defend in review if it were written out as prose — an invoice we could not read is an invoice for nothing — and every one of them is shorter to type than the alternative. Deciding the outcome explicitly is not extra work on top of leaving it undecided. It is the same decision, made once by somebody who knows what the request was for, instead of many times by whoever is nearest.

Greyed-out fields are not a contract#

A document-extraction endpoint behind a finance UI took a different route. Its cap was two attempts, and on exhaustion it returned HTTP 200 with whichever fields had validated, plus a flag:

{
  "invoice_number": "DE-2024-8871",
  "issue_date": "2024-11-03",
  "total": null,
  "currency": null,
  "line_items": [],
  "partial": true
}

The UI rendered the null fields greyed out with a small "could not be extracted" note, and product was happy, because a user looking at that screen can see three of five fields, correct the rest by hand, and move on. That is a real improvement over an error page, and for a human reading a screen the design is defensible.

It stops being defensible the moment anything other than a human reads the response, and endpoints acquire non-human consumers without anyone deciding they should. A 200 is the signal every HTTP client, every retry library, every cache, every metrics pipeline and every downstream service treats as "this worked". A nightly reconciliation job that pulls the same endpoint sees a well-formed document with total: null; a content delivery network or a client-side cache stores the partial with the same freshness as a complete record; a service level objective computed from status codes reports 100% success on a day when 8% of extractions produced nothing usable; and the aggregation service that sums total across invoices adds null, or adds zero, or throws in a batch job at 04:00. The greyed-out styling communicated the uncertainty to exactly one consumer, and the styling lives in a different repository from the contract.

What makes that a contract violation rather than a styling preference is one property of HTTP, and it is worth stating because the list above otherwise reads as a set of unrelated bugs. HTTP defines a small number of fields every intermediary agrees on, and the status code is the load-bearing one: a cache decides storability from it, a retry library decides retryability from it, a service level objective is computed from it, and a generated client decides whether to raise from it. None of those components parses the body, and none of them can, because the body's shape is your application's business while theirs is the protocol's. A 200 is therefore not a description a downstream reader may choose to interpret. It is the one field every reader has already interpreted before a line of your code runs, so a "partial": true sitting inside a body they never open cannot reach any of them. The greyed-out treatment is a layer below that again, in a repository the nightly reconciliation job has never heard of.

Two things would move the design. If the response is complete-or-nothing for any consumer — a write path, a downstream sum, an export — partial completion is wrong and the answer is rejection with a reason code. If the endpoint is genuinely human-only and expected to stay that way, partial completion earns its place, and it still has to be expressed in the payload rather than in the CSS: a distinct status (206 Partial Content, or a 200 whose body has a required status: "partial" discriminator that a typed client cannot ignore), a machine-readable list of which fields are missing and why, and no null standing in for a value. A field that failed extraction should be absent or carry an explicit {"value": null, "reason": "not_found_in_document"}, because a bare null is indistinguishable from a value that was genuinely empty.

Either way, four things want instrumenting, and they are worth having before the argument rather than after it. Exhaustion rate, split by the reason the budget ran out. Which fields are missing when a partial is returned, because a single field accounting for 70% of partials is a prompt or schema problem with an obvious fix. What users do next with a partial — whether they complete it by hand, abandon it, or re-submit — since a partial that is abandoned 80% of the time is a rejection with worse ergonomics. And the count of non-browser user agents hitting the endpoint, which is the number that tells you whether the human-only assumption is still true.

A cap is not a circuit breaker#

The two get filed together and they protect different things. A circuit breaker — the pattern covered in topic 2.11, on error handling, which watches a dependency's failure rate and stops sending it traffic once that rate crosses a line, letting calls fail instantly until a probe says the dependency is healthy again — is per dependency, and it protects the model provider, and you, from a thousand simultaneous failures. The cap is per request, and it protects the caller from one pathological input. When every request is failing, a cap of three means each of them fails three times and your bill triples while the incident lasts. The two mechanisms compose, and neither substitutes for the other.

A cap also does not improve anything. It bounds the blast radius of a failure that has already happened, and the number of requests reaching the cap is a fact about your inputs, your schema and your model that the cap itself cannot change. Choosing the number is a smaller decision than choosing what the number means: the budget is worth a paragraph of arithmetic, and the exhaustion path is worth a design review, because it is the only part of the loop a customer ever sees.

That number — how often a request runs out of budget — is also the most useful thing this loop produces, and so far nothing in the design has been reporting it.