Error handling

Read these first

• Streaming

What you will know

• Distinguish a rate limit from an overload error from a bad request

  Providers may expose different status codes and headers for these cases, so classification must use the documented response semantics rather than status alone.

• Implement exponential backoff with jitter, and say why jitter matters

  Randomized delays spread synchronized clients across time; cap the delay and attempts so recovery does not become an unbounded queue.

• Make a retry safe when the first attempt may have partially succeeded

  An idempotency key lets the provider recognize a duplicate submission, while non-idempotent operations need reconciliation before replay.

• Choose a timeout for a streaming call, and a different one for a batch call

  Streaming needs separate time-to-first-token and inter-chunk limits, while batch work needs a bounded completion deadline tied to its job and queue policy.

• Explain why retrying a 400 is always wrong

  Repeating a malformed payload cannot repair it and consumes quota, although provider-specific 400 responses still require checking the documented error semantics.

Chapters

1. 1

   Classifying provider error responses

   A failed call to a model provider belongs to one of three classes that live in three different systems: a 429 is your own allowance, a 529 is the provider's capacity, and a 400 is your payload. Status codes alone will not separate them, so classification branches on the documented error type and the rate-limit headers underneath it. This chapter defines the three constraints, shows what the response body and headers report, and works through what a single undifferentiated error counter hides.

   Read chapter
2. 2

   The blast radius of a retry

   A malformed tool schema cannot repair itself between attempts, yet a wrapper that retries every non-2xx will try five times anyway. The token cost is the small part: rejected attempts still draw on the request-rate budget shared with healthy traffic, and a 200 ms failure becomes a 15-second one, so a worker pool saturates and endpoints that had no bug at all start timing out. This chapter measures both shared resources and derives the narrow retry policy that survives the argument.

   Read chapter
3. 3

   Exponential backoff and jitter

   Exponential backoff was specified with a random draw inside it, and the HTTP client libraries that dropped the draw reproduce the burst the mechanism was invented to break. This chapter compares fixed, full, equal and decorrelated jitter against the same fleet, derives the base delay from fleet size and allowance, and sets the two bounds — a ceiling on the delay and a cap on attempts — that keep a recovery from becoming an unbounded queue.

   Read chapter
4. 4

   Retry-After and rate-limit headers

   A rate-limit rejection arrives carrying the provider's own statement of when capacity returns, and a schedule computed from an attempt counter overrides it in both directions at once. This chapter takes apart the two budgets hiding behind one status code, reads the reset headers as a diagnosis, parses them safely across clock skew and both Retry-After formats, and shows why a reported reset still needs a positive random offset on top.

   Read chapter
5. 5

   Timeouts for streaming and batch calls

   A single total timeout answers one question — has the whole call taken too long — and a streamed generation has a third outcome it cannot see: partial output followed by silence. This chapter separates the phased transport timeouts from the content-level clocks that keep-alive pings defeat, sets a time-to-first-token and inter-delta deadline from a measured latency histogram, and derives the per-item and job-level deadlines a bulk path needs instead.

   Read chapter
6. 6

   Circuit breakers and load shedding

   Classify, back off, honour the reset, bound the deadline: every one of those policies is per-request and memoryless, so a fleet applying them through a twenty-minute degradation pays the cost of discovering the outage once per call. This chapter builds the shared state that fixes it — a breaker with a volume floor, an exponential cool-down and half-open probes — and shows why an interactive path sheds the work while a bulk path defers it.

   Read chapter
7. 7

   Idempotency keys and replay safety

   A client timeout is a decision made on the client: the generation you abandoned keeps decoding and is billed, so two retries plus a success is three completions charged for one answer delivered. An idempotency key lets the provider recognise the duplicate and return the original outcome. This chapter covers where the key must be generated, the streaming case where a key does not help because output has already been rendered, and the tool call whose effect no header can recall.

   Read chapter