An untested substitute can breach what the outage does not

The primary provider has been returning 529s — the status code it sends when its own capacity is overloaded — on 78% of requests since 09:04. The status page says "elevated error rates, investigating" and has said so for thirty-five minutes with no ETA. The circuit breaker opened at 09:07 exactly as designed, so requests now fail in 40 milliseconds instead of after three backoffs, and the customer-visible error rate is 71%. Support has fourteen tickets. At 09:41 someone pastes the runbook into the incident channel, and step 4 says, in full: "If the primary is unavailable for more than 15 minutes, fail over to Provider B."

Provider B is configured. There is an account, an API key in the vault, a client wrapper, and a chain constant that lists it second. It has carried zero production requests in the eight months since it was set up. The account was created by whoever was doing the integration spike, in the US region, with the provider's default 30-day prompt retention, because that is what you get when you click through the signup. Roughly 30% of current volume comes from tenants whose contracts specify EU-only processing and zero retention by the subprocessor, which is the contractual name for any third party you hand their data to in order to serve them.

The mechanics of the breaker, the retry policy, and what makes two providers behaviourally equivalent were covered in topic 2.21 and are assumed here. What that topic could not settle is the decision in front of the incident commander at 09:41, which is not about mechanism at all.

The failover is a data-processing decision wearing an availability costume#

Sending a request to Provider B means sending the contract text in that request to a US-based processor that will retain it for thirty days. For the 70% of tenants whose contracts do not restrict this, that is an ordinary vendor call. For the other 30%, it is a processing operation their data processing agreement — the DPA, the contract term that fixes where their data may be handled and by whom — forbids, performed without instruction, on data that will then sit in a third party's storage for a month.

There is no undo. A configuration change can be reverted in four seconds, as chapter 3's pin showed, and a deployment can be rolled back, but a byte that has crossed a network boundary into someone else's retention window has crossed it. The best available remediation is a deletion request to the provider, which is a support ticket with a service-level of days, produces an attestation rather than proof, and does not change the fact that the processing happened. The obligation that follows is a notification to each affected customer under their DPA — typically within 24 to 72 hours — describing what was sent where, plus, depending on jurisdiction and data category, a regulator notification.

Put those two failures side by side and the asymmetry is stark. The outage is 71% errors for an unknown number of additional minutes, fully recoverable, with no artefact left behind once the provider recovers. The failover, for that 30% of traffic, is a disclosure letter to your largest enterprise customers, a security-review re-opening at renewal, and a permanent entry in their vendor file. The first is an availability incident; the second is the kind that gets discussed at a board meeting. Availability is one obligation among several, and it is not automatically the one that wins.

It is worth putting rough numbers on both sides, because "contractual exposure" sounds abstract next to a very concrete 71% error rate. This platform's enterprise service level agreement, the SLA, promises 99.5% monthly availability with a 10% service credit below that threshold. A two-hour total outage consumes most of the month's error budget — the roughly 3.6 hours of unavailability that 99.5% permits before credits are owed — and, against a $2.1M enterprise book, exposes something like $17,000 in credits. The breach side has no comparable ceiling: notification letters to twelve enterprise customers, a re-opened security review at each of their renewals, forty-odd hours of legal and engineering time assembling the disclosure, and a regulator's discretion on top. The two figures are not close, and they point the same way every time.

Ten minutes, and what gets refused inside them#

The decision splits the traffic rather than the difference. Tenants whose contracts permit US processing fail over; tenants whose contracts do not fail closed, which means their requests are refused outright rather than served by a route that is not permitted, with an explicit error that says why. That requires knowing which is which in under ten minutes, which in turn requires the classification to already exist — a residency_class on the tenant record, populated when the contract was signed. If it does not exist, the only honest move at 09:41 is to fail everything closed, because "we believe most of these are fine" is not a basis for an irreversible processing decision.

Two further things get checked before any traffic moves. The secondary's rate limits are the first: a cold account on a starter tier might have 40,000 tokens per minute, and this system's eligible traffic runs at about 380,000. Pointing everything at it converts an outage into a slower outage plus a wave of 429s, the rate-limit rejections a provider returns once you exceed the quota on your account, and burns the goodwill of the customers who were still working. Ramp in steps — 5%, then 25%, then the rest — and watch the secondary's error rate between steps. The second is the degraded semantics: which features you are turning off for the duration. If the summariser's obligations extract depends on strict tool-call schemas and Provider B returns tool arguments as a JSON string rather than an object, the extract path stays off and the review pane runs without it, and that is a decision to make deliberately at 09:45 rather than discover at 10:20 from a parse-error alert.

What gets refused is short and worth saying out loud in the channel, because incident channels generate pressure toward action of any kind. No traffic from a tenant whose residency class is unknown. No 100%-in-one-step cutover to a provider that has never served a request. No permanent configuration change made under time pressure — the failover is expressed as a temporary override with an expiry, so that forgetting to undo it is not a possible outcome. And no decision to process restricted data taken by the engineers in the channel; that call belongs to whoever owns the DPAs, and if they cannot be reached in ten minutes, the answer defaults to no.

The call itself has a shape worth rehearsing before you need it. The commander states the decision and its deadline in one sentence, the person who owns the data processing agreements answers yes or no per tenant class, and the engineer reports the secondary's quota headroom and whatever was last measured about its behaviour. Anything unknown is called a no out loud, so that the default is visible to everyone rather than being an omission somebody later reconstructs as a decision. Then the outcome goes into the channel with a timestamp and a name — "09:52, I authorised failover for us_standard only; eu_restricted stays closed" — because in six weeks the question will not be what happened, it will be who decided and on what basis.

Failing closed is the right call more often than it feels like at 09:41, and it is the correct default for any request whose output is acted on automatically. A refund-eligibility decision or an obligations extract produced by an unmeasured substitute is worse than no answer, because the error is silent and lands in a system of record. A summary a human reads next to the source document can tolerate a weaker answer, provided it is labelled as one — which topic 2.21 argued at length and which does not get easier under incident conditions.

Failing closed is also not the only thing available between full failover and doing nothing. Requests that do not need a fresh answer can be served from a cache; requests that are not interactive can be queued and drained when the primary recovers, which converts an outage into a delay for the batch traffic and costs nothing contractually; and features can be shed one at a time rather than all together. On this system about 22% of contract-summary traffic is a re-summarisation of a document already summarised within the last 24 hours, so a one-day cache carries a fifth of the load through the outage with no provider involved at all — which is worth building precisely because it is the only mitigation that works for the tenants who have no compliant alternate.

A runbook that names a provider has specified almost nothing#

"Fail over to Provider B" tells the person at 09:41 the destination and none of the things they have to know to get there. The step is the last line of the procedure, not the procedure. What the runbook owes them is everything that turns a judgement call into a lookup:

• Which traffic is eligible, expressed as a query they can run — not a description of a policy they have to interpret.
• Where the secondary account lives and what its retention setting is, with the date that was last verified, because someone may have changed it.
• What was measured about the substitute and when: the eval score per slice on the secondary, its context limit, its refusal behaviour, its tool-call format, its max output length. A number from eight months ago is a stale number, and saying so in the runbook is more useful than omitting it.
• The quota headroom and the ramp steps.
• Which features are disabled while the substitute is serving, and how to turn them off — one flag, named.
• Who authorises, with a name, a role and a phone number, and what happens if they cannot be reached in ten minutes.
• How it is reverted, and what the expiry is if nobody remembers to.

Runbooks for provider failure are a small genre with a lot of shared structure, and The LLM Provider Incident Runbook is a serviceable skeleton for the failover, degraded-mode and communication sections. What no external template can supply is the residency matrix, because that is a property of your contracts rather than of the technology, and it is the part that decides whether executing the rest of the runbook is lawful.

Most of that belongs in configuration rather than prose, because prose gets interpreted under stress and configuration gets executed:

# routing/failover-policy.yaml — read by the running service, no deploy needed
failover:
  expires_at_utc: "2026-06-23T14:00:00Z"   # hard stop; reverts to primary on expiry
  policies:
    - tenant_class: eu_restricted          # EU-only processing, zero retention
      eligible_targets: []                 # none configured today
      on_no_target: fail_closed
      customer_message_key: outage.eu_no_alternate
    - tenant_class: us_standard
      eligible_targets: [provider_b_us]
      ramp: [0.05, 0.25, 1.0]              # advance manually between steps
      disable_features: [obligations_extract, tool_strict_schemas]
      label_responses: degraded
    - tenant_class: unknown
      eligible_targets: []
      on_no_target: fail_closed            # unclassified is never eligible

Failing closed then has to be something the product can actually express. A 500 with a stack trace teaches the customer nothing and generates a support ticket; a response that names the constraint lets their engineer stop investigating:

{
  "error": {
    "type": "service_unavailable",
    "code": "no_compliant_processor_available",
    "message": "Summarisation is paused for your account. Our primary model provider is degraded, and the alternate processor is outside the EU-only processing region required by your agreement, so we are not routing your data to it.",
    "incident_url": "https://status.example.com/incidents/2026-06-23-a",
    "retry_after_seconds": 900
  }
}

What to build so the next outage is a smaller decision#

The gap this incident exposed is that the only compliant processor for 30% of the business is the one that is down. Closing it is procurement work before it is engineering work: an account with the secondary provider in an EU region, with zero retention agreed in writing, added as a subprocessor in the DPAs, which takes weeks of contract review and cannot be done at 09:41. Until that exists, eligible_targets: [] for the restricted class is the honest configuration, and the corresponding product commitment — that these tenants have a single-provider dependency — belongs in the contract and the SLA rather than being discovered by them during an outage.

Everything else is comparatively cheap. Populate residency_class on every tenant and make it a required field at onboarding, then add a check that fails the deploy if any active tenant is unclassified. Move the failover from a code constant to the runtime policy above so it changes in seconds. And send a small, named slice of live traffic to each configured secondary on a schedule so its credentials, quota and behaviour are facts rather than assumptions — topic 2.21 made the case for deliberate exercises, and residency adds a reason to run them: an exercise that routes an EU tenant to a US account is a rehearsal for the breach, so the exercise itself has to respect the same policy file.

There is a limit to how far this goes. For a tenant with a single-region requirement and one qualified provider in that region, no amount of engineering produces a compliant substitute, and failing closed is the permanent answer rather than a stopgap. Saying that plainly during procurement is much cheaper than saying it during an incident.

By 10:05 the decision has been made and executed, whichever way it went: some cohort of customers is receiving errors, another is receiving answers from a substitute nobody has measured, and both groups are about to ask what happened. That conversation is next, and it starts with the CEO asking for a message that does not alarm anyone.