Provider appeals and fallback design

A content-dependent block is a failure mode most availability commitments were not written for. The service is up, requests are being served, responses are being returned, and a specific class of work fails every time. Two questions arrive from the customer within a day of each other: is this a breach, and when will the items that already failed be processed. The comfortable answer to each is wrong.

An appeal here means any of the documented paths by which a provider will reconsider a filter outcome — a severity-threshold change, a modified-filter application, an abuse-monitoring exemption, a support case, a false-positive report. Every one of those paths is asynchronous, and the customer is not.

Whether a content-dependent failure is a breach#

An availability commitment — the service-level objective, or SLO, written into a contract — on a document-processing service is almost always request-based rather than time-based: 99.5% means 99.5% of submitted work completes successfully, which is 0.5% allowed failure.

MONTHLY_UPLOADS = 12_000
SLO             = 0.995
BLOCK_RATE      = 0.08

budget   = MONTHLY_UPLOADS * (1 - SLO)             # 60 documents
failing  = MONTHLY_UPLOADS * BLOCK_RATE            # 960 documents
burn     = failing / budget                        # 16.0x
days_to_exhaust = (budget / failing) * 30          # 1.9 days

At 12,000 uploads a month the error budget — the quantity of failure the commitment permits before it is breached — is 60 documents. An 8% block rate produces 960. That is sixteen times the monthly budget, and at a steady rate it consumes the entire month's allowance in about seven hundred and fifty uploads, roughly a day and a half of normal traffic.

Every figure there is one arithmetic step from the two the contract and the incident supply, and they are worth doing in the open rather than reading off the code. The budget is the allowed failure fraction times the monthly volume: half a per cent of twelve thousand is sixty documents, and that sixty is the entire quantity of failure the commitment tolerates in a month, of any kind, from any cause. The failures are the block rate times the same volume: eight per cent of twelve thousand is nine hundred and sixty. Dividing the second by the first gives the sixteen — the month's allowance is being spent sixteen times over. And the time to exhaustion is the reciprocal of that burn applied to the month: a sixteenth of thirty days is a little under two. The upload count comes from the same place, since sixty blocked documents at eight per cent requires about seven hundred and fifty submissions, and twelve thousand a month is four hundred a day.

The sentence that arithmetic produces is the one worth having ready before the customer asks, because it is what makes the answer specific rather than apologetic. The commitment is not being missed by a margin that could be recovered with a good week. It is being exceeded by a factor, and the factor is large enough that the month was decided in its first two days.

Where a time-based commitment is written instead, the arithmetic lands in the same place if the definition is honest. 99.5% of a 30-day month allows 216 minutes of unavailability. A defect affecting 8% of a specific document class is not conventional downtime, but it is also not zero — and the "it isn't downtime" argument depends entirely on a definition of availability the customer never agreed to. Arguing definitions with a customer whose staff cannot do their work is a losing position even when the wording is on your side, and it is usually a sign that the SLO was written before anybody thought about partial, content-dependent failure.

The defensible answer is that it is a breach of the commitment's intent and very likely of its letter, that the cause is a third-party policy classifier rather than an outage, that the failure is deterministic for the affected document class rather than random, and that you know exactly which class. That is worse news than "we're looking into it" and considerably better received, because it is specific enough to plan around.

Recourse paths and their turnaround#

Two rows are commonly misread. Threshold configuration is fast and often overlooked, but it helps only for categories that are configurable and only within the bands the provider permits — some detectors, particularly jailbreak and protected-material classifiers, are on or off and not tunable, and on many plans the "off" position requires the same eligibility review as a modified filter. And a false-positive report is worth filing for the sake of the next customer while being worth nothing to this one; treating it as remediation is how a two-week incident becomes a six-week one.

Because most of these forms are free text, what you put in them determines how far they travel. A submission reading "our requests are being blocked incorrectly, please fix" is routed to a first-line queue and answered with a link to the policy documentation. A submission carrying the synthetic fixture, the exact category and severity, the deployment name, three provider request ids with timestamps, the affected volume as a rate and a count, and one sentence describing the workload in domain terms — structured clinical risk assessments submitted by licensed clinicians in a service commissioned to assess self-harm risk — is a document a reviewer can act on without asking you anything. Every input to the second version is something the earlier chapters of this topic produced as a by-product.

Say plainly what you are asking for, too. "Please review" invites a review and nothing else; "please raise the self-harm severity threshold to high on deployment gpt-4o-clinical-eu, or advise which eligibility process applies to this workload" names an action and a fallback question, and gets an answer to at least one of them. Record what each path returns, in writing, alongside the fixture — not "we filed an appeal" but the constraints: what was requested, what scope it would apply to if granted, what the provider committed to, and what they explicitly declined.

The items that already failed will not un-fail#

A filter verdict is made at request time on the request's content. Nothing is stored, nothing is queued, and no record of the decision exists on your side except the log you wrote. If an appeal is granted and a threshold changes, the effect is that requests made after the change get a different verdict. There is no queue that drains, no backfill job on the provider's side, no list of your blocked requests they can go back and approve.

"Nothing is stored" needs one condition attached, because on some platforms it is not literally true and the exception is the kind a customer's compliance team will raise. Providers running abuse monitoring do retain request content for a period so that human reviewers can look at it, which is precisely why an abuse-monitoring exemption is a separate application in the table above. What does not exist is a stored decision with a path back into your results. The retained copy feeds a review process about your account, not a queue of work awaiting a better verdict, and no configuration change causes anything in it to be scored again on your behalf. So the operative claim survives with its scope stated: your failed requests are not held anywhere that reprocessing could come from.

The expectation that they would be is reasonable, which is why it needs dislodging explicitly rather than by assertion. Almost every other refusal an engineer deals with leaves an artefact behind — a rejected payment leaves a transaction, a denied claim leaves a case, a failed job leaves a row in a queue — and in each of those the appeal and the artefact are the same object, so winning the appeal updates the thing that was refused. A filter verdict has no artefact. It was a function evaluated on bytes in flight, and the bytes went to the model or did not. Winning the appeal changes the function. It cannot change an evaluation that has already happened, because that evaluation was never written down anywhere either side could point at.

The reprocessing therefore has to be yours, and it has to have been designed for. If a pipeline treats a blocked document as a terminal error and drops it — logged, counted, and gone — then when the policy does move, there is no list of what to resubmit. The failed documents are recoverable only if the customer still has them, which for a batch integration pulling from a document management system may be true and for an interactive upload flow often is not.

The quarantine state is the whole design. A blocked document goes into a durable store with its identifier, its content hash, the captured signal, the fixture it matches, and a status. It is not retried on a timer, because the verdict is deterministic. It is re-run exactly once, when something has changed — and the thing that tells you something has changed is the fixture suite flipping from blocked to passing on its nightly run.

Both halves of that follow from the determinism rather than from taste. A timer is the right trigger when the thing you are waiting for is a state that recovers on its own, and nothing here recovers on its own: the verdict is a function of content and policy, the content in quarantine is frozen, so the only variable left is the policy. A re-run before the policy moves is guaranteed to reproduce the same verdict, which makes every scheduled retry an expense with a known result. And a re-run after it moves is worth exactly one attempt, because the same determinism that made the timer pointless makes the second attempt pointless too — if the document passes, it is done, and if it does not, nothing will change until the policy moves again, at which point the fixture will say so and the queue will be re-run then. What the fixture suite contributes is the only thing missing from that argument: an observation of the one variable, taken on a schedule, on an input you are allowed to keep.

def on_policy_change(fixture_id: str) -> None:
    """Called when a nightly fixture run reports a verdict change."""
    fx = load_fixture(fixture_id)
    docs = quarantine.find(status="blocked", deployment=fx["deployment"],
                           categories=fx["expect"]["categories"])
    log.info("policy_change_replay", fixture=fixture_id, candidates=len(docs))
    for doc in docs:                       # rate-limited, off-peak, not in the request path
        result = summarise(doc)            # one attempt only
        quarantine.update(doc.id,
                          status="summarised" if result.ok else "blocked",
                          attempts=doc.attempts + 1,
                          last_checked=now())

Retaining blocked content is itself a decision with an owner. Quarantining clinical documents means holding patient data in a new place for an indefinite period, so the store needs the same retention rules and the same processing basis as the rest of the pipeline, and a TTL — a time to live, after which the record expires — beyond which the entry keeps its metadata and drops its content. A quarantine store nobody agreed to is a data-protection finding waiting to be written up.

What you build during the five days#

Route the affected class. If the block is confined to one deployment's configuration, a second deployment with a different threshold, or a different model, or a different provider, may serve the same documents. This is the fastest lever and the one with the most hidden constraints: a second provider may sit in a different region, which for health data may violate data residency, the contractual requirement that data stay inside a named jurisdiction, and a different model changes summary quality in ways the customer will notice. Route by the segment you already have — cohort and category — rather than globally.

# Narrow, temporary, and dated so it does not become permanent by neglect.
routing:
  - when:
      cohort: healthcare-enterprise
      endpoint: /documents/summarise
      prior_block_category: violence
    route_to: gpt-4o-clinical-eu-relaxed      # medium threshold, same region
    fallback: quarantine
    expires: 2026-09-15
    owner: platform@
    ticket: INC-4471

Give the operator a path. For a workload where a machine summary is genuinely unavailable, the answer for a specific urgent document is a person: a queue view of quarantined items, visible to the customer's own staff, from which they can mark a document for manual handling. Eight per cent of 12,000 is 32 documents a day, which is a human-sized queue and an unpleasant surprise if you discover that on the day rather than in the plan.

Degrade honestly rather than silently. Splitting a document and summarising the sections that pass produces a summary with a gap in it, and a clinical summary with an unmarked gap is more dangerous than no summary at all. If you ship partial output, mark the omission in the output itself.

An appeal changes future verdicts only#

Even when granted, an appeal is not a rollback. The language in the confirmation email — "the requested configuration has been applied to your resource" — describes a forward-looking state change and nothing else.

A fallback is not a retry either. It is a different request, to a different target, chosen deliberately by application logic, with different quality and possibly different residency and cost properties. Putting it inside the retry decorator turns it into an invisible policy where every blocked request silently becomes a request to a second vendor — including requests from the tenant whose contract forbids exactly that.

And filing is not free. Appeals draw on a finite amount of provider attention, and an account that files ten thin tickets a quarter is read differently from one that files two with reproductions and metrics attached. The fixture and the segmented rate exist partly so that when you do ask, you can show the block, the volume, the cohort, the date it started, and the workload it affects — which is also the only version of the request with a good chance of coming back approved.

The division of labour is the thing to carry. The appeal is the only path that can change the provider's behaviour, and it is asynchronous, uncertain, and scoped in ways you do not control. Everything that determines what your customer experiences in the meantime — whether failed work is recoverable, whether the affected cohort has a route, whether an urgent item can reach a person — is yours, buildable in days, and independent of the reply. Teams that file and wait have one plan; teams that file and build have two, and only one of them depends on a stranger.

Sometimes the reply comes back quickly and not in your favour. Five business days later: reviewed; the behaviour is operating as intended for this category and no configuration change will be made. That removes the assumption every plan above was resting on.