Filter blocks in an error taxonomy

Most retry layers are configured from a list of status codes: 429, 5xx, connection errors, three attempts, exponential backoff. That list encodes an assumption that is true of everything on it — that the world changes between attempt one and attempt two. A 429 means a shared resource was momentarily exhausted, and waiting changes the resource's state. A 503 means an instance was unhealthy, and the load balancer will route you elsewhere. A connection reset means a socket died.

A filter verdict has nothing to wait for. The classifier runs at a fixed threshold: feed it the same tokens and it produces the same scores, compare those scores against the same configured severity threshold and you get the same decision. There is no sampling temperature in the path, no load-dependent behaviour, no partial state.

The general test behind all four examples is one question, and it is worth extracting because it settles cases this chapter does not cover. A retry is an identical request sent again after a wait. It can therefore only help if something outside the request changes during that wait. For the 429, the thing that changes is how much of a shared quota has been consumed; for the 503, which instance the load balancer picks; for the reset, whether a socket exists. Ask the same question of a filter block and there is no answer to give. The classifier's weights are the same weights, your bytes are the same bytes, the threshold is the same threshold, and none of the three is a function of time or of load. Waiting is not a step toward a different outcome. It is the same outcome, later.

The verdict is a function of the input#

The exception proves the shape of the rule. If a prompt is not byte-identical between attempts — because it interpolates a timestamp, because a retrieval step re-runs and returns a slightly different document set, or because a prior assistant turn was sampled — then the second attempt is a different request and may well pass. That is not a retry succeeding. That is a lottery, and treating it as a retry succeeding is how teams end up with a filter-block rate that looks like 0.6% when the true rate of "this user's work could not be done on the first try" is 2.4%. A nondeterministic prompt builder plus a retry loop means your filter metrics are already wrong in a direction that flatters you.

Put those two rates next to each other and the size of the misreporting is visible without any further data. The true rate is four times the reported one, which means that of every four requests blocked on their first attempt, three came back green on some later attempt and were never counted as blocks at all. Three quarters of the evidence is being deleted by the machinery that was supposed to be recovering from it. Nobody chose that; it is what a counter incremented on final outcome does when the thing in front of it is re-rolling the input.

Why a re-roll can flip a verdict at all is the part to have straight, because it is also what makes the lottery feel like a working retry. The classifier does not return a verdict; it returns a score, and the verdict is that score compared against a threshold. Scores are continuous and the threshold is a single cut point, so inputs cluster on both sides of it and some sit very close to it. For an input near the line, swapping one retrieved paragraph for another, or letting a timestamp change the token count, is enough to move it across. For an input well inside the blocked region, nothing of the kind will move it at all. The engineer who observed that "retrying sometimes works" was observing the first population and generalising to both, which is a reasonable inference from the data they had and produces a retry policy that is right for none of it.

What each pointless attempt actually costs#

Cost depends entirely on which side of the model the filter sits, and the difference is large enough to change how the bug feels.

An input-side block is cheap in money and moderately expensive in everything else. Nothing is billed, because no model ran. What you spend is a request against your rate limit, one round trip of latency, and one slot of whatever concurrency your service allocates per user. On a provider metering requests per minute, four attempts instead of one means the effective ceiling for that slice of traffic is a quarter of what capacity planning assumed.

An output-side block is where the money is. The model generated the full completion, you were billed for the input and every output token, and only then was the result withheld. At 2,100 input tokens and 780 output tokens per attempt, priced at $3 per million input and $15 per million output, one attempt is $0.0063 + $0.0117 = $0.018. Three wasted retries per block is $0.054. Across 41 blocks in a week that is $2.21 — trivially small, and exactly why nobody catches it early.

Scale it and the shape becomes clearer. The same wrapper in front of a document-summarisation endpoint handling 240,000 uploads a month at a 3% output-side block rate gives 7,200 blocked documents and 21,600 wasted attempts, at 6,000 input tokens and 1,200 output tokens per summary — $0.036 per attempt.

BLOCK_RATE      = 0.03
MONTHLY_CALLS   = 240_000
RETRIES         = 3          # attempts beyond the first
IN_TOK, OUT_TOK = 6_000, 1_200
IN_USD, OUT_USD = 3.00 / 1e6, 15.00 / 1e6

per_attempt = IN_TOK * IN_USD + OUT_TOK * OUT_USD          # $0.0360
blocked     = MONTHLY_CALLS * BLOCK_RATE                   # 7,200
wasted_usd  = blocked * RETRIES * per_attempt              # $777.60
wasted_rpm  = blocked * RETRIES                            # 21,600 requests
added_wait  = 1 + 2 + 4                                    # 7 s per blocked user

Each line of that is one multiplication and none of them is hidden. A summary consumes six thousand input tokens, which at three dollars a million is eighteen thousandths of a dollar, and produces twelve hundred output tokens, which at fifteen dollars a million is another eighteen thousandths — so an attempt costs the thirty-six thousandths the code prints as $0.0360, split evenly between reading and writing. Three per cent of 240,000 uploads is the 7,200 blocked documents. Each of those is attempted three more times after the first failure, which is where 21,600 wasted attempts comes from, and 21,600 attempts at thirty-six thousandths of a dollar each is the $777.60.

That is $778 a month in tokens generated and thrown away, plus 21,600 requests drawn against a provisioned throughput allocation — a block of model capacity reserved and billed by the hour rather than by the token — that you are paying for whether or not the response reaches a user. The latency cost lands on top: with backoff at 1s, 2s and 4s, every blocked user waits seven seconds longer before seeing an error, and where a gateway has a ten-second timeout, some fraction see a timeout instead of an error, which routes the incident to the wrong team.

None of these numbers is catastrophic on its own. The reason to remove the retry is not that $778 matters; it is that the retry cannot possibly work, and every architecture decision downstream of "we retry safety errors" is built on a premise that was never true.

Where the block goes in your taxonomy#

Most error taxonomies have two axes that people conflate: is it retryable, and is it ours. A filter block is not retryable and it is nobody's fault in the usual sense, which means it does not fit any bucket a typical taxonomy ships with. Give it its own class. The name matters less than the fact that it is not a subtype of UpstreamError, because the moment it inherits from that class, every dashboard panel aggregating upstream errors absorbs it, and a provider policy change becomes indistinguishable from a regional outage.

PolicyBlockInput and PolicyBlockOutput are separated because their cost profiles differ by two orders of magnitude and because a shift from one to the other is meaningful on its own: a provider that moves enforcement from the output filter to a stricter input filter shows a flat total rate while your token bill drops, and you want to see that rather than have it net out.

AbuseMonitoringBlock is separate because it is a genuinely different mechanism wearing similar clothes. Abuse monitoring is a second layer that scores patterns across your account over time rather than scoring one request, and its enforcement is account-scoped: it can degrade or suspend access regardless of what any individual request contains. A per-request filter block affects one user's one request. An abuse-monitoring action affects every user you have. They arrive with adjacent-looking error codes and belong in different incident classes, which is why that is the one row above that pages unconditionally.

class PolicyBlock(AppError):
    retryable = False          # read by the retry decorator, not by convention
    def __init__(self, *, stage, categories, request_id, provider, model, endpoint):
        ...

def classify(exc_or_resp) -> AppError:
    if isinstance(exc_or_resp, HTTPError) and exc_or_resp.status in (429, 500, 502, 503, 529):
        return TransientUpstream(status=exc_or_resp.status)
    if is_content_filter_error(exc_or_resp):
        return PolicyBlock(stage="input", **extract_filter_signal(exc_or_resp))
    if getattr(exc_or_resp, "stop_reason", None) in ("refusal", "content_filter"):
        return PolicyBlock(stage="output", **extract_filter_signal(exc_or_resp))
    if is_abuse_monitoring(exc_or_resp):
        return AbuseMonitoringBlock(**extract_account_signal(exc_or_resp))
    ...

@retry(on=lambda e: getattr(e, "retryable", False), attempts=3, backoff="exponential")
def call_model(payload): ...

Making retryable an attribute of the error class rather than a list of status codes in the retry decorator is the change that makes this stick. A status-code list lives in one place and gets copied into three; a class attribute travels with the error wherever it is raised, and the next engineer who adds a provider cannot accidentally reintroduce the behaviour by matching on a 400.

There is a second component that quietly consumes filter blocks and that nobody thinks to check: the circuit breaker, the component that watches the error ratio against a dependency and, once it crosses a threshold, stops sending traffic there for a cooling-off period so a failing upstream cannot drag your own service down. Breakers are configured on an error ratio against an upstream, and if policy blocks are counted as upstream errors, a customer whose content trips the filter can push the breaker over its threshold. The breaker then opens against the provider for every tenant, including the 99.6% whose requests were succeeding. A cohort-specific content problem becomes a service-wide outage produced entirely by your own resilience machinery, and the post-incident review records it as a provider incident because that is what the breaker's telemetry says. Whatever predicate the breaker uses to count failures must exclude the policy classes: a request the provider answered correctly and promptly, with a verdict you asked for by using their platform, is evidence the upstream is healthy.

Each step of that escalation is worth walking, because the outcome sounds implausible until the mechanism is laid out and then sounds inevitable. A breaker's unit of state is the dependency, not the caller: there is one counter per upstream, and every call any tenant makes lands in it. The ratio it computes is therefore a ratio over pooled traffic. A single tenant whose documents trip the filter contributes failures to that pool in proportion to their own volume and their own block rate, and neither of those is bounded by anything the other tenants do. When the pooled ratio crosses the threshold, the breaker opens — and opening is per-dependency too, so what it stops is every tenant's traffic, not the traffic that produced the count. The component installed to contain a blast radius has just set one.

The belief underneath the misconfiguration is that a breaker's failure count measures how unhealthy the provider is. It does not. It measures how often calls to the provider ended in something your code called a failure, and those are the same quantity only for as long as your classification is honest. Feed it a class of event that means "the provider answered correctly and quickly" and the breaker will faithfully report an outage that is not happening, to the 99.6% of requests that were succeeding, in telemetry a post-incident review will read as the provider's fault.

What should wake somebody up#

Nothing about an individual filter block warrants a page. It is a single user's single request, it has already failed, and no engineer at 3 a.m. can change the outcome. What warrants a page is a change in the population of blocks, because that is the only thing indicating something moved — and something moving is exactly what you cannot see if these events are pooled with upstream errors.

The paging rule that follows from determinism is inverted from the usual one. For transient errors you page on volume, because volume means capacity is gone. For policy blocks you page on composition: a category that was never firing starts firing, a cohort that was never blocked is now blocked, a rate stable at 0.4% for eleven weeks moves on one endpoint and one model while every other segment holds still. A doubling of the global number tells you almost nothing about which of those happened.

The reason volume is so weak a signal here follows directly from the determinism. Each block was certain given its input, so the count of blocks is the count of requests users sent whose content sits inside the blocked region. That number has two independent ways to move: the region can move, which is a provider change, or the population of requests can move across a region that stayed still, which is your customers' content changing. A count cannot tell those apart, and both are ordinary. One enterprise customer finishing their onboarding doubles a block count without anything having gone wrong anywhere. A category that has never once fired starting to fire cannot be explained that way, because a population shift moves the size of an existing number and cannot conjure a category the classifier was not previously applying. That asymmetry is the whole reason composition beats volume, and it is why the alerting chapter later in this topic spends its effort on labels rather than on thresholds.

Determinism holds only for a fixed policy version#

Determinism is a property of the classifier given fixed inputs and a fixed policy version, and both qualifiers do real work. Providers ship classifier updates and threshold changes without a version pin you control; a request allowed on Tuesday can be blocked on Thursday with no change on your side. That is not nondeterminism, it is a policy change, and telling the two apart is a measurement problem a later chapter builds for. It does mean "we verified this input passes" has a shelf life, and a regression suite that ran once at build time is not a substitute for a check that runs continuously.

Determinism also does not mean "never send the request again". Re-sending a modified request is legitimate and often the right product behaviour: strip an attachment, drop a retrieved chunk, split a long document into sections and summarise the ones that pass. That is a fallback, not a retry, and it belongs in application logic where a human made a decision about what degradation is acceptable — not in a generic decorator that will happily re-send identical bytes forever.

The practical shape of all of this is four decisions taken once, at the boundary. A filter block is never retried. It carries its own error class rather than inheriting from the upstream one. It is excluded from the circuit breaker's failure count. And it pages on composition rather than volume. Each is small, each is the opposite of the default the surrounding machinery ships with, and together they are what stops a vendor policy change from arriving in your organisation disguised as a regional outage.

None of it tells you whether a particular block was correct. You now know not to retry it, where to file it, and what a change in the population means. What you do not yet know, for any specific failure, is which policy category fired and on which part of the request — and without that, the conversation with the provider cannot start and the conversation internally degenerates into people guessing at prompts.