Adapters and gateways

A gateway is a service you deploy and operate that your own services call instead of calling vendors, so that provider keys, spend limits, rate limits and audit logs live in one place. The translation work inside it is the same translation work an in-process adapter does — parse a request in one schema, reshape it, call a vendor, reshape the response back — and moving that code out of your process does not change what it does. It changes who owns it, when it can be deployed, whose pager it rings, and, most consequentially, whether it can fail independently of everything it fronts.

That framing settles a lot of arguments quickly, because it moves the question off which design is cleaner and onto whether what you are centralising is worth a new failure domain. The answer depends almost entirely on how many parties share the thing being centralised, which is why the same proposal is right for one organisation and wrong for another with identical code. This chapter prices both sides: what centralising genuinely buys and how that scales with the number of services, then what a gateway makes strictly worse — availability, tail latency, and a second normalization layer written by somebody else — and the configuration choices that keep the new failure domain from becoming an outage.

What centralising actually buys#

Take a company with twelve services in Python, Go and TypeScript, each holding its own provider key, each logging model calls in its own format, and finance unable to say which team spent what. Four things are genuinely shared problems there, and each is expensive in proportion to the number of services rather than to the amount of code.

Keys are the clearest. A key rotation with twelve services means twelve secret updates, twelve deploys and a coordination window; one such rotation took a team three weeks, and two services were missed until an audit found them. Behind a gateway, the provider key exists in one place and each service gets a virtual key that the gateway maps; rotation is a config change with no service deploy at all, and revoking one team's access is a one-line edit rather than a hunt.

Budgets and attribution are the reason finance is in the conversation. There is no way to attribute spend from a provider's invoice when eight services share a key, because the invoice knows about the key and not about your org chart. A gateway sits where every request passes and can tag each one with the virtual key that made it, which turns attribution into a group-by. Enforcement follows from the same position: a hard monthly cap per virtual key is something only a component that sees all the traffic can implement.

Rate limits behave the same way. Eight services each running their own retry logic against a shared provider quota will collectively exceed it while each one individually believes it is being polite, and the resulting 429s land on whichever service was unlucky rather than on the one causing the pressure. One process holding one token bucket — a counter drained by each request and refilled at the rate the provider's quota allows — can allocate that quota deliberately, 60% to the customer-facing service, 15% each to two batch jobs, the rest shared, which is a policy you cannot express at all when the quota is enforced by the provider against a key eight processes share.

Logging is the least glamorous and often the actual trigger. Compliance wants every prompt and completion retained for 30 days with PII — personally identifiable information, names, emails, account numbers and the like — redacted, and implementing that consistently in three languages across twelve codebases is twelve implementations of a redaction rule that must not drift. Doing it once at the hop is one implementation and one retention policy.

Most of that table is a function of N. At N = 1 every row on the left is cheap and the two rows at the bottom are pure cost, which is why a single service on a single provider should not run a gateway. At N = 12 across three languages, four of the rows are genuinely painful and stay painful forever, because they grow every time someone adds a service.

Reading the table as arithmetic rather than as a list of opinions gives a decision rule that survives an argument. The left column's entries are per service: one secret update each, one instrumentation each, one redaction implementation per language. Their total is N times a per-service cost. The right column's entries are per organisation: one config change, one group-by, one retention policy, however many services there are. Their total is a constant, plus one more constant that is not written in money at all — the operational weight of a service that can be down. So the comparison is a line rising with N against a flat line with a step in it, and the only question is where they cross. At N = 1 the rising line is at its lowest possible value and the flat one is unchanged, which is why a gateway is never right there. The crossing point moves left when the per-service cost is high, which is what "three languages" means: a redaction rule written three times is three times the per-service cost and three chances for it to drift.

The rule that follows is to count parties rather than requests. Traffic volume does not appear anywhere in that comparison, so a single service making an enormous number of calls does not need a gateway on those grounds, and twelve tiny services do.

The configuration is mundane, which is part of the appeal — the policy that took twelve codebases becomes a file:

model_list:
  - model_name: chat-default
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: chat-default          # same alias, second entry = fallback order
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  fallbacks: [{"chat-default": ["openai/gpt-4o"]}]
  num_retries: 2

general_settings:
  max_budget_per_key: 2000            # USD per month
  budget_duration: 30d

A service now sends "model": "chat-default" with its own virtual key, and routing, failover order, budget and attribution are properties of that file rather than of anybody's application code.

Availability, tail latency, and a second normalization layer#

The availability arithmetic is the item most often skipped. Suppose each provider is independently available 99.9% of the time. A service with in-process adapters and a real failover path can call the second provider when the first fails, and if the failures are close to independent the combined availability of "at least one provider answers" is around 99.9999%. Put a gateway in front and the ceiling becomes the gateway's own availability, because every path runs through it. A gateway at 99.95% — a reasonable number for an internally operated service with a small team — is 22 minutes of unavailability per month, and it is unavailability that no provider-level failover can rescue you from, since the failover logic lives on the far side of the thing that is down.

Both of those numbers come from one step each, and doing the steps is what makes the comparison believable. Availability of 99.9% means unavailable one part in a thousand of the time. Two providers are both unavailable at the same instant one time in a thousand times a thousand — a multiplication, which is allowed only because the failures were assumed independent, since independence is exactly the property that lets two probabilities be multiplied. One part in a million of unavailability is 99.9999% available, and that is the figure quoted. The gateway is a different calculation because it sits in series rather than in parallel: every request passes through it and then through a provider, so the combined availability is the product of the two, and a product of numbers below one can never exceed the smaller of them. The gateway becomes a ceiling by arithmetic, not by misconfiguration.

Turning 99.95% into minutes is the same kind of step. Availability of 99.95% leaves five parts in ten thousand unavailable, a thirty-day month is a bit over forty-three thousand minutes, and five ten-thousandths of forty-three thousand is around 22 minutes. The reason those 22 minutes are worse than 22 minutes of provider outage is where the failover code sits. Failover between providers is a decision made inside the gateway, so it is available only while the gateway is running, and the case it is meant to protect against is the one case where it is not.

That is also the belief this section exists to displace, and it is a reasonable one to hold: a gateway improves reliability, because it can fail over between providers. The first half is true and the second half is the reason the first half is false. Adding failover raises availability against provider failure and adds a component whose own failure has no failover at all, and for a system whose providers were already at three nines the second effect is much the larger of the two. What a gateway buys is not availability. It is control over what happens during a provider incident, which is a different and genuinely valuable thing, purchased with a new way for everything to stop at once.

The independence assumption is worth stating as an assumption rather than carrying it forward as a fact. Two providers are not independent when they share a cloud region, an upstream network, or a certificate authority, and they are certainly not independent with respect to a bad prompt template or an expired payment method on your own account. So 99.9999% is an upper bound on what parallel providers can give you, reachable only for the failure modes that really are uncorrelated. The way to find out which ones those are is to make somebody name the shared dependencies before the number goes in a design document.

Latency is the cost people expect and it is usually the smaller one. A same-region hop adds roughly 3–6 ms at p50, which against a 1,200 ms model call is noise. The p99 is where it can bite, because the gateway has its own connection pool, its own queueing behaviour under load and its own timeout defaults, so a saturated gateway adds tens or hundreds of milliseconds precisely when the system is already stressed. Streaming makes this sharper still: a proxy that buffers rather than passing chunks through can turn a 400 ms time-to-first-token into two seconds, and that is a configuration mistake no non-streaming test will find.

The third cost is that a gateway is a second normalization layer, with all of the previous chapter's losses, implemented by someone else. If it presents one schema for every model it fronts, whatever that schema cannot express is unavailable to every service behind it — and now the field that has no home is in a codebase your team does not own and cannot patch on the afternoon you need it.

Which one each team should build#

A team running one service against one provider should write an adapter, because every benefit in the table above is a benefit of centralising across parties and there is only one party. A team with twelve services in three languages and no spend attribution should run a gateway, because the four shared problems are already costing them real weeks and will keep growing.

The interesting part of that answer is what would flip it. The first team should reconsider the moment a second service needs model access, or the moment compliance requires a uniform audit log, or the moment somebody wants a spend cap enforced rather than reported. The second team's answer flips if their twelve services are all in one language and can share a well-maintained internal library that solves keys, logging and budgets in-process — which is a real option, and the honest reason it usually fails is organisational rather than technical, since a library requires twelve teams to upgrade in step and a gateway does not.

Why a permanently warm direct path works, when a dormant fallback branch does not, is the same mechanism that lets a recorded test cassette — a saved provider response replayed in place of a live call — go stale without any test failing. A code path that is never executed has no way to report that it has stopped working. Its credential can expire, its dependency can be removed by a tidy-up, its request format can drift behind the provider's current one, and nothing in any build or any dashboard says so, because nothing ran it. Sending a small, continuous share of real traffic down it converts the fallback from a claim into an observation: it is exercised on every deploy, by production, and it fails at a time when one percent of requests are affected instead of at a time when a hundred percent are. The cost is real key sprawl and it is the price of the path being true rather than intended.

Neither answer is about code quality, and that is the point worth carrying out of this chapter. The same translation logic is correct in both places; what differs is whether you have bought a shared failure domain and whether the thing it centralises is shared by enough parties to pay for it. When it is, the arithmetic is compelling and the mitigations are known: run it stateless and horizontally, fail open on optional concerns, break the circuit per caller, and keep a direct path warm with real traffic.

Everything above assumed the gateway is yours: your deployment, your pager, your ability to ship a patch at 03:00. The market's more common offer is a hosted router that fronts dozens of providers for you, and it changes the calculation in a specific way, since every cost in this chapter still applies to a service nobody at your company can page, patch, or roll back.