A library cannot hold a key the caller never sees

RFC-114 was three pages long and its title was "Deprecate llmkit in favour of a platform gateway". The library it proposed to delete had been in production for eighteen months, imported by 22 services across nine teams: one complete() interface, three vendor implementations behind it, retries with exponential backoff and jitter — each attempt waiting a multiple of the last, plus a random offset so that callers who failed together do not retry in unison — and error normalisation that turned each vendor's idea of a rate limit into the same RateLimited exception carrying a retry_after. It shipped as a Python package and a Go module, and the Go module was usually one release behind. The RFC's opening argument was that the company had just acquired a team writing Rust and another writing Kotlin, and neither was going to get a port.

The RFC used the word gateway thirty times without ever saying what one was, which is worth fixing before the argument starts. An AI gateway is an ordinary HTTP service of your own, run inside your network, that sits between the applications in your company and the model vendors' APIs — the application programming interfaces, meaning the public HTTPS endpoints a vendor publishes for sending a prompt and getting a completion back. A service that would have called api.anthropic.com directly calls https://ai-gw.internal instead, and the gateway holds the vendor credential, checks whether this caller is allowed to make this particular request, forwards it upstream, streams the answer back, and writes down what it cost. The word proxy names the same component after the forwarding rather than after the policy, and the two are used interchangeably in most documentation; where this topic separates them, the proxying is the mechanism and the gateway is the set of rules that the mechanism makes it possible to enforce.

A company reaches for one because of a short list of things it needs to do about model usage that no single application can do on behalf of the others: keep vendor credentials out of application processes, hold each team to a monthly spending limit, stop a model or a vendor from being used at an hour's notice, and keep a record of every prompt that left the building. Each of those needs a place that every call already goes through, and a gateway is that place. In practice you meet one either as an internal service a platform team runs on top of an open-source proxy such as Envoy, LiteLLM or Helicone, or as a feature of an API-management product the company already operates, such as Azure API Management's AI gateway policies. From the calling code the whole thing is a base URL and a token, so the migration is often an afternoon; the argument about whether to do it is what takes three pages.

The polyglot argument is correct, and it is the weakest one in the document. Language coverage is a real cost — two ports at roughly six engineer-weeks each, plus the ongoing tax of keeping four implementations in step — and it is the kind of cost a determined team can absorb. If polyglot support is the whole case for a gateway, the counter-proposal writes itself: fund the ports, keep everything in-process, and avoid adding a network hop to the critical path of every model call in the company. The teams who wrote llmkit made exactly that counter-proposal, and on the evidence in the RFC they were winning.

What was missing from the document is that a library and a gateway are not two implementations of one idea. The library is code you compile into a caller; the gateway is a process the caller talks to. That difference decides what each can enforce, and it decides it absolutely, because a library runs inside the trust boundary of the service that imported it and can therefore only ever ask that service nicely.

The hop is a real cost and you should price it first#

Before the case for a gateway, the case against it, measured. In one region, a gateway deployed as three replicas behind an internal load balancer added 6 ms at p50 and 41 ms at p99 to a request that would otherwise go straight to the provider, p50 being the median request and p99 the slowest one in every hundred. Against a chat completion that takes 900 ms to first token and 3.2 s to finish, 6 ms is under one per cent of wall time and nobody will ever feel it. Against an embedding call that returns in 45 ms, the same 6 ms is a 13% latency increase, and a batch job embedding 4 million documents at 200 concurrent requests pays for it four million times.

The tail is the part that costs more than the median suggests. The p99 figure of 41 ms was not the gateway doing work; it was garbage collection pauses on the gateway's JVM, TLS handshakes on cold connections, and the load balancer occasionally picking a replica that was mid-deploy. None of those exist when the caller opens its own connection to the provider. In exchange, the caller now has a dependency that can be down while the provider is perfectly healthy, which chapter 6 takes apart in detail.

The honest summary of the latency argument is that the hop is cheap for interactive text generation, meaningful for embeddings and classification, and never zero. A gateway has to buy something with that money.

Three things that live outside the calling process#

The first is the credential. With llmkit, every one of the 22 services holds a provider API key, because the code that signs the request runs in that service's process and there is no arrangement of software design that changes this. The key is in the service's environment, visible to anything that can read the process's memory, printable by a debug logging statement someone adds at 2 a.m., and present in the container image's secret mount. Move the signing to a gateway and the calling service holds a token that the gateway issued and only the gateway accepts, which means a caller that is fully compromised gives the attacker access to a model endpoint under a budget, and not a credential that works against the vendor's API from anywhere on the internet. The general form of this is on every platform checklist as "manage keys centrally rather than per service", usually alongside a recommendation for short-lived credentials where the vendor supports them, and the reason it is on the list is that a static provider key leaked from one service is an incident scoped to the whole account rather than to that service.

The second is a budget that spans services. Suppose you want the recommendations team to spend no more than $4,000 a month on models, and the recommendations team is four services. A library-side budget would have to coordinate a counter across four processes on different hosts, which means a shared store, which means a network call, which means you have built a gateway with none of a gateway's other properties. The moment enforcement has to be consistent across callers that do not know about each other, the enforcement point has to be somewhere both callers already go.

The third is the ability to stop something without a deploy. A provider ships a model update that degrades your extraction accuracy from 94% to 71%, or a vendor has an incident, or legal tells you at 4 p.m. on a Friday that a particular model may not process customer data pending a contract review. With the library, blocking that model means a version bump, a release of llmkit in two languages, a pull request against 22 repositories, CI on each, and deploys coordinated with nine teams. The last time that company measured an emergency dependency bump across the same fleet, it took 3 hours 20 minutes with everyone at their desks on a Tuesday, and the realistic figure with any team on holiday was two days. With a gateway it is a config change with a propagation time measured in seconds, applied to callers who do not know it happened.

The rows split cleanly into two groups. Everything the library wins is about being inside the caller's process; everything the gateway wins is about being outside it. That is the whole shape of the trade, and it is why "pick one" is the wrong instruction to take away from the RFC.

The shape is familiar because it is not new. Everything in the gateway column above is an ordinary API gateway with a model-shaped policy layer bolted on, and the lineage runs straight through Kong, Envoy and Apigee — centralised credentials, per-consumer quotas, routing and a request log are what every one of those products sold, for exactly the reasons in the table above. Most of the deployments in production today are literally that: Azure API Management's AI gateway policies, or an Envoy build with a token-counting filter. What the model workload adds is a handful of genuinely new problems. Quotas denominated in tokens rather than requests need the gateway to parse both the request and the response body, which an HTTP proxy normally avoids. Responses stream, so the proxy is holding a connection open for 3 seconds rather than 30 milliseconds and its concurrency model has to survive that. And the logged payload is the prompt, which is often the most sensitive text in the company, so the observability decision becomes a data-classification decision. Reading a gateway's documentation with that history in mind tells you which parts are decades-proven and which parts were written last year.

Which layer owns what, written down before anyone builds it#

Keeping both is the normal outcome, and it is fine as long as two ownership questions get answered explicitly rather than by accident.

The key belongs to the gateway. If the calling service still holds a provider key so it can fail over to a direct call, then the gateway has bought you nothing on containment, budgets or blocking, since the bypass path exists on every host and will be used the first time the gateway is slow. Chapter 3 is about what a bypass path does to guarantees that only hold when they hold everywhere.

Retries belong to the gateway too, and this is the one teams get wrong, because the library already had retries and deleting working code feels like a regression. Leaving them in place means two independent retry policies stacked on one request path, which multiplies rather than adds, and produces its own outage on the first provider brownout, meaning a stretch of minutes during which the vendor is reachable and failing most requests rather than being cleanly down. Chapter 5 does the arithmetic. What the library keeps is everything that is genuinely better in-process: a typed interface, streaming that yields a native iterator rather than raw server-sent events, request construction, and one thing it must now own that it did not before — a deadline it propagates to the gateway.

# llmkit 4.0: same interface, different transport, retries removed.
client = LLMClient(
    endpoint="https://ai-gw.internal",
    token=os.environ["GATEWAY_TOKEN"],   # issued per service; not a provider key
    retries=0,                            # the gateway owns this now
    timeout=8.0,                          # propagated as a deadline header
)

resp = client.complete(
    model="anthropic/claude-sonnet-4-6",
    messages=[{"role": "user", "content": prompt}],
    metadata={"team": "recommendations", "feature": "product-summary"},
)

The metadata block is worth noticing, because it is the part that only works if everyone agrees to it. A gateway can attribute spend to the calling token without any help, and it cannot attribute spend to a feature unless callers label their requests. Making that field required at the library layer, so a request without it fails locally at development time rather than silently producing unattributable usage, is the cheapest governance you will ever ship.

What the gateway is not, and where the library alone is right#

A gateway does not give you provider parity. If one vendor supports a JSON schema response format and another does not, routing through a shared HTTP endpoint does not change that; the normalisation work is the same work the library was already doing, and someone still has to do it, either in the gateway's translation layer or in the client. Teams sometimes adopt a gateway expecting the abstraction problem to be solved by the deployment topology, and then discover they have moved it rather than removed it.

It also does not make anything cheaper by itself. Central metering tells you what you are spending, and chapter 4 is about how much harder that is than it sounds, but the tokens are the same tokens. Prompt caching — where the vendor keeps a prefix of your prompt on its side and charges a fraction of the base rate to re-read it — along with smaller models and shorter contexts, is what reduces the bill, and all three are available to a library.

The clean case where a gateway is the wrong answer is the small isolated workload: one team, one provider, one prompt, a company where nothing else uses models yet. Adopting a gateway there means the team's feature acquires a shared runtime dependency, an on-call rotation they are not on, and a deploy window they do not control, in exchange for central controls over a population of one. Chapter 7 works through what you owe that team when you tell them to wait, and how that case differs from a team that has quietly gone around a gateway that already exists.

The strongest of the three arguments above is the credential one, and it is also the one that gets challenged hardest in review, because a competent platform organisation already has a secret manager. If provider keys are stored in a vault, injected at deploy time, rotated on a schedule and audited on access, it is not obvious what a gateway adds. Both designs put a secret in a vault; both are rotatable; both leave a trail. Settling that requires looking at what happens on the day one of the holders is compromised, and how long the difference takes to matter.