A gateway takes a weekend and then takes years

At 03:14 a stream to a customer's browser stopped at token 412 of what would have been a 900-token answer. The provider had closed the connection without sending an error frame and without sending the final chunk that carries usage. The gateway did what it had been told to do on a transport failure, which was retry, so it opened a second request to the fallback provider and streamed a fresh answer down the same connection. The customer's UI, which had already rendered the first 412 tokens, appended the second answer to them, and the support ticket the next morning described the assistant "repeating itself and then contradicting itself".

Two other things happened that nobody noticed for eleven days. The provider billed for both generations, because the first one had produced 412 tokens of output before the socket died, and the gateway's cost attribution recorded one call, because the usage record was written from the final chunk and the first attempt never sent one. And the per-tenant token quota, decremented from that same record, under-counted by the same amount. Over eleven days, at a mid-stream failure rate of 0.4% on 2.1 million requests a month, that came to roughly 3,100 unrecorded generations and about 1.9 million tokens that one tenant consumed outside its contracted allowance.

Every one of those behaviours was correct with respect to the requirements document, which said: route requests to the right provider, manage API keys centrally, retry on failure, rate limit per tenant, and attribute cost. Five bullet points, all of them true of the system that shipped.

The weekend version genuinely does the five things#

Here is roughly what a competent backend engineer produces in a weekend, and it is not a straw man. It routes, it holds keys in one place, it retries, it limits, and it attributes.

from fastapi import FastAPI, HTTPException, Request
import asyncio, httpx, time, os

app = FastAPI()
KEYS = {"provider_a": os.environ["A_KEY"], "provider_b": os.environ["B_KEY"]}
ROUTES = {"chat-fast": "provider_a", "chat-deep": "provider_b"}
BUCKETS: dict[str, list[float]] = {}          # tenant -> request timestamps
USAGE: dict[str, int] = {}                    # tenant -> tokens this month

def allow(tenant: str, limit_per_min: int = 600) -> bool:
    now = time.monotonic()
    hits = [t for t in BUCKETS.get(tenant, []) if now - t < 60]
    hits.append(now)
    BUCKETS[tenant] = hits
    return len(hits) <= limit_per_min

@app.post("/v1/chat")
async def chat(req: Request):
    body = await req.json()
    tenant = req.headers["x-tenant-id"]
    if not allow(tenant):
        raise HTTPException(429, "rate limited")

    provider = ROUTES[body.pop("route", "chat-fast")]
    async with httpx.AsyncClient(timeout=60) as client:
        for attempt in range(3):
            try:
                r = await client.post(
                    f"https://{provider}.example.com/v1/messages",
                    headers={"authorization": f"Bearer {KEYS[provider]}"},
                    json=body,
                )
                r.raise_for_status()
                data = r.json()
                USAGE[tenant] = USAGE.get(tenant, 0) + data["usage"]["total_tokens"]
                return data
            except (httpx.HTTPError, KeyError):
                if attempt == 2:
                    raise HTTPException(502, "upstream failed")
                await asyncio.sleep(2 ** attempt)

Forty lines, five requirements, all satisfied. It will serve a demo, an internal tool, and a single-tenant product with one provider for a surprisingly long time. What it cannot do is survive contact with the conditions that only appear at volume, and the gap between this and a gateway you would put a customer behind is not a matter of polish. It is a set of decisions that the five bullet points do not contain and could not have contained, because each one only becomes a question once you have seen it go wrong.

Where the years actually go#

Take the retry line by itself. retry on failure is a phrase that hides four separate policy questions, and the first incident forces all of them at once.

The retry is safe before the first byte reaches the client and unsafe after it, because after that point the client holds state the gateway cannot revoke. A production gateway therefore has a notion of stream commitment: once deltas have been forwarded, a mid-stream failure is either surfaced to the client as a terminal error on that stream, or repaired by an explicit protocol the client understands, such as a reset frame that tells the UI to discard what it has. Neither of those is more code than the retry loop above; both are contracts, which means both have to be designed, versioned, documented, and honoured by every client SDK you ship.

Usage accounting has the same shape. The correct record is not "what the final chunk said" but "what the provider will invoice", and those differ whenever a call is abandoned partway. That means writing a provisional usage record when a call starts, reconciling it when the call ends in any way including failure, and accepting that a socket that dies without a usage frame leaves you estimating output tokens from what you forwarded. Quota enforcement then has to decide whether it is authoritative before the call (an estimate, which can be wrong) or after (accurate, which means a tenant can overshoot by one large request). Chapter 1's evaluation platform had one number; a gateway's quota logic has a whole family of them, and every one is a place where a customer's bill and your ledger can disagree.

Then there is the fact that raise_for_status treats every provider's failures as one kind of thing. The table below sets three back ends side by side: two commercial providers and a self-hosted deployment of vLLM, the open-source inference server most teams reach for when they run open-weight models on their own GPUs.

Five conditions across three providers is fifteen behaviours, and the table is the abbreviated version — it omits per-model differences within a provider, regional endpoints that fail independently, and the beta headers that change error shapes between API versions. Each row was learned from an incident by somebody. This is what "years of edge cases" means concretely: not that the work is hard, but that the requirements are discovered one production surprise at a time, and the discovery rate does not fall off for a long while because the providers keep shipping.

Rate limiting counts the wrong unit

The allow function above limits requests per minute, which is the correct unit for an HTTP API and the wrong one here. Three weeks after the stream incident, a single tenant started a backfill that sent 40 requests a minute — comfortably inside its 600 limit — each carrying a 180,000-token context. That tenant consumed about 7.2 million input tokens a minute and roughly 78% of the organisation's shared per-minute token allowance with the provider, and every other tenant on the platform started receiving 429s that the gateway had not issued and could not explain, because from the gateway's point of view nobody was over their limit.

Fixing it means limiting on tokens, which means estimating the token cost of a request before it is sent, which means running a tokenizer in the request path, which means the gateway now has a per-model tokenizer dependency and a cache and a fallback for models whose tokenizer is not published. Then the estimate is wrong for streamed output, so the limiter needs to reconcile after the fact and carry the debt into the next window. Then a tenant with a legitimate large job needs a burst allowance, so the bucket needs a refill rate and a ceiling rather than a fixed window, and the ceiling has to be lower than what the shared upstream quota can absorb. None of this is difficult; all of it is a design with consequences, and all of it is downstream of one word in the requirements list.

Key management goes the same way. Centralising keys is an environment variable until the first customer asks to use their own provider account, at which point the gateway holds third-party credentials on behalf of tenants and needs envelope encryption — each secret encrypted with its own data key, and that data key encrypted in turn by a master key held in a key management service, so that rotating the master does not mean re-encrypting every secret — an audit trail of which key served which request, a rotation path that does not fail in-flight requests, and a story for what happens when a customer's key runs out of credit halfway through a conversation. Central key storage is a weekend; being a custodian of other people's credentials is a security review, a threat model, and an on-call procedure.

Per-tenant attribution has a similar tail. The first version keys on a header, which works until a background job runs under a service account and every trace attributes to internal, until a request fans out into six tool calls and you have to decide whether the parent or the leaves carry the cost, until prompt caching makes the input token price depend on whether another tenant happened to warm the same prefix, and until finance asks for last quarter's figures restated because a model's price changed mid-month and your records stored token counts without the price that applied at the time.

Buying does not mean having no opinion#

A managed gateway is not a way to avoid making these decisions; it is a way to inherit somebody's answers to them and to spend your time on the two or three where your answer must differ. That is a real distinction, because teams that buy a gateway and then treat routing as somebody else's problem discover during their first provider outage that the failover policy is a business decision — which tenants degrade to the cheaper model, which get queued, which get an error — and no vendor default encodes yours.

There is also a shape of gateway that is genuinely worth building, and it is much smaller than the one people propose. A thin internal layer that holds credentials, stamps the request context that chapter 1's layering assigns to it, and forwards to a managed gateway underneath is a few hundred lines, changes rarely, and gives you a seam to move providers or vendors behind. The failure is not building anything; it is estimating the weekend version, committing to it in a roadmap, and then discovering that what you committed to was the fifteen-cell table.

Costed the way chapter 1 costed the evaluation platform, the honest build estimate is not the weekend and not much like it. The team that owned this gateway reached the incident-surviving version after about fourteen engineer-weeks spread over nine months, most of it arriving as unplanned work after production surprises, and then settled into roughly six engineer-weeks a year of maintenance: provider API changes, new models with new pricing shapes, tokenizer updates, and the on-call rota. At £10,000 an engineer-week that is £140,000 to get there and £60,000 a year to stay, against managed gateway pricing that for their volume of 2.1 million requests a month quoted in the region of £55,000 a year. The comparison is not close, and it only looks close if the left-hand number is the weekend.

Nor is an LLM gateway a reverse proxy with a different upstream, which is the analogy that makes the weekend estimate feel reasonable. nginx or Envoy will route, terminate TLS, and rate limit by request count, and none of that touches the parts that were hard above, because those all depend on understanding the body: tokens rather than bytes, streams that carry semantics in their final frame, errors whose meaning is in a JSON field rather than a status code, and cost that is a function of content. A gateway that does not parse the payload cannot attribute cost, cannot enforce a token budget, and cannot tell a refusal from an outage.

The honest limitation is that even a bought gateway leaves cost attribution partly unsolved when caching and batching are involved, because the price of a call now depends on cross-request state that no single request can observe. Everyone in this space is approximating, and the difference is whether you are approximating with a vendor's telemetry or with your own.

Cost was the frame in chapter 1 and it is the frame here, but both assumed something worth examining directly: that if a bought component stops suiting you, you can put it down. The gateway contract on the table is month-to-month with a published API, and the team will say, correctly, that they can leave whenever they like. The next chapter is about what that sentence actually measures.