Hosted routers and ownership

The previous chapter's gateway was yours: your deployment, your image tag, your rota. A hosted router — OpenRouter is the best known, and LiteLLM and Portkey offer managed versions of the same idea — is the same hop with the ownership removed. It has its own uptime, its own schema, its own capacity model and its own release cadence, and it can be down while every provider it fronts is serving other customers normally. What changes relative to a gateway you operate is not the list of costs but the set of actions available to you when one of them lands.

This chapter is about those actions, and about a second thing the ownership change brings with it, which is that you stop being the vendor's customer. Rate limits, model availability, capacity during a crunch and the fidelity of an error message all become properties of somebody else's commercial relationship. None of that is an argument that routers are a mistake — they are the correct choice for a set of workloads larger than most sceptics admit, and this chapter names those too — but the ownership question is answerable from an architecture diagram and a contract, before a single request is sent, and it is routinely skipped in favour of a feature comparison.

What an error looks like through an intermediary#

A 429 is the HTTP status a provider returns when a rate limit has been exceeded, and answering it normally means reading the vendor's own response headers to see which quota ran out and when it refills. Through a router the same event arrives like this:

{
  "error": {
    "message": "Provider returned error",
    "code": 429,
    "metadata": {
      "provider_name": "Anthropic",
      "raw": "{\"type\":\"error\",\"error\":{\"type\":\"rate_limit_error\",\"message\":\"...\"}}"
    }
  }
}

The upstream error is in there, as a JSON string inside a metadata field, which is better than nothing and worse than the typed exception the vendor's own SDK would have raised. What is not in there at all are the response headers — anthropic-ratelimit-tokens-remaining, retry-after, the request id you would quote in a support ticket — because those belong to a connection your process never had. Topic 2.2 covered why each vendor's error taxonomy needs handling on its own terms; a router does not remove that work, it moves the taxonomy one layer away and re-encodes it, and the re-encoding is lossy in exactly the fields you need during an incident.

The levers you have when it misbehaves#

If your own gateway leaks memory at 02:41, somebody on your rota can restart it, roll back the deploy, scale it out, or push a one-line patch. Each of those is a real lever, and the whole point of an on-call rota is that a human can pull one. If a hosted router degrades, the levers are: open a support ticket, watch a status page, and fail over to something else if you built that path. The first two are not levers; they are waiting with paperwork. The third is the only real one, and it exists only if you kept direct provider credentials and a code path that uses them, which most teams adopting a router explicitly delete on the grounds that the router is the abstraction now.

"We will keep the direct SDK around as a fallback" is the standard reassurance offered at this point, and it decays on a predictable schedule. The direct path stops being covered by continuous integration within a sprint or two, the credentials it uses expire on their own rotation and nobody notices because nothing calls them, and a refactor eventually deletes the branch as dead code with a review comment saying it has not been exercised in months — all of which is true. A fallback is a thing you run, not a thing you have; the one percent of permanently routed production traffic from the previous chapter is the version of this that survives, and it is worth insisting on before the migration rather than after.

There is a version of that story where the reviewer was wrong, and it is worth saying that it is not this one. The comment on the deletion — nothing has called this in months — was an accurate statement about the code, arrived at by exactly the process code review is supposed to follow. A reviewer cannot be expected to know that an uncalled branch is load-bearing during an incident that has not happened yet, because nothing in the repository says so and nothing in the test suite exercises it. The defect was not in that review. It was in a design that placed a critical capability somewhere no signal reaches, and then relied on individual memory to protect it. Anything protected only by memory is on a schedule, whether or not anyone has written the schedule down.

The release cadence goes the same way. A provider SDK is a pinned version in your lockfile, so a change to how it serialises requests arrives when you decide to upgrade and can be reverted by changing one line. A hosted router's normalization layer is deployed by its team on its schedule, and when it changes how it maps a parameter — the way tool definitions are translated, or whether an unknown field is passed through or stripped — the change reaches your production traffic without a deploy on your side. The property that makes one revertible and not the other is worth isolating, because it is the same property in both cases and it is not about who wrote the software. A dependency can be rolled back when a version identifier for it is stored in your repository, since rolling back is then an edit to a file you own, reviewed and deployed by your own process. A lockfile stores that identifier for an SDK. Nothing in your repository stores an identifier for the code running inside somebody else's service, so there is no edit that changes which version you are talking to. The absence of a lever is not a policy the router chose; it follows from where the version number lives. Self-hosting the same software gets this back, since a container tag is a version you control, which is most of the substance behind the decision rule in merge.dev's OpenRouter vs LiteLLM: when to use one over the other: managed breadth against self-hosted control.

You stop being the vendor's customer#

The useful comparison is a reseller. When you buy electricity through a reseller you get one bill, a simpler contract and often a better rate, and you also lose the direct relationship: your allocation is a slice of theirs, their credit standing is your supply risk, and if the generator prioritises its direct customers during a shortage you find out by having less power rather than by being told.

A router relationship has the same shape, and the specifics matter more than the analogy. Your rate limit is an allocation of the router's aggregate quota with each provider, so your headroom depends on the behaviour of every other customer sharing it, and a spike from someone else is indistinguishable from a spike of your own. When a vendor releases a new model, you get it when the router has integrated it, which is typically days and occasionally weeks. When a vendor changes an API, the router absorbs the change and you inherit whatever compromises that absorption required. During a capacity crunch, direct enterprise customers with committed throughput are served first, and you are downstream of an intermediary that is itself downstream.

The reseller is an analogy rather than a description, and the place it breaks is worth marking before it does any damage, because the break is in the part people carry away. Electricity is fungible: a reseller changes the contract and the billing and cannot change the product, so a customer who accepts worse terms still receives exactly what a direct customer receives. A router is not in that position. It chooses which model answers, it rewrites your request into a shared schema on the way in, and it rewrites the reply on the way out, so the thing delivered is not identical to the thing a direct customer would have received. The reseller comparison is sound for the commercial half — allocation, credit risk, priority during a shortage — and it understates the technical half by assuming a product that survives the intermediary unchanged. Everything in the previous two chapters about what a shared schema drops applies here as well, and it applies to a schema you did not write.

Governance is where this most often becomes a hard stop rather than a trade-off. A router terminates TLS on your prompts, meaning the encrypted connection your process opens ends at the router rather than at the model vendor, so prompt and completion content passes through, and often is logged by, a third party. Any data-residency commitment, processing agreement or sector-specific rule now has to be satisfied by that party too, in whatever region their infrastructure runs. The comparison in wavect.io's LLM Gateways Compared is worth reading for that angle specifically, because residency and governance constraints tend to eliminate options before any feature discussion begins, and eliminating options early is cheaper than discovering them in a security review two sprints in.

The last two rows are why routers win arguments, and they are real: forty models behind one client, in an afternoon, with per-model pricing visible in one dashboard, is a genuine capability that would cost a platform team a quarter to approximate. The developersdigest.tech comparison of LiteLLM, Portkey and OpenRouter is a reasonable feature-by-feature starting point if breadth is what you are optimising for.

Where a router is the right answer#

It would be a bad reading of this chapter to conclude that routers are a mistake. Evaluation and research work, where the entire point is calling many models and the cost of any single call failing is that you rerun it, is better served by a router than by nine integrations. Prototypes benefit for the same reason. A small team with no platform capacity and no compliance constraints gets keys, budgets, fallbacks and a spend dashboard for an afternoon of work, and building the equivalent would consume engineering they do not have. Low-stakes, high-variety traffic — classification, tagging, enrichment — sits comfortably behind one schema because it uses none of the capabilities a shared schema struggles with.

The direction of travel also softens some of this. Routers increasingly offer pass-through of provider-native parameters, bring-your-own-key modes that restore the direct commercial relationship while keeping the routing and observability, and per-provider endpoints that bypass normalization for the calls that need it. Those features narrow the gap, and they narrow it unevenly, which is why the evaluation has to be done against the specific router and the specific capabilities your service depends on rather than against the category.

The question this chapter answers, then, is not whether to use a router but which traffic to put behind one. Where a bad hour means a requeue, the breadth and the dashboard are worth an ownership dependency. Where a bad hour means an on-call engineer needs a lever, the levers are the product, and they only exist on a path you can page, patch and roll back.

What none of this settles is the other half. Everything here is about availability, ownership and the ability to act during an incident, and those are independent of how much capability the shared schema drops on the way through, which is a question no diagram answers and no compatibility matrix answers either. A migration argued purely on features never asks the ownership question; an argument won purely on ownership never gets around to measuring the capability loss. The next chapter is about producing that second number before you commit, because it is the one that is cheap to obtain and almost never obtained.