A lowest-common-denominator layer discards the features you pay for

A platform team at a company with three enterprise customers who each insisted on a different model vendor built the obvious thing: one internal ModelGateway with a single request type, three adapters behind it, and a rule that no application code may import a vendor SDK. It took a quarter to build and it worked. Two years later someone costed it.

The gateway's request type had a messages list, a system string, a temperature, a max_tokens, and a tools list using a JSON Schema dialect that all three adapters could translate. It had no cache control, because one of the three vendors did not have prompt caching when the gateway was designed. It had no response_format, because structured output was not universal then either. It had no reasoning-effort parameter, no tool_choice beyond auto, and no way to express a stop sequence, since the semantics differed by vendor and the team had chosen to leave the field out rather than implement three subtly different behaviours behind one name.

Each of those omissions was correct at the time and by the rules of the design. Together they cost, by the finance team's reckoning, $41,000 a month by the second year. The gateway was insurance against a migration that would have cost about six engineer-weeks, and the premium had come to more than the claim inside the first six months.

The arithmetic of the caching omission#

The largest single line was prompt caching, and it is worth doing the arithmetic rather than asserting it, because the shape of the calculation is what generalises.

The company's document-review task sends a 14,200-token system prefix — policy text, a schema, eleven few-shot examples — followed by a user message averaging 900 tokens. It runs about 1.4 million calls a month. On the vendor that serves most of that traffic, ordinary input is $3.00 per million tokens, a cache write is billed at 1.25x, and a cache read at 0.1x.

Without caching, every call pays full price for the prefix:

per call:   14,200 + 900 = 15,100 input tokens
            15,100 / 1e6 * $3.00            = $0.0453
monthly:    1.4e6 calls * $0.0453           = $63,420

With a cache breakpoint after the system prefix — a marker in the request saying "everything above this line is the reusable part" — and a five-minute TTL, the time-to-live after which the vendor drops a stored prefix that nothing has touched, nearly every call reads the prefix from cache instead of paying for it, because a workload at 1.4M calls a month keeps the entry warm almost continuously:

cache reads:  14,200 / 1e6 * $0.30          = $0.00426
fresh input:      900 / 1e6 * $3.00         = $0.0027
per call:                                     $0.00696
monthly:      1.4e6 * $0.00696              = $9,744
plus writes:  ~9,000 refreshes/month * 14,200 tokens * $3.75/M = $479

That is $63,420 against $10,223, a saving of $53,197 a month on one task. The company did not capture it, because the gateway had no field to put a cache breakpoint in, and by design no application code could reach past the gateway to set one. Adding the field was a two-day change that nobody made for eighteen months, since the gateway's stated invariant was that every field must be meaningful on all three adapters, and the third vendor had no caching to map it onto.

Latency moved with it. Cached prefixes cut time-to-first-token — the delay between sending the request and the first word of the answer appearing, which is what a user watching a spinner experiences as slowness — on that task from about 1,900 ms to about 640 ms in the vendor's own published range for large cached prefixes, and the review UI had a spinner that users complained about for two years.

What "all three support it" excludes#

The general form of the problem is that a lowest-common-denominator interface exposes the intersection of provider capabilities, and the intersection shrinks every time a vendor ships something good. It never grows on its own, because a new capability arrives on one vendor first and by definition is not in the intersection on the day it lands.

The last row is the one to keep. Everything the gateway did well is transport, which is the same conclusion the complete(prompt) helper reached from the opposite direction. The difference between a thin shared transport and this gateway is that the gateway also claimed jurisdiction over the request's semantics, and that jurisdiction is what forced the intersection rule.

The retry-on-parse-error loop deserves its own note, because it is a cost that hides inside a success metric. Without strict schema enforcement, 11.4% of document-review calls returned something that would not validate, and the standard fix is to retry with a repair instruction appended. Those retries succeeded 93% of the time, so the task's end-to-end success rate looked healthy at 99.2%. They also meant the company paid for 1.4M × 0.114 ≈ 160,000 extra calls a month, at full input price for a 15,100-token prompt, which is another $7,200 a month spent on the absence of a field.

Portability is a funded property of specific seams#

The alternative is not to abandon portability, which would put you back at the seven modules with direct SDK imports from chapter 1. It is to decide, per task, how much portability that task's role justifies, and to accept that the answer differs across the nine tasks in a way a uniform gateway cannot represent.

Document review is a case where the answer is "less than we thought". It is one task, it has a 240-case eval set, and rewriting its binding against a new vendor is roughly two engineer-days, as chapter 2's table showed for its siblings. Two days of exposure, once every couple of years, against $53,000 a month is not a close call, and the decision that produced the gateway was made without either number in the room.

Contrast that with the authentication and traffic-shaping layer, where portability is cheap because the interface genuinely is the same everywhere, and with the eval harness, where portability is what makes comparison possible at all. A harness that can only run against one vendor cannot tell you whether a candidate binding is better, and that seam is worth funding hard, including the parts that are annoying.

The rule that follows from this is narrow enough to apply in a design review. A shared abstraction may own anything where the providers' semantics genuinely coincide, and it may not own anything where they differ, because owning a divergent thing means either exposing the union — which is not an abstraction, only a namespace — or exposing the intersection, which throws away the divergence you were paying for. The task seam handles divergence by containing it inside a per-provider binding, so the intersection rule never has to be applied.

How to notice one forming before it costs two years#

The gateway was not designed badly on day one; it accreted its constraint through a series of individually reasonable review decisions, and those decisions have a recognisable form. Three of them appear in this team's own pull request history.

In November of the first year, an engineer opened a change adding cache_control to the request type, mapped onto the one vendor that supported it and ignored elsewhere. The review comment that closed it read: "we can't have fields that only work on some adapters — callers won't know which behaviour they're getting." That is a correct objection to the specific implementation and it was treated as an objection to the goal, so the field never came back.

In March of the second year, someone proposed a capabilities() method so callers could branch. It was rejected as defeating the purpose of the abstraction, which it does, and the rejection is where the design should have been revisited, because a caller that needs to branch on capability is telling you the thing you are abstracting over is not uniform.

The third form is quieter and shows up as an absence. The gateway's request type had not gained a field in fourteen months, during a period when every vendor in it shipped several significant capabilities. An interface that stops changing while the domain underneath it changes weekly is not stable; it has stopped tracking its domain, and the difference between those two readings is worth an hour in a design review.

The practical test that separates a healthy shared layer from an LCD one is to ask what the layer does when providers disagree. A healthy layer routes the disagreement to a per-provider implementation that the caller selects deliberately, as the task bindings do. An LCD layer resolves the disagreement by deleting the feature, and it does so silently, because the deletion happens in a review conversation rather than in the code. Anything that has been resolved that way should leave a trace — a comment in the request type naming the capability it declines to expose and the reason — so that the next person costing the layer can find the list without doing archaeology.

The failure mode is silent, which is why it persists#

A migration that goes badly produces an incident, a review, and a paragraph in a postmortem. A forfeited capability produces an invoice that looks normal, because there is no line item for the discount you did not take. Nobody at this company was ever surprised by the $63,420, since it grew smoothly with traffic and every month's bill was consistent with the last month's. The gateway's cost was visible only to someone who asked what the same workload would cost with a breakpoint in it, and that question does not get asked by a system; it gets asked by a person, once, if they happen to read a changelog.

That asymmetry is worth designing against directly. The practical countermeasure is an annual review of what each provider has shipped that the shared layer cannot express, with a cost estimate attached to each gap, presented as a number rather than as a list of features. When this team finally did that exercise, the list had six entries and the total was $41,000 a month, and the gateway was gutted to a transport client within the quarter.

There is an honest counter-argument, which is that some organisations really do need one interface for procurement or compliance reasons, where a customer contract specifies that their data may only touch a named vendor, and the routing must be a runtime decision. That constraint is real and it does force a common interface at the routing layer. What it does not force is that the common interface must be the only way to reach a model, and the shape that satisfies both is a per-task binding table keyed by tenant, where a tenant on the vendor with caching gets the cached binding and a tenant on the vendor without it gets the other one. The gateway's mistake was treating a routing requirement as a semantics requirement.

Everything so far has been about where to put the boundary so that a swap is possible. It has not touched what a swap actually consists of, and the two-day estimate for a rebinding is doing a lot of quiet work in the argument above. That estimate covers writing a binding, and writing a binding is the part of a migration that can be estimated from the code. The next chapter takes a real migration schedule apart and shows which line items dominate it, because the API surface is not among them.