Compatibility claims and provider abstractions

An OpenAI-compatible endpoint is one that a second vendor implements to match the first vendor's request and response schema, so that existing client code can be pointed at it by changing a base URL and an API key. Many hosting providers, inference platforms and self-hosted serving stacks advertise it, and the claim is true as far as it goes: the route exists, the body parses, and your deserialiser works. That is why the first request succeeds, and why the argument against treating the claim as a contract is a hard one to win in advance.

What the claim covers is checked immediately. What it omits is checked by production over the following fortnight. This chapter separates those two sets explicitly, gives a battery of checks — assembled from the diagnostics of the earlier chapters in this topic — that turns the omissions into a report you can read in under a minute, and then applies the same reasoning one level up, to the internal abstraction a team builds over three providers, which is a compatibility claim written by your own colleagues and inherits every property of the vendor kind.

What the claim covers, and what it is silent about#

The line through that table is not where a vendor chose to stop being generous. It is where the thing being claimed stops being writable down, and following that once tells you what any future compatibility claim will and will not cover. To implement somebody else's schema you read their published request and response shapes and put a translation in front of your own serving stack: accept these field names, emit those. That is a finite job with a document to work from, and when it is done the route exists and the body parses.

Everything in the lower half of the table is a property of the stack behind that translation rather than of the document in front of it. A tokenizer is the vocabulary a particular model was built around, so matching another vendor's token counts would mean adopting their tokenizer and therefore their model. A default is a decision about what to do when a field is absent, which is a statement about serving behaviour and appears in no schema. Error taxonomy, content filtering and rate-limit semantics are the same: each describes what the system does, and a schema describes what the system accepts. So the claim is not a partial effort. It is complete with respect to the only thing it could have been about, and the omissions are not omissions — they are the parts that were never expressible in the form the promise takes.

The pattern in that table is the useful part: everything a compatibility claim covers is checked on the first request, and everything it omits is checked in week three. A schema mismatch fails immediately and loudly, in staging, in front of the person who wrote the code. A defaults mismatch, a tokenizer difference, a missing usage field or a divergent error code all produce successful-looking responses, and they surface as a billing discrepancy, a retry storm or a quality complaint — routed to three different people, none of whom is looking at the integration.

The timing is forced by what does the checking in each case, which is why it is reliable enough to plan around. A schema is checked by a parser, on every request, automatically, and a parser has no tolerance and no memory: the first request either matches or does not, and the answer arrives in front of whoever sent it. Nothing plays that role for the rest. A default that differs from the one you expected produces a valid response. A token count that runs longer produces a valid response. A 429 meaning something else produces a valid, correctly-shaped error. There is no component anywhere in the path whose job is to notice, so the detector falls back to a person comparing an aggregate against an expectation — a margin against a forecast, a latency graph against last week, a summary against the one before it. That is a slow instrument by nature, and it needs a population before it can say anything, which is the fortnight.

Two things reliably break on the first request. An optional parameter the endpoint has not implemented comes back as a 400 naming the field, since compatible endpoints are usually strict about the schema they claim to accept and lenient about nothing else. And a model identifier that does not exist on the new provider comes back as a 404 or an invalid-model error, which is obvious in isolation and less obvious when the model name is assembled from configuration.

Two things reliably break silently. Usage reporting is the first, especially under streaming, where the token counts arrive on a final event that a compatible implementation may simply not emit — your parser finds nothing, your counter stays at its initial value, and every downstream cost calculation is wrong in the same direction. Token accounting is the second: the same text under a different vocabulary produces a different count, so an output cap that was comfortable becomes a truncation, and a context window that fit becomes a rejection on your longest inputs only.

It is worth noticing that a single vendor's own surfaces split this way too. The endpoint that everyone means by "compatible" is one of several a large provider offers, and a newer surface from the same vendor changes the request and response shape enough that migrating between them is real work — the comparison written up as OpenAI Responses API vs Chat Completions API is the clearest account of what moves. So "compatible" names a schema that one vendor has itself moved beyond, implemented by a second vendor to attract traffic, with no commitment on either side to keep implementing it as the original evolves.

Run the battery before you believe it#

The checks worth running are the ones the earlier chapters produced, assembled into something that runs on demand against any endpoint claiming compatibility.

import pytest

PROVIDERS = ["incumbent", "candidate"]

@pytest.mark.parametrize("p", PROVIDERS)
def test_usage_is_reported_non_streaming(p):
    r = call(p, "Summarise: " + FIXTURE_TICKET, max_tokens=256)
    assert r["usage"]["output_tokens"] > 0

@pytest.mark.parametrize("p", PROVIDERS)
def test_usage_is_reported_when_streaming(p):
    events = list(stream(p, "Summarise: " + FIXTURE_TICKET, max_tokens=256))
    assert usage_from(events)["output_tokens"] > 0
    assert stop_reason_from(events) in {"end_turn", "stop", "max_tokens", "length"}

@pytest.mark.parametrize("p", PROVIDERS)
def test_context_overflow_is_classified(p):
    with pytest.raises(ProviderError) as e:
        call(p, "x " * 400_000, max_tokens=16)
    assert e.value.bucket is Bucket.INPUT_TOO_LONG

@pytest.mark.parametrize("p", PROVIDERS)
def test_truncation_is_labelled(p):
    r = call(p, "Write 2000 words about invoicing.", max_tokens=64)
    assert r["stop_reason"] in {"max_tokens", "length"}
    assert r["usage"]["output_tokens"] <= 64

@pytest.mark.parametrize("p", PROVIDERS)
def test_token_count_for_the_same_text(p):
    r = call(p, FIXTURE_2000_CHARS, max_tokens=1)
    record(p, "input_tokens", r["usage"]["input_tokens"])   # compare, do not assert

The last row is the one that needs a deliberate 429, which usually means a short burst against a low-tier key, and it is worth the trouble: an adaptive concurrency controller reading rate-limit headers that the compatible endpoint does not send will never back off, and will discover the limit by hitting it continuously. Running this battery takes under a minute and produces the four-column record — schema, defaults, errors, limits — worth keeping next to the adapter. It also gives a planning document an honest answer: the compatibility claim gets you to a working first request in an afternoon, and the battery tells you what the remaining work is instead of letting week three tell you.

The same ceiling, one level up#

An internal abstraction over three providers is the compatibility claim written by your own team, and it inherits the same property. The interface can only expose what all three backends do, so it is a ceiling on capability, and the ceiling is set by the least capable backend. Prompt caching directives attached to individual content blocks, a provider's structured-output enforcement, its particular tool-choice modes, its extended-thinking controls, a beta flag two weeks old — each of those exists on one backend, has no counterpart on the others, and arrives at the interface as a special case.

The ceiling is not a consequence of anyone designing the layer badly, and the step that produces it is one line long. The point of the interface is that a caller does not need to know which backend will serve them. So every method the interface offers has to work whichever backend it lands on — which means the set of methods is the set of things all backends can do, and that is the definition of an intersection. Nothing about better engineering moves it.

Which is also why the usual repair makes things worse rather than better. Adding a field that only one backend implements does not raise the ceiling; it breaks the property the ceiling came from. A caller now has to know which provider is behind the call before they can decide whether the field will do anything, and knowing that was precisely what the layer existed to spare them. Repeat the manoeuvre a few times and you have a surface that is the union of three vendors' interfaces, requires the same knowledge the raw clients required, and adds a layer of indirection on top of it.

The dotted arrows are the whole design. Without them, a team that needs a provider-specific capability has two options and both are bad: widen the interface with a field that only one backend implements, which repeats until the interface is a union of three vendors' surfaces and abstracts nothing, or bypass the layer quietly in one service, which means the layer no longer describes how your system talks to providers and nobody knows it. That second outcome is the single failure that kills these layers, and it is a governance failure rather than a technical one — the layer stops being true, and an untrue map is worse than no map.

The last row on the right is the working test. If using a capability requires the reader of the call site to know which provider is behind it, hiding it does not remove that knowledge, it relocates it somewhere untested.

The escape hatch has to be typed and observable rather than a free-form dictionary that anything can go into, and the reason is the failure this whole topic keeps meeting. A dictionary that accepts anything cannot refuse anything, so an option meant for one backend and sent to another is simply not read — no exception, no log line, and a request that behaves as though the option had never been set. That is the same shape as a dropped sampling parameter and a discarded response field, arriving through a mechanism you built yourself, which is the part that stings. Validation is the only thing that converts it into a signal, because validation is the only step in the path that has an opinion about what should have been there. A provider_options parameter validated per backend, so that passing an option meant for one provider while routed to another is an error rather than a silent no-op, keeps the layer honest about which calls are coupled; and an accessor that hands back the underlying client covers the remaining cases without anyone having to construct a second client in secret. Both of those are the previous chapter's reversible seam moved up a level: visible at the call site, scoped to one capability, and removable when the capability becomes common to all backends.

Deciding not to build the layer is a legitimate outcome. It is the wrong call when there is one provider today and the second is hypothetical, because you will build the abstraction against a sample size of one and discover on contact that you abstracted the wrong axis. It is the wrong call when the reason you chose a provider is a capability the others lack, since the layer's ceiling is exactly the thing you were buying. It is the right call when you genuinely run multiple backends in production — for failover, for cost routing, for regional requirements — and even then the honest framing is narrow: a mapping layer plus a classification layer plus consistent observability, deliberately not a universal model API. If what you want is the agent-level machinery on top, that is a different decision with its own trade-offs, and comparisons such as Anthropic Agent SDK vs OpenAI Agents SDK vs Vercel AI SDK are the right place to start rather than building it yourself.

Whether the claim is a vendor's or your own, then, it covers a schema and stops there. The things that decide whether an integration survives its third week — what a field defaults to, how text is counted, where usage arrives in a stream, what a 429 means, what happens after headers are sent — are outside every such claim by construction, and they are all measurable in about a minute with the battery above. What makes a shared layer survivable on top of that is a written rule: reach through the interface when you need a capability that exists on one provider and you have accepted being coupled to it, do it through the escape hatch rather than around the layer, and record the coupling next to the call. That is what keeps the answer to "can we use this new provider feature?" from being either "no" or "yes, secretly".

Everything in this topic has been diagnostic — how to see what your client sends, what it retries, how it frames a stream, and how each vendor names a failure. The next step is ordinary engineering work with those instruments in hand: capture the wire for the calls your service already makes, run the battery against the provider you are on today, and keep the record where the next person will find it.