Measuring the capability loss

A compatibility matrix is the table a router or gateway publishes saying which provider capabilities it supports. The word does a lot of load-bearing work there, and the levels underneath it are worth separating, because a matrix almost never says which one it is claiming. This chapter is about obtaining the number the matrix does not give you: not whether a feature is listed, but whether it survives your requests, and what its absence costs per month.

The measurement is three steps and takes about a day — replay, diff, evaluate — and it sits deliberately outside your own contract, comparing raw provider payloads to raw router payloads with no shared type in the middle. That positioning is the whole reason it finds anything, and it is also why the instinct that follows an incident like chapter 3's — write a shared conformance suite that every implementation must pass — does not address this class of problem at all.

The positioning follows from what the previous chapters established about mapping code. Any type in the middle of the comparison is a selection step, and a selection step has two outcomes per field, carry and drop, with no third outcome for "I did not recognise this". A comparison performed after a normalized type has run cannot report a field the type has no slot for, because that field stopped existing before the comparison began. So the diff has to happen on the payloads as they arrived, in whatever shape the provider and the router each chose, and every convenience that would make the two easier to compare is a convenience that removes the evidence.

Four things "supported" can mean#

The four levels are ordered, in the sense that each one implies the one above it and none implies the one below. That ordering is what makes the top row almost free to satisfy, and the reason is worth following rather than asserting. A request body is JSON, and the receiving side parses it into whatever structure its own code declares. The near-universal convention for keys the structure does not declare is to ignore them rather than to reject the request, and a surface that presents itself as compatible with somebody else's API has a strong reason to keep that convention: it is receiving requests from clients written against several different vendors' SDKs, each of which sends fields the others do not have, and rejecting the unrecognised ones would break callers for no benefit. So tolerance of unknown fields is the design most likely to be there. A field the implementation never reads therefore produces exactly the same outcome as a field it honours perfectly, which is a request that does not fail — and "the request with this field does not 400" is satisfied by both.

That is why the top row's last column says the test tells you nothing about behaviour. It is not a weak test in the sense of being noisy. It is a test whose passing condition is met by the case it was meant to rule out.

Prompt caching is the capability that most often sits at "accepted" while being documented as supported, and the gap is not dishonesty: a router does accept cache-control input and does front providers that cache. Nothing in that claim is a promise about where the cache breakpoint — the marker saying how far along the prompt the stored prefix should run — actually lands, and the difference between a breakpoint after a system preamble and one after forty worked examples is the difference between caching a few thousand tokens and caching thirty thousand.

Extended thinking typically sits at "round-tripped", which is one level better and still not enough. If the blocks come back, the field exists; whether the cryptographic signature survived, whether the intermediary forwards the blocks on subsequent turns, and whether the model's own reasoning budget was applied are three further questions that the presence of a reasoning key does not answer. Chapter 3's failure mode lives in that gap.

Replay, diff, evaluate#

Sample real requests from production — a few hundred is enough, chosen to cover the shapes you actually send rather than the ones that are easy to construct — and store them with whatever redaction your policy requires. Replay each one through both paths. Then compare at two different altitudes: structurally, field by field on the raw payloads, and behaviourally, by running your evaluation set through each.

import json, asyncio
from deepdiff import DeepDiff

async def replay(sample: list[dict]) -> list[dict]:
    rows = []
    for req in sample:
        direct = await call_direct(req)      # provider SDK, raw response as dict
        routed = await call_router(req)      # OpenAI-compatible client at the router
        rows.append({
            "id": req["id"],
            "diff": DeepDiff(direct, routed, ignore_order=True,
                             exclude_regex_paths=[r"\['id'\]", r"\['created'\]"]).to_dict(),
            "direct_usage": direct["usage"],
            "routed_usage": routed["usage"],
            "direct_text": text_of(direct),
            "routed_text": text_of(routed),
        })
    return rows

Two details make the diff readable rather than noise. Hold sampling constant across both paths — same model, same temperature, same max tokens — and exclude the fields expected to differ, which is what exclude_regex_paths is doing for ids and timestamps; without that exclusion every response differs and the report says nothing. Then aggregate rather than eyeballing: 500 individual diffs is unreadable, while a count of how many responses differ at each JSON path is a page. If your requests carry customer data, the sample has to be redacted or the replay has to run inside whatever boundary the data already lives in, and it is easier to get approval for a one-afternoon offline replay of redacted requests than for permanently mirroring live traffic to a third party you have not yet contracted with — which is also the honest reason to do this before a security review rather than during it.

The structural diff is the part teams skip, and it is the part that finds things nobody thought to look for. Aggregated by the paths that differ, it produces a short, blunt report:

dictionary_item_removed   root['usage']['cache_read_input_tokens']        500/500
dictionary_item_removed   root['usage']['cache_creation_input_tokens']    500/500
dictionary_item_removed   root['content'][0]['signature']                 500/500
dictionary_item_removed   root['content'][2]['input_schema_strict']       118/500
values_changed            root['stop_reason']  'max_tokens' -> 'length'    31/500
values_changed            root['usage']['input_tokens']  ~3.5k -> ~30k    500/500
type_changes              root['content'][0]['thinking']  str -> dict      12/500

Every line there is a fact rather than an opinion, and each one resolves into an action. The missing signature means multi-turn reasoning will not be reattachable, which is chapter 3's first loss appearing before it costs anything. The input_tokens jump from 3.5k to 30k is a cache miss stated in the unit that gets billed. The stop-reason rename is cosmetic and belongs in the pile topic 2.2 taught you to discard. The twelve responses where thinking changed type are a real bug in someone's normalization that would otherwise have been found by a crash in production on a rare path.

What a diff line costs in currency#

Converting the interesting lines into money is quick and worth doing explicitly, because "caching is not working" does not fund a decision and a monthly figure does. Take an agent with a 30k-token system prompt running about 4,000 calls a day. At $3.00 per million input tokens, sending that prompt uncached costs $0.090 per call. A cache read is billed at roughly 10% of the input rate, so $0.009, and a cache write at roughly 1.25×, so $0.1125, paid once per cache lifetime rather than once per call. With a five-minute TTL — the time to live after which the provider discards the stored prefix and the next call has to write it again — and traffic spread across the working day, the agent writes the cache about 290 times a day and reads it on the remaining calls.

Each of those figures is one step from the one before it, and they are worth walking because the conclusion is a monthly number somebody will be asked to defend. A 30k-token prompt is three hundredths of a million tokens, and three hundredths of $3.00 is the $0.090. The read rate is a tenth of the input rate applied to the same tokens, so it is a tenth of $0.090; the write rate is 1.25 times the input rate on the same tokens, so it is 1.25 times $0.090. Nothing about the prompt changes between the three, only the rate applied to it, which is why one multiplication produces all three prices.

The write count is the only figure that is not a multiplication. An entry lives five minutes from the moment it is written and traffic arrives steadily, so at most one write can happen per five-minute window, and at this call volume every window has calls in it, so one does happen in each. The daily write count is therefore the number of five-minute windows the traffic spans — a little under three hundred of them, which is where the 290 comes from. The remaining calls read the entry, and 4,000 calls less the 290 writes is the 3,710 in the second row of the table below. That figure is the one to test if the comparison is ever challenged: it moves with traffic density rather than with traffic volume, so an agent making the same 4,000 calls in two bursts rather than spread across the day would write far fewer entries and the direct column would fall further.

The table's rows are the same three multiplications carried through. The 290 writes at $0.1125 apiece are $32.6; the 3,710 reads at $0.009 apiece are $33.4; adding them gives the $66.0 the system prompt costs in a working day when the cache lands. Through the router none of the calls read an entry, so all 4,000 pay the uncached $0.090 and the day costs $360.0. Multiplying each by 30 puts them on a monthly footing, and subtracting one month from the other gives the gap of $8,820 — a subtraction that is allowed because both columns price the same 4,000 calls carrying the same prompt, and every term the two share has already cancelled by construction.

The gap is $8,820 a month and 1.5 seconds of perceived latency on every interaction, that second figure being the difference between the two time-to-first-token rows. That is a comparison two people can have in one meeting, which it was not while both sides were arguing about whether a matrix entry was trustworthy.

One detail in the left column is worth noticing rather than skipping, because it says something about how fragile this arithmetic is to the traffic shape. The writes and the reads cost almost the same amount per day. A few hundred writes at more than the full input rate come to about what several thousand reads at a tenth of it come to, which means this agent is spending roughly half its cached prompt budget on filling entries. That is not a general fact about caching; it is what this particular arrival rate against this particular TTL produces, and it is the reason a measurement of the router has to be made against your own traffic rather than against a worked example.

The evaluation half#

Structural diffs catch what changed in the payload. They do not catch what changed in the answer, so the second measurement is your existing evaluation set, run through both paths with everything else held constant. Sizes matter here: on a 200-task suite the 95% interval on a single run is roughly ±6 points, so an eight-point gap is suggestive rather than settled, which means running it three times on each path. The variance across those runs is also information, since a routed path with visibly higher run-to-run variance usually indicates non-deterministic routing between models you did not intend to compare.

Where that interval comes from decides how much running it again buys, so it is worth one paragraph. A pass rate is a proportion measured on a fixed number of tasks, and the uncertainty in a proportion falls with the square root of the task count rather than with the count itself. Quadrupling a suite therefore halves the band and does not quarter it. On a suite of two hundred tasks with a pass rate somewhere near four fifths, the typical run-to-run wobble works out at a few points, and a 95% interval is about two of those either side — which is the ±6. The same square-root behaviour applies to repeating the run: averaging three runs narrows the band by the square root of three, not by three, which is why three repetitions turn a suggestive gap into a settled one and would not have turned a two-point gap into anything.

Deciding in advance what result changes your mind is what makes this an experiment rather than a search for justification. A defensible rule reads like this: agree to the migration if the evaluation gap is within the noise band across three runs and the cost delta is under some stated figure; migrate a subset if the gap is confined to one workload while the rest shows no difference; decline for that workload if either the cache or the reasoning loss reproduces. Written before the data arrives, that rule is a commitment. Written afterwards, it is a rationalisation, and everybody in the room can tell which one they are reading.

Why a conformance suite goes green on exactly the wrong thing#

A conformance suite is defined in terms of your contract: it constructs a request in your shape, runs it through each implementation, and asserts on your response type. Every assertion it can make is an assertion about a field the contract has. The losses that matter are, by construction, in fields the contract does not have — there is no assert response.reasoning_survived to write when the type has no reasoning, and no way to assert that a cache breakpoint landed when the type does not model cache breakpoints.

That is not a gap you can close by writing more tests, because it is a property of where the suite sits. A conformance suite answers "do my implementations agree with my contract?" and the answer is yes, which is why it is green. The replay diff answers "what does my contract never see?", and that is the question with a six-week aftermath attached to it. Both are worth having, for different reasons: the suite protects the contract you rely on, and the diff measures what the contract cannot express.

The belief this whole chapter exists to dislodge sits underneath the green tick, and it is not a belief about routers. It is that a shared interface makes two providers interchangeable. The reason it is tempting is that the interface is the part of the system you can see and reason about, and it genuinely does make the code interchangeable — the migration in the scenario above really was four days of work for the traffic that could take it. What it cannot do is make the capabilities interchangeable, because a capability is something the provider does and the interface is only a description of what you agreed to name. Every check that lives at the level of the interface — the type checker, the adapter tests, the conformance suite, the compatibility matrix — is asking about the description. The replay is the only one of them asking the provider.

Both measurements are also cheap in the sense that matters: an afternoon of engineering, no production change, no commitment made. What they produce is a list of capabilities that demonstrably do not survive the shared shape, each with a cost or a score attached. That list is not an argument against a router, since traffic that uses none of those capabilities passes the same test with a zero-line diff. It is a list of exceptions, and the remaining question is what an exception looks like in code, given that the obvious answer of adding it to the shared interface is the one chapter 2 already ruled out.