Measuring provider equivalence

Adding a second provider to a fallback chain is cheap once a multi-provider abstraction exists: a second adapter, a second set of credentials, one more entry in the ordered list of places a request can be served from. What that work establishes is that the second vendor can be called. Equivalence is a stronger claim — that the second provider produces the same shape of output as the first on your prompts, at a rate your parser can live with — and a compiling adapter is not evidence for it.

This chapter is about measuring that claim before a failover tests it for you. The differences that matter cluster around structured output — constraining a model to emit JSON matching a schema you supply — and around tool calls, and they are measured with a suite drawn from production rather than from imagination.

The differences that survive a good abstraction#

Normalising messages and tool schemas across vendors removes the differences in how a request is expressed. It does not remove the differences in how each vendor's model behaves, and a handful of those are systematic enough to plan for.

Structured output is the largest. Vendors expose constrained generation through genuinely different mechanisms — a response-format flag that guarantees syntactic JSON, a forced tool call whose input schema does the constraining, a grammar or regex constraint applied at decode time, or nothing at all beyond prompt instructions. An adapter that maps your one output_schema concept onto a strict mechanism for the primary and onto prompt instructions for the secondary is not wrong, exactly, and it will produce prose wrapped in an apology on the requests where the model is uncertain. That is where a 30% failure rate comes from, and it is invisible until the secondary carries real traffic, because ten easy prompts are precisely the ones the model is not uncertain about.

The gap between those two mechanisms is a difference in kind, not in strength, and it is the step this whole chapter rests on. A model generates by scoring every candidate for the next piece of text and something then picks one. A decode-time constraint sits at the picking step and removes every candidate that would make the output stop matching the schema, so a closing brace arrives because no alternative was selectable — validity is a property of the machinery rather than of the model's behaviour on this input. A prompt instruction reaches none of that machinery. It changes the text the model is continuing, which changes the scores, which makes valid JSON more likely; and "more likely" is a quantity that competes with everything else the model might do. On a document where the answer is plain, the JSON continuation wins by a wide margin. On a document where a field is genuinely absent or contradictory, the continuations that explain the difficulty become competitive, and one of them wins often enough to show up in a parser's error log.

Two consequences follow, and the second is the one that ruins small samples. The first is that the secondary's failure rate is not a constant of the model: it is conditioned on how hard your inputs are. The second is that anyone choosing ten prompts by hand chooses ones they can check, which means ones with a clear answer, which means precisely the population where the two mechanisms agree. The sample is not merely small. It is drawn from the part of the distribution where the difference being measured does not occur.

Tool-call encoding is the second:

// Primary: arguments as an object
{"type": "tool_use", "name": "lookup_contract",
 "input": {"contract_id": "C-8841", "clause": "renewal"}}

// Secondary, through the same normalising adapter
{"type": "tool_use", "name": "lookup_contract",
 "input": "{\"contract_id\": \"C-8841\", \"clause\": \"renewal\"}"}

Normalising that in the adapter takes about six lines and it is the right fix. It is not the right diagnosis. The defect is that a provider was declared a fallback without anyone measuring whether its output was equivalent, and patching the one difference that happened to show up first leaves every other difference in place, waiting for a failover long enough to expose it.

Ten prompts read by a human answer the question "can this model do this task at all", which is rarely in doubt. Thirty per cent of ten is three, and three odd-looking answers out of ten read as normal variation, so a sample that size cannot measure a 30% failure rate.

Underneath the adapter work sits a belief that is worth naming because it is the reason nobody measures: that once the schema matches on both sides, the two providers are interchangeable. It is tempting for a good reason. The schema is the contract, the adapter type-checks against it, the request serialises, the response deserialises, and every tool an engineer normally uses to establish that two things are compatible reports that they are. What those tools are checking is the shape of the message, and the shape of the message is the part both vendors agreed to implement. What differs is which text the model is likely to produce inside that shape, and there is no type in any language that constrains a probability distribution over continuations. Compilation is evidence that the request will be accepted. Nothing about it is evidence about the answer, and the only instrument that reaches the answer is a measurement.

A suite that is representative rather than sampled#

The word doing the work is representative. Build the suite from production traffic rather than from imagination: take a few weeks of real requests, strip or pseudonymise anything sensitive, and stratify the sample so that the shapes that are rare in volume but expensive in consequence are over-represented rather than absent. Three hundred prompts is a workable size — large enough that a 5% difference is visible, small enough to run against three providers in about twenty minutes and cheap enough to run on every model version change.

Two different things are being bought by that number, and confusing them is how suites end up either uselessly small or unaffordably large. Size buys resolution: a rate measured over a few hundred cases wobbles by a couple of points from run to run, so a five-point gap between two providers stands clear of the wobble and a one-point gap does not, and no amount of stratification fixes a suite too small to separate the two. Stratification buys coverage: it decides which rate you are measuring, and a suite of three hundred cases that contains no redlining prompts measures a refusal rate of zero on redlining with perfect precision. The first is arithmetic and the second is judgement, which is why the sampling is worth arguing about with whoever owns the feature and the count is not.

Each case carries its inputs and the assertions that make a response acceptable, and the harness scores four things per provider.

@dataclass
class Case:
    id: str
    messages: list[dict]
    tools: list[dict] | None
    schema: dict | None          # JSON Schema, if the output is structured
    must_contain: list[str]      # deterministic facts the answer needs
    judge_rubric: str | None     # for free-text answers

def score(case: Case, provider: Provider) -> Result:
    t0 = time.monotonic()
    resp = provider.complete(case.messages, tools=case.tools, schema=case.schema)
    latency_ms = (time.monotonic() - t0) * 1000
    return Result(
        case_id=case.id,
        schema_ok=validate(resp.content, case.schema) if case.schema else None,
        facts_ok=all(f in resp.text for f in case.must_contain),
        quality=judge(resp.text, case.judge_rubric) if case.judge_rubric else None,
        refused=resp.stop_reason in ("content_filter", "refusal"),
        truncated=resp.stop_reason == "max_tokens",
        latency_ms=latency_ms,
        input_tokens=resp.usage.input_tokens,
        output_tokens=resp.usage.output_tokens,
    )

Schema adherence and fact coverage are deterministic and are worth more than the judged quality score, because they are the failures that break a parser rather than mildly disappoint a reader. Judged quality is still worth having for free-text features, with the usual caution that a judge is itself a model and needs a human-labelled calibration set before its numbers mean anything. The token counts are there because cost per request, not price per million tokens, is the number you compare.

Run on a 300-case suite, the results for one contracts extraction endpoint looked like this:

Every row in that table is a decision a team would otherwise have made blind. The truncation rate came from the secondary defaulting to a 4,096-token output cap where the primary allowed 8,192, and a JSON object cut off mid-key fails validation in a way that looks identical to the model ignoring the schema. The token counts differ by 13.6% on identical inputs because the tokenisers are different — each vendor splits text into billable units its own way — which narrows the apparent cost advantage and, more importantly, means the longest 2% of contracts fit in the primary's context window, the total number of tokens a model can hold in view at once, and not in the secondary's. The refusal rate is the one that reaches product rather than engineering: 4.1% on a corpus of contract redlining is a hundred documents a week coming back as a polite decline, and no amount of adapter work changes it.

The token row is the one people skim, and it is the row that quietly invalidates the comparison sitting under it. The whole suite sent the same characters to both providers, and the primary counted 412k tokens where the secondary counted 468k: fifty-six thousand more units of billing for text that was byte-for-byte identical, which against 412k is the 13.6%. A token is not a property of your document. It is a unit each vendor defines by choosing where to cut text up, and tokenisation — that cutting-up step, performed before the model sees anything — is a component each vendor built separately. That makes the published price per million tokens a rate quoted in two different currencies, and comparing the two rates directly is meaningless in the same way as comparing a price per litre to a price per gallon. Cost per request, computed from tokens each vendor counted on your own traffic, is the only figure that resolves the units, and it is the reason the table's bottom row is expressed that way rather than as a price.

The same fact reappears as a hard limit, which is the part that cannot be absorbed by budget. A context window is measured in that vendor's tokens, so the same contract occupies a different fraction of it on each side. Documents near the top of your length distribution are therefore not merely more expensive on the secondary — some of them do not fit, and a request that does not fit is not a degraded answer but a rejection. That is the mechanism behind "the longest 2% of contracts", and it is worth checking against your own length distribution rather than the suite's average, because an average is exactly the statistic that hides it.

Tool loops, empty results and turn counts#

Everything scored so far is a single request producing a single response, and a large share of production traffic is not that. A tool-using feature runs a loop: the model asks for a tool, your code executes it, the result goes back, the model continues until it answers. Providers differ least at the first turn of that loop and most at its awkward points, which is exactly the part a ten-prompt check never reaches.

The empty result is the classic. When a search tool returns [], one model reports that nothing was found and stops, a second re-invokes the same tool with slightly reworded arguments two or three times before giving up, and a third writes a plausible answer as though the tool had returned something. All three are syntactically valid, none of them raises an error, and only the first is what your product wants. Tool errors split the same way: one model surfaces "the contract service returned 503" to the user, another treats the error string as data and reasons over it. Parallel calls split a third way, with one vendor emitting two tool blocks in a single assistant turn where another emits one and waits for the result before asking for the second.

That last difference is invisible per response and expensive per query. On a 40-case tool-loop subset the primary averaged 2.3 model turns to resolve a query and the secondary averaged 3.8, which turned a per-call p50 of 1.2 seconds — the median call, and the secondary's advantage in the table above — into a per-query p50 of 4.6 seconds against the primary's 4.4. The provider that looked faster on single calls was slower on the thing users actually wait for.

The multiplication is the point, so it is worth doing rather than asserting. What a user waits for is a query, and a query is however many model calls the loop needs, one after another, because each call cannot start until the previous tool result is in hand. So the time a user experiences is the per-call latency multiplied by the number of calls, not the per-call latency on its own. The primary's median call takes 1.9 seconds and it needs 2.3 of them, which is just under four and a half seconds. The secondary's median call takes 1.2 seconds and it needs 3.8 of them, which is a little over four and a half. Those two products land a fraction of a second apart and on the wrong side: the faster call lost because it was made half again as often, and the extra turn and a half consumed the whole of the seven tenths of a second it was saving on each one. Rounding the two products to a tenth of a second gives the 4.4 and the 4.6 in the sentence above. Your own tool execution time adds to both sides and adds more to the side that calls tools more often, so this arithmetic understates the gap rather than overstating it.

Suites that only measure single requests report the opposite of the truth here, so include loop cases, and include the ones where a tool returns nothing, returns an error, and returns more rows than fit in the context.

Version pinning in the chain and in the suite#

vendor-b-large is usually an alias, and vendors move aliases. If your chain names an alias and your suite measured an alias, every number in that table has an expiry date you do not control, and on the day the alias moves your fallback silently becomes an unmeasured provider while your configuration remains identical character for character. Name the dated version in both places, treat a version change as an event that requires a suite run before it is allowed into the chain, and keep the previous version pinned until that run passes.

The cost of doing this is small enough to remove the argument. Three hundred cases across three providers is about twenty minutes of wall time and roughly six dollars of inference at the prices above, which is less than the meeting in which someone proposes running it quarterly instead. The expensive part is maintaining the suite's ground truth as your product's definition of a good answer shifts, which is a job for whoever owns the feature and not for whoever owns the gateway.

One caution about the judged half of the scoring. A judge is a model, its agreement with human labels needs measuring on a set of fifty hand-scored cases before anyone quotes its numbers, and it should never be a model from one of the vendors being compared, because a judge scoring its own family's outputs against a competitor's is a conflict you can avoid for free. Where the output is structured, prefer the deterministic checks entirely; the judge is a tool for prose, where nothing else works.

The verified block on a chain entry#

The behavioural fix that follows from all this is small. A chain entry stops being a provider name and becomes a provider name plus a claim about what was measured and when:

{
  "provider": "vendor_b",
  "model": "vendor-b-large-2026-03",
  "verified": {
    "suite": "extraction-v4",
    "run_id": "eval_2026_04_11_a",
    "schema_adherence": 0.968,
    "notes": "refuses 4.1% of redlining prompts; cap raised to 8192"
  }
}

An entry without a passing run against the current suite is allowed in the chain only below the tier where output is treated as authoritative, and the same rule fires when either side changes: a provider's model version bump invalidates the run, because the suite is measuring a model's behaviour and the model is now a different one. Teams that skip this end up re-litigating from scratch every time someone asks whether the secondary is safe to promote.

A narrow claim about a specific set of prompts#

A suite is not a benchmark, and reading it as one leads to the wrong conclusion in both directions. A provider that scores worse here may be better in general and simply worse at your prompts; a provider that scores better may have been tuned by coincidence to the pattern in your traffic. The suite makes a narrow claim about a specific set of prompts, and that narrowness is the point.

Three hundred sequential calls say nothing about capacity#

Every number above was produced by 300 sequential calls, which describes how the secondary behaves at roughly zero load rather than how it behaves carrying your full peak. Rate limits, queueing behaviour under sustained concurrency and the vendor's own degraded serving tiers are outside what any offline suite can reach.

A passing suite does not establish that two providers are interchangeable. It establishes that their differences are known, quantified and bounded, which is a much more useful thing to have. The secondary in the table above is genuinely worse: it refuses fourteen times as often and it misses deterministic facts on 6.4% of cases against the primary's 2.9%.

Knowing that is what makes the next decision possible, which is whether an answer with those properties is better than no answer at all. That question does not have one answer for a whole system — it has one per feature, and it depends on what acts on the answer afterwards.