The platform serving stack

A cloud model platform — Amazon Bedrock, Microsoft's Azure OpenAI Service, Google's Vertex AI — is a cloud provider's own API sitting in front of foundation models the provider did not train, so that a model built by Anthropic, Meta or Mistral is called through the same account, the same invoice and the same access controls as a company's object storage and its databases. What it solves is a purchasing and governance problem rather than a modelling one. Buying from a model vendor directly means a new supplier contract, a new security review, a new credential to rotate, a second invoice and a second data processing agreement, each of which takes weeks in a company of any size; the cloud provider's existing contract has already paid all of them. Spend draws down a budget finance has committed, access is granted with the provider's IAM — identity and access management — roles the team already administers, and the usage lands on an invoice somebody already reconciles.

An engineer meets that entire arrangement as three lines of configuration: a different base URL, a platform credential in place of the vendor's API key, and a model id in the platform's own namespace. That reading is not wrong about the part it names. The weights — the trained parameter file that is the model itself, and the only thing a model name identifies — are the same checkpoint behind both identifiers, loaded onto the same class of accelerator, and nobody is quietly serving a distilled variant to cloud customers. What the name does not cover is everything between the TLS handshake and those weights: the serving stack that authenticates the caller, renders the prompt, screens it, meters it and prices it.

The request that reaches the weights is not the one you sent#

The reason there are two implementations rather than one starts with what a trained model is. It is a file of numbers, plus a function that consumes a flat sequence of token ids and returns a score for each candidate next token. That function has no notion of a customer, a credential, an invoice, a role marker or a policy category, because none of those things is expressible as a position in a token sequence. Every one of them has to be supplied by code wrapped around the function.

A model vendor writes that wrapping code once, for its own customers, its own API keys and its own billing system. A cloud platform cannot reuse it. Its customers authenticate with the cloud's identity system rather than with the vendor's keys, they are billed on the cloud's invoice rather than the vendor's, and their content is governed by the cloud's contract rather than the vendor's usage policy — so the platform writes its own wrapper. That is the causal step worth holding: identical weights do not imply an identical request path, because the request path is the part neither company could share even if both wanted to. Two independent bodies of code end up on either side of one identical function, and the model name names the function.

Both stacks do the same broad set of jobs. They authenticate the caller, account for quota, render structured messages into the flat token sequence the model was actually trained on, optionally classify the input, run inference, optionally classify the output, meter usage, and translate the result back into JSON. The vendor operates one implementation of that pipeline. The platform operates a second one, written by a different team, shipped on a different schedule, and tuned for a different default customer — typically an enterprise that has asked for conservative content handling and predictable billing granularity rather than for the vendor's own defaults.

The only box the model name pins is the one in the middle. Everything to the left of it decides what the model sees, and everything to the right decides what you are told and what you pay. Each of those boxes has a failure mode that is invisible in a diff of application code.

Safety defaults belong to the platform, not to the model#

The model has trained-in refusal behaviour, which travels with the weights and is therefore identical on both stacks. Sitting in front of and behind it, on the platform, is a separate classifier that the platform operates as a product feature, with categories such as hate, sexual content, violence and self-harm, a severity threshold per category, and a default threshold usually set to block at medium severity and above. A vendor's own API generally ships with that layer either absent or set far more permissively, because the vendor is relying on the model's own behaviour and on your acceptance of their usage policy.

The symptom this produces is narrow and therefore easy to miss. A claims-processing service handling 8,400 requests a day found that 0.4% of them — about 34 a day — came back from the platform with an empty completion and a finish_reason of content_filter, where the vendor's endpoint had answered normally. Four in a thousand of 8,400 is thirty-three and a bit, which is where the 34 comes from, and the smallness of that number is the whole difficulty: it is far too few to move any average anybody was watching and far too many to be noise. The blocked slice was not adversarial traffic. It was descriptions of workplace injuries, which score as violence to a general-purpose classifier and are the literal subject matter of the product. Nothing in the application logged that as an error, because the HTTP status was 200 and the body was well formed; the downstream parser saw an empty string and wrote a null summary.

That last sentence is doing more work than its length suggests, and it is worth walking the chain rather than asserting it. The filter is a separate classifier from the model, so it does not decline in prose the way a model does; it either passes the text through or withholds it. Withholding the text is not a transport failure — the connection succeeded, the platform processed the request, and it has a well-formed answer to give, which is that there is no completion. HTTP has no status code for the request worked and produced nothing on purpose, so the platform returns 200 and puts the verdict in a field. The verdict therefore lives in finish_reason and nowhere else. Any client that reads the text and ignores the reason cannot distinguish a blocked request from a request the model answered with silence, and neither can any error rate computed from status codes.

Most platforms let you turn the categories down or off, sometimes only after an approval process taking days and requiring a written justification of the use case. It helps to know why the layer is there at all. Cloud platforms are reselling models to customers who have never signed the model vendor's acceptable-use policy and whose contracts run with the cloud provider instead, so the platform carries the reputational and legal exposure for whatever comes out of the endpoint. A configurable classifier in front of the weights is how that exposure gets bounded, and setting it conservatively by default is the only choice that is safe for a platform serving tens of thousands of unknown workloads. The same reasoning explains why the threshold is a per-deployment setting rather than a per-request one on several platforms: it is a compliance control, and controls any caller can disable in a request body are not controls.

Tokenizers and chat templates carry revisions the name does not#

The model id names a checkpoint. The chat template — the exact string of special tokens, role markers and whitespace that turns [{"role": "user", "content": "..."}] into a token sequence — is packaging around that checkpoint, and packaging gets revised. A platform that vendored the template at integration time can be a revision behind, which shows up as small differences in how system prompts are joined, whether a trailing newline is emitted before the assistant turn, or how tool definitions are serialised into the prompt.

Why any of that should change the answer needs saying before the measurement, because the reader who skips it is left with a rule and no way to predict the next case. The weights compute a function of a token sequence and of nothing else. The roles, the message boundaries and the tool definitions in your request body do not reach the model as structure; they reach it as tokens, in whatever arrangement the template chose. Two templates that differ by one newline therefore hand the model two different sequences. The model is not being inconsistent when it answers them differently — it is being exactly as consistent as it always is, on inputs that are not the same input.

Counting tokens is the cheapest way to see it. The same 4,100-character prompt, sent to both endpoints, came back reporting 1,007 input tokens from the vendor and 1,032 from the platform, a 2.5% difference that is entirely template overhead rather than anything about the text. The gap is twenty-five tokens, and twenty-five against roughly a thousand is the two and a half percent; since the character count was identical, every one of those twenty-five is a role marker, a separator or a whitespace decision rather than a word anybody wrote.

Two and a half percent is not interesting on its own. It becomes interesting at the boundary, and the boundary arithmetic is worth doing once. A prompt assembled to sit at 199,000 tokens against a 200,000-token window has about a thousand tokens of headroom, which is half a percent of its own size. The excess does not have to reach two and a half percent to break it; it has to reach half a percent. Template overhead is largely per-message rather than proportional to the text, so a long prompt made of many messages accumulates it per turn and a long prompt made of one message may not — which is why the honest form of the rule is that a prompt sitting inside half a percent of a context ceiling fits on one stack and is rejected on the other, and only a measurement on your own message shape says which. The same reasoning explains the softer failure: a retrieval step that packs context up to a computed budget will overflow on whichever stack counts higher, because the budget was computed with one tokenizer and spent against another.

The subtler effect is on output. Formatting differences in the rendered prompt are exactly the kind of distributional shift that moves behaviour on ambiguous inputs — few-shot examples separated by one newline instead of two, a system message concatenated rather than placed in its own slot, a tool schema serialised with keys in a different order. Average quality does not move measurably. The specific inputs sitting near a decision boundary do.

There is a practical consequence for anything that keys off token counts. Prefix caches are addressed by an exact prefix, so a template revision invalidates every cached prefix at the moment the platform ships it, and a workload running at a 92% cache hit rate can drop to zero for an hour with no deploy on your side. Cost estimators built on the vendor's tokenizer will be systematically low against the platform's meter. Retrieval systems that pack context to a computed budget need the budget computed with the same tokenizer that will do the billing, which means calling the platform's own count endpoint rather than a local library pinned at whatever revision was current when the library shipped.

Endpoint defaults decide how long an answer is allowed to be#

Every serving stack has to pick a value for the parameters you did not send. The direct API and the platform pick different ones, and the parameter that bites hardest is the cap on output tokens.

# Vendor SDK: max_tokens is required, so the value is in your code and reviewable.
resp = anthropic_client.messages.create(
    model="claude-sonnet-4-5-20250929",
    max_tokens=4096,
    messages=[{"role": "user", "content": prompt}],
)

# Platform SDK: inference config is optional, and the omitted field takes a
# platform default that is frequently far lower than the model's capability.
resp = bedrock.converse(
    modelId="anthropic.claude-sonnet-4-5-20250929-v1:0",
    messages=[{"role": "user", "content": [{"text": prompt}]}],
    # inferenceConfig omitted -> maxTokens defaults to 512 on this path
)
print(resp["stopReason"])  # "max_tokens" on any answer longer than ~380 words

The difference between those two snippets is a design choice, not an accident, and knowing which choice was made tells you where to look on any platform. The vendor SDK makes the cap a required argument, which means the value is in your repository, appears in a diff and can be reviewed. The platform SDK makes the whole inference configuration optional, because an optional block is what lets a console generate a two-line example — and an omitted field has to take some value, so the platform supplies one. A default chosen to make a console snippet work is chosen to be cheap and safe rather than to match the model's capability, which is why it sits so far below it.

The comment on that last line is worth converting into something checkable. At roughly three quarters of an English word per token, a 512-token ceiling is around 380 words, which is a long paragraph and not a document. Any answer the model would have written past that point is cut, mid-token, wherever the counter runs out.

A migration that copies the platform's console-generated snippet inherits that default, and the failure it produces looks like a model regression: summaries that stop mid-sentence, JSON objects missing their closing brace, a downstream json.loads throwing on roughly the 12% of requests whose answers ran long. The stopReason field says exactly what happened on every one of those responses, and almost no application logs it.

The reason this reads as a model regression rather than as a configuration change is worth being explicit about, because the misreading costs weeks. A truncated answer is not a wrong answer with an obvious defect in it. It is a correct answer with its ending removed, so it is fluent, on-topic and well-formed right up to the character where it stops. Everything a reviewer looks at when they suspect a model — the wording, the reasoning, the first paragraph — is intact. The only evidence that the model was not the cause sits in the one field nobody logged, which is why the second thing to do after a platform cutover, before opening any evaluation, is to count responses by stopReason and see whether a category that used to be empty is now carrying traffic.

Output length is not the only generation default that differs between the two stacks in this way. Temperature, top_p and the stop sequences applied when you name none of your own are set independently by each stack, and each of them can change behaviour on a request whose body you never touched. The consequences are less dramatic than a truncated answer, because none of them produces a well-formed response with its ending removed, but the mechanism is identical: a value you did not send, chosen by whoever built the endpoint.

The invoice counts a unit you did not choose#

Token prices are usually quoted per million on both stacks and often match to the cent, which invites the assumption that the cost model transferred. What can differ is the unit being counted and the rounding applied to it. Platforms bill in their own metering granularity — a common one is to round each request's input and output up to the next 1,000 tokens — and a workload of many small requests pays for that rounding on every call.

Both rows of that table are one operation applied twice, and working the first through makes the second predictable. 8,400 requests a day at a mean of 1,050 input tokens is 8.82 million tokens actually sent, which is the vendor column. Under rounding to the next thousand, a request carrying 1,050 tokens is billed as two thousand, because it is above the first thousand and the meter has no smaller step to stop at. Eight thousand four hundred requests at two thousand billed tokens each is 16.8 million, and 16.8 against 8.82 is a little under twice as much — the +90% in the delta column.

The output row is worse, and the reason is the general rule hiding inside these two figures. A mean of 310 output tokens rounds up to a full thousand, so each request is billed for something over three times what it produced, and that is the +223%. What sets the penalty is not the size of the workload but the ratio between the rounding step and the typical request. A request already sitting just under the step pays almost nothing extra. One sitting at a third of the step pays three times over. Rounding does its damage to small requests, and a chat-shaped or classification-shaped workload is made of nothing else, which is why this line item surprises exactly the teams whose per-request costs looked negligible.

One condition belongs on that arithmetic, because it is the kind of claim that travels further than it should. Both multipliers are computed from a mean, and rounding is applied per request. A workload whose input sizes cluster tightly around the mean pays what the table says. A workload whose sizes straddle a thousand-token boundary, with some requests below it and some above, pays a different multiplier on each group, so the honest figure has to be computed over the distribution rather than over its average. The direction of the error is never wrong — rounding up can only ever cost more — and the magnitude is only right when the spread is narrow.

Cache pricing is the other axis worth naming, because the discount for a repeated prefix is a product decision rather than a property of the model, and a platform with a shorter cache TTL or no cache read discount at all turns a heavily prefix-cached workload into a much more expensive one at identical token counts. Rounding is not universal and not always at 1,000 tokens; some platforms meter per token exactly as the vendor does, some meter in characters for certain model families, and some apply a minimum billable request. The unit is a field in the platform's pricing documentation rather than something the model name implies, and the way to check it is to send a known workload and reconcile the resulting line item against your own usage counters. The first invoice is where the difference becomes legible, and the first invoice arrives after the cutover.

None of this makes the platform's copy of the model worse#

The weights are the weights, and on the great majority of requests the two stacks return answers of indistinguishable quality, which is precisely why the differences survive a migration. This is also not the same problem as a model version bump, where the vendor changes the checkpoint under a floating alias and behaviour moves for everyone; here the checkpoint is fixed and the surrounding stack moves. And it is not a wire-protocol problem of the kind covered under provider SDKs, where the fix is a parameter rename or a different header. Every difference in this chapter survives a perfectly correct port of the request format.

The differences also do not announce themselves, and each one surfaces in a different part of the organisation. No single dashboard sees more than one of them, which is why a migration reviewed by one team is reviewed by the team least likely to be told about three of the four.

It is worth naming why the opposite belief is so easy to hold, because nobody arrives at it carelessly. A model name really is the strongest identifier in this system. It pins a checkpoint exactly, it survives across providers, and for the whole history of software before hosted inference the name of a component did determine its behaviour, because the name and the running code were the same artefact. The reasoning fails at one step and only one. A hosted model is not a component you run; it is a function somebody else runs on your behalf, and the name identifies the function rather than the run. Every row of the table above is a property of the run.

A model name is a promise about weights and about nothing else on the request path. The four properties in that table are the ones a cloud platform sets on your behalf, and every one of them is readable before a cutover: the filter categories and thresholds are in a deployment configuration, the template revision is visible in a token count, the endpoint defaults are in the API reference, and the metering unit is in a pricing page. What makes them expensive is not that they are hidden but that nothing in your code changes when they differ, so there is no diff to review and no error to log.

The obvious response to a list like that is to run the evaluation suite against both endpoints, confirm that the scores match, and treat matching scores as proof that the swap is like for like. That response is reasonable and it is what most teams do. It is also structurally incapable of catching two of the four rows in the table, which is what the next chapter works through.