Cloud model platforms

Read these first

• Provider SDKs and the differences that actually matter

What you will know

• Deploy the same model through a cloud platform and through the vendor directly

  Identical model names can hide differences in tokenizer versions, safety defaults, endpoint behavior, and billing units.

• Run a quota exhaustion test and capture the resulting errors, retry behavior, and recovery path

  Rate limits may apply separately to requests, input tokens, output tokens, or regions, and naive retries can amplify an outage.

• Compare the configured region with the provider's residency documentation and the request's audit evidence

  A regional endpoint may use failover, logging, support access, or subprocessors outside the location the application team assumed.

• Find the region a request was actually served from

  Response metadata may be incomplete, so production verification can require provider trace records or an auditable test rather than the endpoint name alone.

• Calculate whether provisioned throughput is justified from measured traffic, latency targets, and burst patterns

  Reserved capacity can improve tail latency while wasting money during idle periods, and it may not absorb sudden demand beyond the purchased allocation.

Chapters

1. 1

   The platform serving stack

   The same model, under the same name, served by a cloud platform instead of by the vendor that trained it. This chapter takes apart the serving stack that sits between your TLS handshake and the weights — the content filter and its defaults, the chat-template and tokenizer revision, the values applied to parameters you did not send, and the unit the invoice counts — and names the production symptom each one produces.

   Read chapter
2. 2

   Validating a platform cutover

   A parity eval scores average quality on a fixed test set, which is precisely where the differences between a vendor endpoint and a cloud platform's are invisible: a rejection rate in the tail, a cache TTL, a metering unit. This chapter works the arithmetic on why the scores match anyway, then builds the instrument that does see them — a shadow run against live traffic — and states the five results that should block a cutover.

   Read chapter
3. 3

   The dimensions of a quota

   A cloud model endpoint enforces several independent ceilings at the same moment — requests per minute, input tokens, output tokens, requests in flight — scoped per model and per region, and sitting under an account-wide total that other teams also spend. This chapter shows how to read a 429's headers to find which dimension bound, why long documents exhaust the token dimension first, and how an account quota, a per-model rate limit and provisioned throughput differ.

   Read chapter
4. 4

   Saturation tests and kill switches

   Driving an account to its quota ceiling is the only way to learn what your client does there, and on a cloud platform that ceiling is shared with every other team on the same subscription. This chapter builds the containment a saturation test needs before its first run: an isolated project or region, an off-peak window taken from account metrics, a bounded ramp, an external watchdog wired to somebody else's error rate, a kill switch, and a notification list.

   Read chapter
5. 5

   Synchronised backoff and jitter

   Throttled callers all receive the same retry-after, all wait exactly that long, and all return in one wave, so a service can stay degraded for minutes after offered load returns to normal. This chapter reads a saturation trace layer by layer, derives why the waves decay geometrically and why the degradation graph has a clean sixty-second period, and shows what full jitter, a capped attempt count and a concurrency limit each fix.

   Read chapter
6. 6

   Regional endpoints and data residency

   A regional endpoint hostname names where a request is received and, under normal capacity, inferred. Cross-region inference profiles, abuse-monitoring retention, support engineer access, the platform's subprocessor chain and your own observability pipeline each move or copy the payload elsewhere, and none of them appear in any response your client sees. This chapter maps the surfaces a hostname does not settle and shows which ones you can close this afternoon.

   Read chapter
7. 7

   Proving the serving region

   Evidence that one specific production request was inferred inside a given geography has to come from the platform rather than from you, because your logs record intent and most responses never name the serving region. This chapter compares the three candidate sources, shows the data-plane invocation record that carries the field, covers what must already be running for the lookup to exist, and separates per-transaction evidence from control-operation evidence.

   Read chapter
8. 8

   Costing provisioned throughput

   Reserved capacity is billed for the time it is held, saturated or idle, which makes it a spending commitment rather than a bigger quota. This chapter measures a workload in the platform's own weighted-token unit, works the arithmetic on a load averaging 15% of a unit with a twenty-minute peak at 300%, derives the utilisation at which reserving beats on-demand, and asks what a one-year term does when a cheaper model ships.

   Read chapter