Automatic prefix caching and breakpoints

Both designs are described as prompt caching, both bill a discounted rate on matched tokens, and both are switched on by default in the sense that no account-level setting stands between you and them. What differs is where the boundary of a cacheable segment comes from. In one regime the provider infers it from the traffic; in the other the caller declares it in the request body, and a request without a declaration is a request that has asked for nothing.

Establishing which regime an endpoint is in is the first thing to do, before any of the design decisions in the chapters that follow, because the levers, the pricing shape and the failure modes are all different. This chapter defines the two and gives a two-call experiment that settles the question for the specific model you are calling.

Two designs, and what each of them asks of you#

Under automatic prefix caching, the provider hashes the leading tokens of every incoming request and looks for a match without being asked. There are no markers in the request body, no configuration, and usually no way to switch it off. Matching happens at a fixed granularity — commonly 128-token increments on hosted APIs, 16-token blocks in the self-hosted engines that pioneered the design — so the reusable span is your true common prefix rounded down to the nearest increment. The saving appears as a discounted rate on the matched tokens, somewhere between half and a quarter of the base input price depending on the model, and there is no premium for the first request that fills the entry.

Under explicit breakpoints, the request body carries a marker on one or more content blocks, and the marker means: everything from the start of the prompt up to and including this block is a cacheable segment. Blocks after the last marker are never cached. The rates are more aggressive at both ends — a read at a tenth of base input, a write at 1.25 times base — and the boundary sits exactly where you put it rather than at an increment the provider chose.

What the marker actually causes is worth spelling out, because the word breakpoint suggests a boundary being observed rather than an instruction being obeyed. Serialization runs from the top of the request. When it reaches the marked block, the provider takes the prefill state computed up to and including that block, writes it as an entry, and charges for the write. With no marker anywhere, that write never happens. An unmarked prompt under this regime is therefore not cached less than a marked one — there is nothing stored to be read, on the first request or the millionth, and that is exactly what the counters report.

The asymmetry in the "worst realistic outcome" row is the practical difference between working in the two regimes. Automatic caching cannot make anything more expensive; a request whose prefix matches nothing pays exactly what it would have paid with no cache in the world. Explicit breakpoints hand you a mechanism that charges a premium up front and refunds it only through later hits, so a badly placed marker is not a missed optimisation — it is a surcharge that has to be earned back.

The mechanism behind that asymmetry is the absence of a write premium and nothing more subtle. Under automatic caching a request that matches nothing is billed at base rate on every token it carries, which is the bill it would have had if the provider ran no cache at all, so the downside is bounded at saving nothing rather than at losing something. The row is a claim about the token bill on that request, which is worth stating because it is not a claim that the regime is the better one to be in — the same row shows the explicit regime reading at a rate the automatic one never reaches.

Where the split came from#

The self-hosted engines got there first, for a reason that had nothing to do with billing. vLLM, the open-source inference server most teams reach for when they run models on their own hardware, shipped automatic prefix caching to cut prefill work on shared system prompts, hashing fixed-size blocks so that a lookup was a cheap forward walk with no user involvement — an inference server has no idea which parts of a prompt its callers consider stable. Hosted APIs that adopted the same design inherited that property: automatic caching is the right shape when the provider knows nothing about a prompt's structure and has to infer reuse from the traffic itself.

Explicit breakpoints arrived from the opposite direction. If the caller can declare where the stable region ends, the provider can offer a much deeper discount on reads, because it now knows which entries are worth retaining and can charge for the retention decision at write time. The 0.1× read rate and the 1.25× write premium are two sides of that bargain: the caller takes on the risk of a cold write that never gets read, and in exchange the hits are nearly free. The four-breakpoint limit exists for the same reason — each boundary is a candidate entry the provider may have to keep, and an unbounded number of them turns the cache into a storage product.

Reading the two regimes as two bargains over that one risk turns the table above from a list of unrelated facts into a single decision seen twice. Under automatic caching the provider is holding the risk that an entry is never read, so it keeps entries only on terms that suit itself: a granularity it picks, a lifetime it does not publish, eviction whenever the memory is wanted for something else — and it pays for carrying that risk by discounting reads less. Under explicit breakpoints the caller is holding the risk, having asked for the entry and paid a premium for it up front, so the caller gets the terms: a boundary exactly where they put it, a stated lifetime, and a read priced almost as though the work had never been needed. Neither regime is the more generous one. They differ in who is left holding the position when an entry expires unread.

Some providers now run both at once, with implicit caching — the automatic behaviour above, under a different name — applied to every request and an explicit caching API layered on top for cases where you want to pin content for hours and are willing to be billed for storage by the token-hour. That combination is worth knowing about mostly so a hit rate you did not configure does not surprise you, and so the storage line on the invoice has a name when it appears.

What the two look like in a request#

Automatic caching means the request is the request. The only caching-relevant decisions visible in the code are the order of the blocks and, on providers that offer it, an optional routing hint so that requests sharing a prefix land on the same cache shard — one of the many machines the provider spreads entries across, each holding only the entries it wrote.

resp = client.responses.create(
    model="gpt-4.1",
    input=[
        {"role": "system", "content": TOOLS_AND_POLICY},   # ~23,000 tokens, stable
        {"role": "user", "content": question},             # ~200 tokens, per request
    ],
    prompt_cache_key=f"shop-assistant:v7",                 # routing hint, not a key
)
print(resp.usage.input_tokens_details.cached_tokens)       # 22,912 — rounded to 128

The cached_tokens figure comes back rounded down to the provider's increment, which is why 23,000 stable tokens report as 22,912: the last 88 tokens fall inside a partial block and are billed at full rate. 22,912 is the largest multiple of 128 at or below 23,000, and the 88 tokens are the remainder.

Rounding down rather than to the nearest increment is not a pricing choice, and the reason it cannot go the other way is the hashing argument again. The value a block is looked up by covers all 128 of that block's tokens, so a block is either matched in full or not matched at all. A partial block at the end of a prefix never had a value computed over it and has no entry to be found. The loss is capped at one block short of the increment whatever the prompt's length. That is why the rounding is invisible at this scale and matters when the prefix is close to the minimum, which is the next chapter's subject.

Explicit breakpoints mean the boundary is a field you write, and the block it sits on determines the extent of the segment. A request assembled in a well-sorted order but carrying no marker caches nothing at all:

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=2048,
    tools=TOOL_SCHEMAS,                                   # ~3,000 tokens
    system=[
        {"type": "text", "text": CATALOGUE},              # ~20,000 tokens
        {"type": "text", "text": ""},
    ],
    messages=[{"role": "user", "content": [
        {"type": "text", "text": profile_block},          # ~500 tokens, per user
        {"type": "text", "text": question},               # ~200 tokens
    ]}],
)

The repair is one field, placed on the last block that every request shares:

    system=[
        {
            "type": "text",
            "text": CATALOGUE,
            "cache_control": {"type": "ephemeral"},        # segment ends here
        },
    ],

Because tool definitions are serialized ahead of the system blocks, the single marker on the catalogue covers the tool schemas as well — 23,000 tokens in one segment, with the profile and question outside it. The response reports the outcome directly, and the two counters answer different questions: cache_creation_input_tokens at 23,000 with cache_read_input_tokens at 0 on every call means you are writing an entry that nothing reads, while the reverse means the marker landed correctly.

Finding out which regime you are in#

Documentation answers this, and a two-request experiment answers it faster and with less ambiguity about the specific model you are using. Send an identical request twice, thirty seconds apart, with no markers of any kind, and read the usage block on the second response. A non-zero cached count on an unmarked request means automatic caching is running. Zeros on both calls, with a prefix comfortably over a couple of thousand tokens, means the provider is waiting to be told where the boundary is. The experiment costs two calls and removes a class of assumption that survives code review indefinitely, because a missing marker looks exactly like a marker that is not needed.

The experiment is decisive in one direction and conditional in the other, and knowing which is which is what stops it being run badly. A non-zero read on the second call proves automatic caching outright: a read requires an entry, an entry requires a write, and the only request that could have written one carried no marker. There is no other route to that counter being non-zero. Zeros on both calls prove only that nothing was written, which becomes evidence about the regime once three other explanations are ruled out — a prefix under the provider's minimum, a gap longer than an entry survives, and a routing layer that sent the two calls somewhere they do not share entries. Those are where the conditions on the recipe come from. "Comfortably over a couple of thousand tokens" rules out the first, "thirty seconds apart" the second, and sending both calls through one client against one base URL the third.

The same experiment tells you the shape of an entry's lifetime, which also differs between the regimes. Automatic caching is generally described in terms of inactivity — an entry survives some minutes without a matching request and is dropped, with the exact figure unpublished and load-dependent, and no parameter you can set. Explicit breakpoints usually come with a declared duration and sometimes a choice of durations at different write prices, which makes the lifetime something you can reason about and budget for rather than something you infer from a hit-rate graph. That difference is small in code and large in planning, and chapter 6 turns it into arithmetic.

One more thing the experiment surfaces is what the provider reports at all. Some responses partition the prompt into three counters that sum to the total, some report a single cached_tokens field alongside a total that already includes it, and reading the second kind as though it were the first double-counts the prefix and makes a working cache look like a broken one.

Work the direction of that error once and it stops being abstract. Under the partitioning shape the counters are disjoint, the prefix appears in exactly one of them, and adding them up gives the prompt. Under the single-field shape cached_tokens is a slice of a total that already contains it, so adding them up gives the prompt plus the cached span — and a hit rate computed as cached over that inflated total comes out at roughly half of the truth. A dashboard built on it shows a cache that appears to be working some of the time, which is worse than one that appears broken, because nobody investigates it. Confirm the arithmetic once — add the fields up and compare against a token count for the same prompt — and write the assertion down somewhere your dashboards can inherit it.

Strategies that exist in only one regime#

The reason to establish the regime before doing anything else is that entire designs are available in one world and absent in the other. Layering — a boundary after the shared policy, another after the tenant playbook, another after the conversation so far — is a real technique under explicit breakpoints, and chapter 7 works through the placement. Under automatic caching there is nothing to layer; the match runs as far as the bytes agree and stops, and asking for a second boundary is asking for a feature that does not exist. Conversely, "did we remember the markers?" is not a question that can be asked on an automatic provider, and a code review checklist ported between the two will contain items nobody can act on.

What survives the move in both directions is block ordering, prefix length and prefix stability. Those are the three levers in the automatic regime and three of the four in the explicit one, which is why a migration like the one above passes review: the assembler was correct, and correctness in the old regime is a strict subset of correctness in the new one.

Markers do not create namespaces#

A breakpoint is not a cache key. Placing a marker after a tenant-specific block does not scope an entry to that tenant in any security sense; the entry is still matched by content within your organization, and two tenants whose playbooks serialize identically would share it. Isolation between tenants, when you need it, comes from the provider's scoping guarantees and from what you choose to put in the prompt, never from where a boundary sits.

The distinction is easier to hold in the terms chapter 1 sets out. What identifies an entry is the value computed over every byte from the start of the prompt up to the boundary. A marker chooses where that value is taken and has no say in what it is computed from. Two requests arriving at the marker with identical bytes behind them reach the same value whether or not they belong to the same tenant, and two requests with different bytes reach different values whether or not a marker sits between them. Placing a boundary is a statement about extent, never about audience.

A marker is also not an instruction the provider is obliged to obey. One placed on a block that leaves the segment below the minimum length is accepted, returns no error, and produces nothing — a request that looks configured, passes review, and reports zero cached tokens for as long as it runs.

There is a third thing a marker cannot do, and it is the one that composes badly with the two chapters before this. A boundary declares where a segment ends. It says nothing about where matching begins, which is always the first byte of the prompt. So a marker placed after a perfectly stable catalogue earns nothing at all if a per-request field sits in front of that catalogue: the segment is written, it is written under a value no later request reproduces, and it is billed at the premium on every call. Ordering and marking are not alternatives to each other. A prompt needs both, and a prompt that has only the marking is the most expensive of the four combinations.

The regime is therefore the fact to establish first and the one most likely to be assumed rather than checked, since both worlds accept the same well-ordered prompt and only one of them pays for it. An unmarked prefix under explicit breakpoints is ordinary input at full price, however stable and however long, and a marker under automatic caching is a field the provider ignores. Two calls, thirty seconds apart, settle which of those you are looking at.

That leaves the quietest of those failures still to be worked out: a boundary in the right place, on a prefix that is stable and shared, still produces nothing when the segment it closes is too short to qualify.