System prompt layout and cache breakpoints

A prompt is a sequence, and a prefix cache reuses the longest run of leading token IDs that two requests have in common. That makes prompt layout a cost decision rather than a stylistic one: the same content in a different order produces the same output at up to four times the price. This chapter is about the ordering rules that follow, which are short, and about the one genuine argument against them, which is not.

Two terms are needed throughout. The system prompt is the standing instruction block — role, operating rules, escalation policy, tool descriptions, worked examples — that a service sends on every request ahead of whatever the user typed. The cache breakpoint is the explicit marker some hosted APIs require you to place in the request to say "everything up to here is the cacheable prefix"; self-hosted engines with automatic prefix caching infer the boundary instead, but the layout question is identical either way. The failure this chapter is built around needs neither of them to be misconfigured. It happens when the prompt is assembled correctly, the breakpoint is placed correctly, and one varying line sits above both.

The belief that fails here is worth stating in the form somebody would confidently write it: the breakpoint marks the cacheable region, so if the breakpoint is in the right place the region is cached. It is tempting because that is how every other explicit marker in a configuration behaves, and because the breakpoint really is the thing you control and the thing the documentation talks about. What is true instead is that the breakpoint marks where the cacheable region ends and says nothing at all about where it begins. The beginning is always position zero, whether or not anybody wrote a marker there. A region declared correctly at its far end and polluted at its near end caches nothing, and no field in the request records that this has happened.

Why eight tokens in front of nine thousand is the whole loss#

Chapter 4 established that the block hash chain runs from position zero and that block n's identity incorporates every block before it. Put a value that changes at the very front and the consequence is total rather than proportional: the first block's hash differs, so the second block's hash differs even though its content is identical, and so on for all 563 blocks of a 9,008-token prompt. The cache does not fail to reuse eight tokens. It fails to reuse everything, because everything is downstream.

Two numbers in that sentence come from the same division and are worth separating. A 9,008-token prompt is 9,008 divided by 16, which is where the 563 blocks come from; the block size is a property of the engine and the block count is a property of the prompt. What the date changes is one of those blocks, the first. What it costs is all of them, because block two's identity is built from block one's hash, block three's from block two's, and so on to the end. The loss is not proportional to the size of the edit, and there is no partial version of this failure to settle for.

That is the structural claim, and it holds regardless of how fast the leading value changes. What decides whether the loss is total in practice is that rate, measured against how often the prefix is re-served.

The third column is a count of how many different first blocks the template can render in twenty-four hours, and each row is one division. Second-level precision produces a value that never repeats, so every request writes a fresh entry and reads none. Minute precision repeats once a day for each of the 1,440 minutes in it, and whether that is survivable depends entirely on whether more than one request arrives inside the same minute — at low request rates most of those entries are written and never read, which is the worst of both. Hour precision gives 24 values a day, so an entry is rewritten once an hour and read by everything in between. Date precision gives one, which is the only row where the leading value behaves like a constant. The rule underneath the table is that a varying value at the front is harmless when its change rate is slower than your request rate and ruinous when it is faster.

The last two rows are survivable and the first two are not, and the difference between them is a format string. This is why the failure is so common: nobody writes datetime.now() at the top of a system prompt intending to disable caching, and the templates that produce these lines were usually written before caching was a consideration at all. A template that renders {{ now }} and a template that renders {{ today }} look equally reasonable in review, and only one of them costs the whole cache.

The fix is not clever. Move the line to the end, after the last cache breakpoint, where it varies for free alongside the user's message:

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=4096,
    system=[
        {
            "type": "text",
            "text": STABLE_INSTRUCTIONS,      # ~9,000 tokens, unchanged for months
            "cache_control": {"type": "ephemeral"},
        },
    ],
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": f"Current date: {date.today().isoformat()}"},
                {"type": "text", "text": user_question},
            ],
        }
    ],
)

Tools render before system, and system renders before messages, so the breakpoint on the last system block covers the tool schemas as well. Everything volatile now lives after it. The date is still in the context, the model still sees it, and its eight tokens are billed at full rate — eight tokens, once, instead of nine thousand.

The word "before" there is literal rather than conceptual, and the literalness is what makes the rule usable. A request arrives at the API as several named fields, and the provider concatenates them into one flat sequence of token IDs in a fixed order — tool definitions, then the system blocks, then the messages in turn order — before anything is hashed or prefilled. There is no separate channel for any of them. So a breakpoint on the last system block covers everything the concatenation placed ahead of it, including tool schemas nobody mentioned, and a value moved out of system into the first content block of a user message has moved from in front of nine thousand tokens to behind them. Position in that flat sequence is the only property the cache reads.

The same reasoning generalises past dates to everything a prompt assembler is tempted to put up top: a request ID, a session UUID, the user's display name, an experiment bucket, a "you are running in staging" banner, a remaining-quota counter. Each is small, each feels like context that belongs at the beginning, and each converts the entire prefix behind it into a per-request rewrite.

Position bias is a real lever pointed at the wrong end#

Now the harder version of this argument, because it comes from a colleague with evidence rather than from an accident.

Someone reorders the prompt so that retrieved documents sit at the very top, ahead of the system prompt, on the grounds that the model attends better to the beginning of its context. Latency and cost both jump by roughly a factor of four. The reflex is to revert and say position bias is folklore, and that reflex throws away something true.

Position bias is not folklore. Language models retrieve information placed near the beginning or the end of a long context more reliably than information buried in the middle — the "lost in the middle" effect measured by Liu and colleagues across several models and retrieval setups, where accuracy on a fact traces a U-shaped curve against its position. A colleague who moved documents to a privileged position and measured a quality improvement measured something real. The mistake is which privileged position they chose.

What produces that curve is not settled, and saying so is more useful than offering a mechanism the evidence does not support. Candidate explanations include the position encodings themselves, the distribution of where answers happened to sit in instruction-tuning data, and the tendency of attention to concentrate on the opening tokens of a sequence. The effect is measured robustly; its cause is contested. For the decision in this chapter that difference does not matter, because what a team needs is the shape of the curve rather than its origin, and the shape says that both ends are privileged.

Here is where the 4× came from. Take a common shape: 9,000 tokens of stable instructions, 3,000 tokens of retrieved documents that differ per request, and a 200-token question.

Prefill is roughly linear in token count for these lengths, so 12,200 tokens against 3,200 is a 3.8× increase in prefill work, which is a 3.8× increase in time to first token once fixed overhead is netted out. On the cost side it is 12,200 against 4,100, or 3.0×. Round both and you have the "about 4x" the colleague observed, and neither number has anything to do with the model attending differently — the model is doing the same arithmetic in both layouts. One layout lets it skip 9,000 tokens of that arithmetic and the other does not.

The two ratios differ because they divide different things, and knowing which is which prevents a later argument. Time is set by work done, and a cached block is work not done at all, so the latency comparison is 12,200 tokens prefilled against 3,200 prefilled, with the cached 9,000 contributing nothing to either side. Money is set by what is billed, and a cached token is still billed, at a tenth of the rate — which is where the 900 in the second row's total of 4,100 comes from, 9,000 tokens at 0.1 each. A hit is free in time and merely cheap in money. That is why one layout change produces two different multipliers, and why quoting either of them for the other is how a rollout note ends up wrong.

The resolution is that the end of the context is privileged too, and the end is the position the cache does not charge you for. Both goals are satisfiable at once:

[tools]                      stable   ─┐
[system instructions]        stable    ├─ cached prefix, breakpoint here
[few-shot examples]          stable   ─┘
[retrieved documents]        varies    ─┐
[current date]               varies     ├─ suffix, full price, small
[user question]              varies    ─┘

Documents go after the cached prefix and immediately before the question, which puts them in the recency-privileged region rather than the primacy-privileged one. If a specific evaluation shows that particular set of documents still needs more prominence, the cheap intervention is to restate the question after the documents so the model reads task, evidence, task — a hundred tokens of suffix, not nine thousand tokens of invalidated prefix. What you do not do is reorder the prefix, because the prefix is the part whose order has a price.

It is worth saying plainly to the colleague that their measurement stands and only the placement changes, because the alternative framing — "revert it, that effect is not real" — loses a quality lever the team may need later and teaches everyone that caching is a constraint on doing good work rather than an ordering discipline.

Where the boundary goes when the prompt grows#

A single-turn prompt has one obvious place for the boundary. A conversation has a moving one, and getting that wrong is the second most common way to end up paying write premiums for entries nothing reads.

In a multi-turn exchange the reusable region is not just the system block; it is the system block plus every completed turn, because the history up to turn n is byte-identical on the request that produces turn n+1. Placing the breakpoint at the end of the system block only, and leaving the accumulated history behind it, means a conversation that has grown to 30,000 tokens reuses the original 9,000 and prefills 21,000 from scratch on every turn. Placing it on the last content block of the most recently appended turn instead lets the reusable region grow with the conversation, so turn twelve reads everything through turn eleven.

That works because earlier breakpoints stay valid as read points even after a later one is added — the entries accrue rather than replacing each other. What it needs is that the history behind the moving breakpoint is genuinely append-only. A harness that re-renders the whole transcript each turn from a data structure, rather than appending to a list, is free to change the formatting of turn three while composing turn twelve, and any such change invalidates from turn three onward. Trimming or summarising old turns has the same effect, which is worth knowing before deciding where in a long session to compact: compaction rewrites the prefix and costs the entire cached history, so it is worth doing once at a threshold rather than continuously.

The accrual claim rests on the same chaining that causes all the trouble elsewhere, and this is the one place it works in your favour. A breakpoint at the end of turn eleven names a prefix, and the blocks holding that prefix are physically the same blocks that hold the prefix ending at turn ten, plus a few more on the end. A later, longer entry therefore does not replace the shorter one; it extends the run of resident blocks. A request whose prompt stops earlier walks the same chain from position zero and stops earlier too, finding everything it needs already there. Because identity accumulates forward, every prefix of a cached prefix is also cached, at no extra cost and with nothing to configure.

There is also a hard ceiling on how many breakpoints a request may carry — four, on the hosted APIs that expose them explicitly — which is a budget rather than a limitation once you see what it is for. The idiomatic allocation is one at the end of the tools-plus-system region, one at the end of any large stable context such as few-shot examples — the worked input-and-output pairs a prompt carries to demonstrate the task — or a fixed document set, and one that moves with the conversation. The fourth is spare, and spending it on something that varies per request wastes it, since a breakpoint on volatile content creates an entry that will never be read.

Reproducing the failure so you believe the numbers#

None of this needs a production incident to observe. Take one prompt, render it four ways, and read the usage fields back on the second call of each pair.

variants = {
    "date_first":   f"Current date: {now}\n\n{STABLE}",
    "date_last":    f"{STABLE}\n\nCurrent date: {now}",
    "docs_first":   f"{docs}\n\n{STABLE}",
    "docs_last":    f"{STABLE}\n\n{docs}",
}

for name, system_text in variants.items():
    for call in (1, 2):                       # call 2 is the one that matters
        u = send(system_text, question).usage
    print(f"{name:12} read={u.cache_read_input_tokens:>6} "
          f"write={u.cache_creation_input_tokens:>6} fresh={u.input_tokens:>5}")

date_first   read=     0  write=  9008  fresh=  200
date_last    read=  9000  write=     0  fresh=  208
docs_first   read=     0  write= 12008  fresh=  200
docs_last    read=  9000  write=     0  fresh= 3200

The write column on the two broken variants is the part to sit with. Those are not neutral misses; each one is 9,000 or 12,000 tokens billed at 1.25× to create an entry that the next request will not match either.

Deploy-cadence changes, suffix cost and the latency claim#

None of this means the prefix must never change. Prefixes change constantly — instructions get edited, tools get added, examples get replaced — and that is fine, because those changes land at deploy time rather than per request. A prompt edit costs you one cold fill per node and then the hit rate recovers. The distinction that matters is not stable versus changing; it is changing on a deploy cadence versus changing on a request cadence.

It also does not mean the suffix is unimportant. Everything after the breakpoint is billed at full input rate and prefilled on every request, so a 3,000-token document set is genuinely expensive whether or not it is placed correctly; putting it in the right position saves you the 9,000 tokens in front of it and not the 3,000 tokens of itself. Retrieval budget is still a real budget.

And the layout above is not the last word on latency. Getting the ordering right moves prefill work off the critical path, which is worth a great deal on a long prompt — but only for the part of the response the reader is waiting for first.

Prompt layout, then, is one rule with one exception worth arguing about. The rule: everything that changes on a deploy cadence goes first, the breakpoint goes after the last of it, and everything that changes per request goes behind the breakpoint, in an order chosen for quality rather than for cost because it is being prefilled either way. The exception is position bias, which is real, and which is satisfied by the end of the context as well as the beginning — so a document set that needs prominence gets it from recency, at a hundred tokens of restated question, instead of from primacy at nine thousand tokens of destroyed prefix.

The invoice is the honest way to check that the rule is being followed, and the four-variant experiment above will produce the numbers in ten minutes. What it will not tell you is how much faster the endpoint got, because a single end-to-end latency figure averages a phase that improved enormously with a phase that did not move at all, and the weights are set by how long the answers are.