Senior · 7 chapters
Context engineering — ordering
Position bias (1.12) wants important content late; prompt caching (2.15) wants stable content early. Resolving that conflict deliberately
Ignoring this tradeoff quietly turns either cache spend or model reliability into a production tax.
Read these first
What you will know
Explain why position bias and cache-friendliness pull in opposite directions
A fixed prefix can maximize reuse while burying instructions in the region models attend to less reliably.
Choose an ordering and say what you gave up
The decision should name the expected traffic mix and which degradation—extra tokens, cache misses, or accuracy—you accept.
Measure the cost of the choice in both cache hit rate and quality
A hit-rate gain can be a loss if the resulting context pushes relevant evidence past the model’s effective attention window.
Identify a prompt where the conflict is not worth resolving
For small inputs, cache setup and prompt complexity can cost more than either optimization saves.
Restructure a prompt to satisfy both where it's possible
Put immutable policy and schemas in a reusable prefix, then inject retrieved or task-specific material in a bounded tail while verifying that instructions remain effective.
Chapters
- 1
Prefix matching in prompt caching
A prompt cache matches from position zero forward and halts at the first token that differs, so cacheability belongs to a whole layout rather than to any block inside it. Moving a user's question and three retrieved chunks to the top of a support agent's prompt improved compliance from 4.1% violations to 0.6% and took the cache hit rate from 78% to 4%, tripling input spend to around $4,400 a day. Treating cacheability as a per-block property is the misunderstanding that makes that trade look free.
- 2
The stable-first layout
Putting the invariant material at the front — a 9k policy and tool block ahead of retrieval — preserves the cacheable prefix and places every rule you care about far from the end, in the region models follow least reliably. Rules that scored 99.4% after the retrieved documents score 95.9% buried in the policy's interior, which on 120,000 requests a day is about 1,500 answers quoting a price the policy forbids. The trade pays in one of three currencies, and a decision that does not name its currency is not a decision.
- 3
Timestamp precision and placement
\"Current date and time: 2026-08-07T14:03:11Z\" ahead of 8k of stable policy is why a scheduling assistant's hit rate is 0.0%: the match fails at the first eleven tokens and nothing behind them can be reused, which puts the input bill 25% above no caching at all. Bucketing the timestamp per minute saves 6% and costs a rotating cache key. The answers are the precision each rule actually consumes and the position each value belongs in.
- 4
Evaluating a layout change
Hit rate 12% to 89%, input spend down 40%, ticket closed — and three weeks later a research assistant is omitting source citations in about one answer in twelve. A permutation of a sequence is a different sequence, so a reordering is a prompt change and needs the same evidence as a rewritten sentence: both layouts scored on one case list at production temperature, compliance predicates against written floors, and a production canary sized to fire on the first day.
- 5
The tail restatement
With a 12k immutable block of policy and schemas first and a bounded tail of documents and the user turn behind it, restating the three rules that carry consequences — cite sources, never quote prices, ask before destructive actions — in 58 tokens at the very end takes citation compliance from 92.0% to 98.9% for about $21 a day. The prefix is untouched, the rules land where they are read, and repeating the whole 12,000-token policy instead scores worse on every predicate and costs $4,062 a day.
- 6
Giving up the prefix
Sometimes the tail restatement does not recover compliance and the remaining option is to move the policy behind the documents, keep the tool schemas in front as a smaller prefix, and pay the input bill. Here you work the decision explicitly — the four things that have to be true first, how routing bought a 0.999 floor for $337 a day instead of $2,409, why a layout with no prefix and its cache_control marker still attached costs 20% more than caching switched off, and how to record which degradation was chosen.
- 7
The minimum cacheable prefix
Providers will not cache a prefix below a floor of about 1,024 tokens, and they do not warn you: a 300-token classifier prompt accepts a cache_control marker, ignores it, and reports both usage counters at zero. Two identical calls in a test settle whether the mechanism engages, and pairing that with a reuse count from production logs settles whether it is worth anything — which for a 220-token prefix at forty requests an hour is about $2.80 a month.
External resources
- Lost in the Middle: How Language Models Use Long Contextscs.stanford.edu
The position-bias evidence that pushes important content to the edges.
- Effective context engineering for AI agentsanthropic.com
This is a substantive first-party Anthropic engineering explainer on context engineering that provides authoritative mechanics and guidance relevant to context organization, attention limits, and agent context management.
- Attention Sorting Combats Recency Bias in Long Context Language Modelsarxiv.org
Shows reordering retrieved documents alone changes answer quality.
- LLM Position Bias: Primacy and Recency Effects in Promptsintuitionlabs.ai
Practical resolution: duplicate critical rules at head and tail.