Long-context vs RAG

Read these first

• Context degradation
• Prompt caching mechanics

What you will know

• Compare stuffing the window against retrieving, with measured cost and quality

  Use the same documents, questions, model, and evaluation set while accounting for input tokens, cache hits, latency, and answer correctness.

• Measure how answer quality and latency change as irrelevant context grows

  Long prompts can dilute relevant evidence, trigger lost-in-the-middle behavior, increase processing time, and leave less room for the answer.

• Use prompt caching to make a large-document approach affordable

  Savings depend on stable prefixes, provider-specific cache semantics, and disciplined invalidation when documents or instructions change.

• Choose per feature and defend it with numbers

  Set explicit break-even thresholds for p95 latency, token spend, freshness, and task success because the cheapest path varies by workload.

• Break the design with corpus growth, document churn, and adversarial queries, then patch the fallback policy

  A strategy that works on a stable small corpus can fail through stale cached context, oversized inputs, empty retrieval, or injected instructions.

Chapters

1. 1

   Input tokens per session, not per request, decide the cost

   Work the per-session arithmetic for a 250k-token document asked twelve questions: twelve full reads against twelve slim retrieved prompts. That fifty-fold gap is real for uncached reads, and it is the number most teams stop at. This chapter sets the comparison up honestly — assumptions stated out loud, output tokens separated from input, retrieval's own indexing and embedding cost included — so later chapters can show what moves it.

   Read chapter
2. 2

   A cached prefix only pays off if nothing mutable sits in front of it

   Prompt caching collapses that raw token gap, but only under one condition: the document must sit ahead of everything that varies. You will see why a 200k-token prefix can measure a 4% hit rate when a timestamp or personalisation block precedes it, how cache-read pricing turns eleven of twelve reads into a fraction of full price, and why prefix order is a design decision rather than an implementation detail.

   Read chapter
3. 3

   A cache entry ages: what invalidation and TTL do to the cost model

   A cached prefix is not free forever. Here you build the cost model for a 400k-token corpus serving 2,000 queries a day and edited twice a week, including what a write does to the entry and what uneven traffic does at a TTL measured in minutes. Two traps follow: appending edits to preserve the cache, which quietly makes answers reflect the corpus as of the last write, and what changes at 1.2M tokens.

   Read chapter
4. 4

   More context can lower accuracy, not just raise the bill

   Filling the window is not a neutral act. Accuracy can fall six points when the same corpus moves from eight ranked chunks to 300k stuffed tokens, with the losses clustering on evidence positioned mid-context. Dilution, position effects and prefill latency are covered here — plus why a near-perfect needle-in-a-haystack score tells you a verbatim string is findable and almost nothing about reasoning over several dispersed, similar passages.

   Read chapter
5. 5

   Retrieve into the window, then cache what you retrieved

   Between eight ranked chunks and 300k stuffed tokens sits the design the binary framing hides: select a subset of a corpus too large to fit, put that subset in a large window, and cache it. The catch is prefix stability, because a subset that varies per query cannot belong to a shared cacheable prefix — it either sits after the cacheable part, or the cache is keyed per document set, which only pays when sessions revisit the same set. What it buys is prefill time, which scales with tokens in the window, so this is the one lever that moves a p95 regression when prefix layout and cache warmth cannot. It is also the third branch any routing rule needs before an engineer can apply it without asking you.

   Read chapter
6. 6

   Proving a long-context design is at least as good, before you switch

   Approval for a switch needs a comparison that means something. Hold model, question set, documents and grader constant; measure input tokens, the measured cache hit rate, p95 latency and answer correctness together. When the first result is better accuracy and 4x worse p95, you will know which knobs — prefix layout, cache warmth, hybrid routing — to try before conceding, and which result would genuinely settle it.

   Read chapter
7. 7

   Truncating to fit answers from evidence nobody can name

   What should happen when a user attaches 1.4M tokens to a feature with a 900k-token ceiling? Silent truncation produces a confident answer derived from an unknown subset of the evidence, which is worse than an error nobody can miss. You will work through corpus growth, document churn and empty retrieval, then write the fallback policy that degrades visibly instead. Injected instructions are the case that does not fit the ceiling frame at all: a retriever admits ranked chunks while a stuffed window admits everything the user attached, so document content has to be marked and handled as untrusted data rather than as prompt.

   Read chapter
8. 8

   Route per feature, and re-run the numbers when the model changes

   A blanket 'always fill the window' rule and a blanket 'retrieval only' rule fail in the same way. Here you write the per-feature rule instead: a 12M-token knowledge base asked one question per session routes differently from a 90k-token diff reviewed across thirty turns. Set explicit break-even thresholds on spend, p95 and freshness, and attach a re-measurement trigger so the next model release does not reopen the argument.

   Read chapter