The needed context fraction

"What fraction of the tokens we send does the model actually need" is a question with a defensible answer, an obvious wrong answer, and an afternoon of compute between them. Take a prompt of 900 tokens of system instructions, 2,400 tokens of few-shot examples, 9,800 tokens of retrieved passages across five chunks, and about 900 tokens of conversation tail: 14,000 tokens on every request, and no way to read the answer off the assembly.

Necessity is a property of the task, not of the text#

The tempting shortcuts all try to do exactly that. Token counts rank blocks by size, which says nothing about whether the large one is load-bearing. Embedding similarity between each block and the question ranks blocks by topical relatedness, which promotes passages that sound like the question and demotes a system instruction that governs the output format on every request. Attention weights look like the most principled option and are the most misleading: a block the model attends to weakly can still be the thing that keeps it from answering in the wrong register, and attention is a mechanism inside one forward pass rather than a measure of what the answer would have been without the block.

The definition that survives a meeting is behavioural. A block is needed if removing it and re-running the frozen eval drops the score below the threshold agreed in advance. That definition is reproducible by anyone with the arm files, it produces a number rather than a ranking, and it forces the two things people argue about — which set, and how much of a drop counts — into the open before the measurement rather than after it. Needed, in that sense, is a verdict returned by a measurement rather than a property anybody can read off the text.

One block out at a time#

The sweep is the ablation from the first chapter of this topic run in a loop — a leave-one-out design, meaning every arm drops exactly one block and keeps all the others — with one arm per removable unit and everything else pinned.

BASE_BLOCKS = ["system", "few_shot", "passages", "conversation"]
THRESHOLD = 0.005          # agreed before the run: a drop this size or larger means "needed"
BASELINE = 0.879           # support-eval-v3, 3 repeats, judge at 3 votes

def leave_one_out(base_arm: dict) -> list[dict]:
    """One arm per block, each identical to base except for the removal."""
    arms = []
    for block in BASE_BLOCKS:
        arm = dict(base_arm)
        arm["label"] = f"without-{block}"
        arm["blocks"] = [b for b in BASE_BLOCKS if b != block]
        arms.append(arm)
    return arms

def verdict(score_without: float) -> str:
    drop = BASELINE - score_without
    if drop >= THRESHOLD:
        return "needed"
    return "removable-on-this-set"

Four arms, three repeats each, about $320 and ninety minutes of wall clock at twenty concurrent requests. The results are unsurprising in three rows and interesting in the fourth.

Two of those columns are computed rather than measured, and each is one operation. The drop is the baseline minus the score the arm without that block achieved, so a larger drop means the arm missing the block did worse, which is what "the block was doing something" means once it is stated operationally rather than intuitively. The verdict is then a comparison of that drop against the threshold fixed before anything ran. What is worth noticing is the column the verdict does not consult: the token count. A nine-hundred-token block and a nine-thousand-eight-hundred-token block are judged by the same rule, and that is the whole content of defining necessity behaviourally — size buys no presumption in either direction.

At block granularity the answer is that 13,100 of 14,000 tokens are needed, which is 94% and is useless — it tells you the prompt is almost entirely load-bearing, and it is an artefact of the granularity rather than a fact about the system. Nobody proposed deleting all nine thousand tokens of retrieved passages. The question is whether all five of them are earning their place on each request, and that needs the sweep run one level down.

Both figures in that sentence come off the table's first two columns. Three blocks were marked needed, and their token counts — nine hundred, two thousand four hundred and nine thousand eight hundred — add to the thirteen thousand one hundred quoted; the same number arrives from the other direction as the fourteen thousand total less the nine hundred tokens of the one block that fell below the threshold. Dividing the first by the second gives the percentage. The reason it is useless is structural rather than arithmetical. A leave-one-out sweep can only return verdicts at the granularity of its arms, and with four arms and three of them saying yes, the largest saving this sweep is capable of reporting is the smallest block in the prompt.

Decomposing the blocks that matter#

The few-shot block is twelve worked examples of roughly 200 tokens each, and each can be removed on its own. Eight of the twelve come out with a combined drop of 0.002; the other four account for the whole 0.018. Retrieved passages decompose by rank.

Per case, the sharper measurement uses the gold_rank field from the chapter on instrumenting a run: the passages a case needs are the ones down to and including its gold chunk. That distribution is 78% of cases with gold at rank 1, 14% at rank 2, 5% at rank 3 and 3% deeper, which averages to 1.35 passages and 2,640 tokens. The conversation tail is needed on the 6% of cases that are genuine follow-ups, contributing about 54 tokens of average need. Adding up per-case needs and averaging over the set gives 900 for system, 800 for the four load-bearing examples, 2,640 for passages and 54 for conversation: 4,394 tokens out of 14,000, or roughly 30%.

Every figure in that paragraph is an average of a per-case quantity, and each is worth reconstructing once, because the reconstruction is the only thing that distinguishes the number from an assertion. The passage count is a weighted mean: multiply each rank by the share of cases whose gold chunk sits at that rank, and add the products. Nine thousand eight hundred tokens divided across five passages is a little under two thousand tokens apiece, so the mean passage count multiplied by the mean passage size gives the passage figure. The conversation figure is a share multiplied by a size: the block is nine hundred tokens and it is needed on six cases in a hundred, so six hundredths of nine hundred is what it contributes to the average request's need. The few-shot figure is the four load-bearing examples at roughly two hundred tokens each. The system block is unconditional, so it enters at full size. Add those four and you have the total; divide the total by the fourteen thousand actually sent and you have the fraction.

That is the number, and by itself it is dangerous, for two reasons that are measurable rather than philosophical.

Marginal contributions do not add#

Leave-one-out measures each block's effect on the score while every other block is still present, and those effects do not sum. Summing the deltas from the table predicts that removing the eight few-shot examples, the conversation tail and passage ranks 3 through 5 together should land at 0.869, comfortably inside the threshold. The combined arm actually scores 0.841.

The prediction in that sentence is a sum, and the sum is the thing that fails. Take the baseline and subtract the conversation tail's drop, then the eight examples' drop, then the drop measured for ranks three through five: three thousandths, two thousandths and five thousandths, ten thousandths in total, which lands exactly on the predicted figure in the table below. The arithmetic is not where the error is. The error is in the assumption the arithmetic encodes, which is that each drop measures a fixed property of its block. It does not. Each drop was measured with the other two blocks still in the prompt, so each one records what its block adds given everything else that is present — and there is no rule saying such quantities add. Whether they add depends on whether the blocks are independent of one another, and blocks in a prompt very often are not.

The per-slice read locates the interaction immediately: 22 of the 26 lost cases are in refunds/out-of-window, where two of the "removable" few-shot examples and the rank-3 passage were encoding the same policy window redundantly. Either one alone was free to remove because the other still carried the fact. Removing both left the model with no statement of the window at all. Redundancy is the normal reason marginal contributions understate joint contributions, and it means every composed cut has to be measured as its own arm rather than derived from the single-block table. The single-block sweep is a screening step that tells you where to look, not a plan you can execute by addition.

The second obstacle is subtler and larger. The per-case passage measurement used gold_rank, which is a label attached to the eval set by whoever built it. A production request does not come with a label saying which of its five retrieved passages is the relevant one — if it did, retrieval would be a solved problem. The 30% figure is therefore an oracle measurement: it is what a system with perfect foresight about relevance would have needed, which makes it an upper bound on what any real pruner can save rather than a description of a prompt you could ship tomorrow.

That it is an upper bound rather than merely an optimistic estimate follows in one step. The oracle measurement asks how few passages would have sufficed given a label saying which one mattered. Anything that could run in production has to guess instead of reading that label. Where it guesses right it keeps what the oracle kept and saves what the oracle saved. Where it guesses wrong it either keeps a passage the oracle would have dropped, which saves less, or drops the passage the case needed, which costs score. Neither kind of mistake can save more than the oracle did, and there is no third kind. So the oracle figure is a ceiling over every selector anybody could ever write — which makes it the right number to give a director who asked how much is theoretically there, and the wrong number to put on a roadmap.

A deployable version needs a selector that guesses which passages to drop, that selector will be wrong on some fraction of requests, and the cost of its mistakes has to be measured with the same machinery. A reranker score threshold that keeps passages above 0.42 recovers about half the theoretical saving at a cost of 0.004 on the frozen set, which is a real proposal with a real number; "cut the prompt to 4,000 tokens" is not.

The number carries its set with it#

The conversation block is the row worth staring at. It scored a drop of 0.003 and was classified removable, and it is the block most likely to cause an incident if you act on that. The frozen set is 6% multi-turn because it was assembled from a sample of tickets that mostly resolve in one exchange. Production traffic over the last quarter was 23% multi-turn, because customers who are unhappy ask follow-up questions and customers who are satisfied do not. Re-weighting the eval to match production's multi-turn share turns the 0.003 drop into 0.014, which crosses the threshold three times over.

Re-weighting is a change of arithmetic rather than a change of data, and the step it makes visible is the one nobody writes down. An eval score gives every case the same weight, which means the set's own composition — how many multi-turn cases somebody happened to collect — is silently functioning as a claim about how often multi-turn traffic occurs. Re-weighting replaces that claim with the measured one: score the multi-turn cases and the single-turn cases separately, then combine those two averages using production's shares as the weights instead of the set's. Because the conversation block only matters on the multi-turn cases, and production carries a far larger share of them than the set does, the same per-case effect arrives multiplied. Nothing about the block changed between the two numbers. The set had been answering a question about itself.

This is the general shape of the trap, and it is not specific to conversation tails. A block looks inert when the inputs that need it are rare in your set, and inputs are rare in your set for reasons — they were hard to collect, they came from an escalation queue nobody exported, they only occur in a locale that was added after the set was frozen. The blocks carrying those inputs are exactly the ones that will read as free to delete, and deleting them concentrates the damage on the traffic you already had the least visibility into. Coverage is the failure the churn floor cannot see, and it is the one that costs the most.

Three practices keep the number honest. Build a stress slice before you cut: 50 to 100 cases deliberately sampled from the input shapes the main set under-samples, scored separately and reported alongside the aggregate, so that a proposal to remove a block has to clear both. Stage the cut behind a shadow comparison, running the reduced prompt on a copy of live traffic and diffing its answers against production's for a week — which will not tell you which answer is correct, but will tell you how many requests changed behaviour and on what. And report the fraction with everything that produced it attached.

{
  "claim": "needed_context_fraction",
  "value": 0.31,
  "needed_tokens_mean": 4394,
  "total_tokens_mean": 14000,
  "set": "support-eval-v3",
  "set_sha256": "9f3c1ab7d2e04c85",
  "cases": 500,
  "threshold_drop": 0.005,
  "baseline": 0.879,
  "repeats": 3,
  "churn_baseline": 4,
  "model": "claude-sonnet-4-5-20250929",
  "measured_at": "2025-08-26",
  "oracle": true,
  "oracle_note": "passage need computed from labelled gold_rank; not available at serve time",
  "composition_checked": true,
  "known_coverage_gap": "6% multi-turn in set vs 23% in Q2 production traffic"
}

The oracle and known_coverage_gap fields are the two that stop the number from being repeated as something it is not. A record without them travels perfectly well into a slide deck and arrives as "30% needed, 70% waste".

Necessity, then, is measured by removal and reported with its conditions, which is what lets the fraction be quoted defensibly instead of merely confidently. The sweep is a screening step that says where to look; the composed arm says what a cut actually costs; the oracle qualifier says the fraction is an upper bound rather than a plan; and the coverage gap says which part of the removable remainder is protecting traffic your set barely contains. What the machinery in this topic buys is not certainty about context. It is the ability to say what you measured, on what, against which threshold, and what would change the answer.

Running it is a cadence rather than a project. The leave-one-out sweep is cheap enough to schedule quarterly and belongs in the same job that re-measures the churn floor, because both move when the model pointer moves. Every ablation, every decision record and every reported fraction carries the model snapshot it was measured against, so a model release is a signal to re-run rather than a reason to distrust the archive. The eval set grows by whatever the last incident revealed the set could not see, and the stress slice grows faster than the main set. Kept that way, the numbers in the records stay close enough to true that the next argument about whether a block is earning its tokens starts from a file, and the argument takes ten minutes instead of a quarter.