Position bias in long prompts

Context degradation is the fall in answer quality that arrives as a prompt grows longer and as the material that matters moves around inside it, and it is what this topic measures. The word degradation is doing honest work: nothing breaks when it happens. The request succeeds, the response comes back fluent and confident, and the only symptom is that it is wrong more often than it was last week. It turns up wherever a prompt grows without anyone deciding that it should — a support assistant that pastes twenty retrieved passages in front of every question, a standing instruction block that four teams have appended to over a year, an agent whose context carries every tool result it has collected since a session opened, a summariser handed a sixty-page contract, a chat on its fortieth turn.

This chapter is about the most reproducible form of that fall. Take one passage that answers a question, put it at the front of an otherwise identical long prompt, and score whether the answer is right; then move that passage to the middle, and to the end, and score again. Accuracy changes, by high single digits on a realistic prompt, with the same bytes of evidence present in every condition. The chapter builds the probe that produces those three numbers, reads the curve they lie on, and shows why the curve is a property of the configuration it was measured on rather than a fact about language models that somebody else can establish for you.

Some vocabulary is worth fixing first, because the rest of the topic runs on it. A retrieval-augmented service is one that, before it calls a model, runs a search over a store of documents, gets back a dozen passages that look relevant to the user's question — chunks, in the vocabulary of the search step — and pastes them into the request as a block of text, with the conversation so far and a fixed set of instructions arranged around them. That assembled string, its sections in a settled order with slots for the parts that vary, is a prompt template. A regression eval is a fixed list of questions whose correct answers are known, replayed after every deployment so that a drop in quality arrives as a number rather than as a complaint. The next chapter gives the search half its proper names and its own metrics; here it is enough that a long prompt is a document with sections, and that somebody decided their order.

One more distinction keeps this from being confused with a limit most teams already track. The context window — the hard ceiling on how many tokens a single request may contain, covered in topic 1.4 — decides whether a request is accepted at all. Quality begins falling well below that ceiling, sometimes tens of thousands of tokens below it, and no part of an API reports the crossing. A team that reads a large window as permission to send more will keep adding retrieved passages, tool schemas and conversation history long after the additions have started costing accuracy, and will look for the cause everywhere except the length and the arrangement of the prompt they assembled themselves.

A probe that varies one thing#

The design is a controlled substitution. You need a question with a known, checkable answer; one passage that contains that answer; and a pool of filler passages that are on the same general subject but do not contain it. The prompt is assembled from the fillers with the answer-bearing passage spliced in at a chosen index. Then you rebuild the identical prompt with the answer passage at a different index and score again.

import json, random
from anthropic import Anthropic

client = Anthropic()

def build_prompt(question, gold, fillers, position):
    """position: 0 .. len(fillers). Total token count is constant."""
    docs = list(fillers)
    docs.insert(position, gold)
    body = "\n\n".join(
        f"[{i}] {d['title']}\n{d['text']}" for i, d in enumerate(docs)
    )
    return (
        "Answer the question using only the documents below.\n"
        "Reply with the answer only, no explanation.\n\n"
        f"{body}\n\n"
        f"Question: {question}"
    )

def score(question, gold, fillers, answer, position):
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=64,
        temperature=0,
        messages=[{"role": "user",
                   "content": build_prompt(question, gold, fillers, position)}],
    )
    got = msg.content[0].text.strip().lower()
    return answer.lower() in got

with open("probe_set.jsonl") as f:
    items = [json.loads(line) for line in f]

for position in (0, 10, 19):           # first, middle, last of 20 slots
    hits = sum(
        score(it["question"], it["gold"], it["fillers"], it["answer"], position)
        for it in items
    )
    print(position, hits / len(items))

Three details in that harness are doing real work, and a probe missing any of them produces a curve you cannot trust. The total token count must be identical at every position: inserting the gold passage rather than appending it, and drawing the fillers from a fixed shuffled pool per item, keeps the prompt the same length in every condition. If the middle condition happens to be 400 tokens longer than the first condition, you have confounded position with length, and length has its own effect that a later chapter of this topic isolates by padding a prompt with material nobody could mistake for evidence. The second is that the filler order is fixed per item across conditions, so the only thing that moves between runs is the index of one passage. The third is temperature=0, the sampling setting that turns randomness off so that the same prompt returns the same answer, plus a strict scorer. Position effects are worth a few points, and the sampling noise you get at temperature 1 across 1,200 items is comfortably larger than that, so a probe run at default temperature mostly measures its own variance.

Run it on a few hundred questions, three positions, at 20 documents per prompt, and the whole sweep costs somewhere around 1,200 calls at roughly 8,000 input tokens each — under ten million input tokens, which is a small four-figure number of cents on a mid-tier model and finishes in a few minutes with modest concurrency.

The probe set itself does not have to be exotic. What it needs is questions whose answers are short and unambiguous enough for a string match to be a fair judge, and gold passages whose answer does not also appear in the fillers. Entity-and-attribute questions over your own documents work well — a policy number, an effective date, a threshold, the name of the team that owns a service — because they fail loudly and score cheaply. Two hundred items is enough to see a spread of ten points; distinguishing a three-point difference between two templates needs closer to a thousand, since the standard error on a proportion near 0.7 with 200 samples — roughly how far the measured score will wander from the true one by chance alone — is about 3.2 points, and you are comparing two of them.

Both halves of that sentence are worth unpacking, because sample size is where a position probe most often produces a result its author should not have believed. A score is a proportion measured on a sample, and the spread of a proportion falls with the square root of the sample size rather than with the sample size itself. Quadrupling the item count therefore halves the wandering and no more, which is why going from two hundred items to a thousand buys roughly a factor of two and why going to two thousand would buy so little more. That is the first half. The second half is that a comparison between two templates is not one measurement, it is the difference of two, and each of them is wandering independently. A difference of two independent measurements wanders further than either one alone, by a factor of the square root of two. So a three-point gap measured on two hundred items per condition is comfortably inside the range that identical templates would produce by luck, and reporting it as a finding is the commonest way to ship a placement rule that does nothing.

The U-shaped curve and its depth#

The characteristic result is that accuracy is highest when the answer-bearing passage is first, drops to its minimum somewhere in the middle of the list, and recovers — usually not all the way — when the passage is last. Plotted against position, that is a U with an asymmetric right arm. The original systematic demonstration is Liu et al.'s Lost in the Middle: How Language Models Use Long Contexts, which ran multi-document question answering with the gold document walked across every slot and reported exactly this shape for every model tested, including models whose closed-book accuracy on the same questions was well below their worst in-context position.

Here is one run of the harness above over 900 questions, at two list lengths, on a single model:

Two things are readable from that table that a single-length experiment cannot show. The gap between the best and worst position is 5.5 points at 10 documents and 14.9 points at 30, so the depth of the U is a function of how much context is in the prompt rather than a constant property of the model. And the recovery at the end is real but partial: last place is 1.3 points below first at 10 documents and 5.4 points below at 30. Ordering a list so that the most important item is last is better than burying it, and still worse than putting it first.

It is worth being explicit about what the numbers in that table are and are not. They come from one harness run on one model with one document pool, and they are reported to a tenth of a point because that is what the arithmetic produced, not because a tenth of a point is meaningful. The quantity to carry away is the spread — the distance between the best and the worst position at the length you actually ship — because that is the size of the prize for getting placement right, and it is the number that tells you whether ordering is worth engineering effort on your system or is noise you can ignore.

Before reading anything into the shape, it is worth asking why position should matter at all. Every token in these prompts is inside the model's limit, none of them was truncated, and the model can quote any of the twenty passages on request. The step that makes the curve intelligible is not what the prompt contains but what the model does with it while producing an answer. Attention is that operation: at each step, the model computes a weighting over every position already in the prompt and uses it to decide how much each of those positions contributes to what it writes next. Two properties of that weighting carry the whole argument below. The weights are never negative, and they sum to one across the prompt.

A quantity that cannot go negative and must sum to one is a budget, and a budget is rivalrous: a share given to one position is a share not given to another. That is the first causal step, and it predicts something before position enters the picture at all. Hold a passage's content fixed and add a thousand tokens of anything to the prompt, and that passage is now competing with a thousand more claimants for a total that did not grow, so its expected share falls. A later chapter of this topic pads a prompt with paragraphs about sedimentary geology, changes nothing else, and watches quality fall — which is that prediction, run.

Rivalry alone predicts a slope, not a U. For the two ends to beat the middle, positions have to differ from each other in some way that is not about their content, and they do, for two reasons that point in opposite directions. The prompt is read in order, so material near the front is upstream of everything computed after it: every later position is represented in a form that was derived with the front of the prompt already in view. And the question sits at the end, so material near the end is separated from the question by almost nothing, while material near the front is separated from it by the entire document block.

The middle is the one region with neither advantage, and its position makes a third problem worse. It is far from the question and it is upstream of little. It is also surrounded by its own closest competitors, because twenty chunks retrieved for one query are twenty passages that a search engine scored highly against the same words. A passage sitting halfway down that list is neighboured by near-copies of itself, and the only thing separating it from them is the content the model is being asked to weigh in the first place.

That chain is worth having because it predicts the rest of this chapter rather than merely restating it. If depth is competition, adding competitors should deepen the U, and the table above shows the spread growing from 5.5 points at 10 documents to 14.9 at 30. If nearness to the question is one of the two advantages, then shortening the distance between them should flatten the curve without raising its best case — which is what repeating the question after the document block does in the third row of the next table. If tokens that carry no evidence still consume budget, then wrapping every passage in structural markup should cost accuracy at every position and deepen the U as well, which is the fourth row. And if the effect is competition rather than reach, then a request that removes the competition — reproduce the passage at index 10, verbatim — should succeed everywhere, because it names the position instead of asking the model to find it among nineteen rivals. It does, and the last section of this chapter is about what that costs a team that has read the two results as contradicting each other.

None of this is settled, and the chain above is an account that fits the measurements rather than a mechanism anyone has traced end to end; a later chapter sets out three competing explanations for the length half of it and declines to choose between them. One link has been demonstrated directly. Found in the Middle: Calibrating Positional Attention Bias in Long Context Use shows that models allocate attention with a positional prior that is largely independent of relevance, and that explicitly correcting for that prior — measuring the attention a position receives regardless of content, then dividing it out — recovers a meaningful share of the lost accuracy. A prior that is independent of relevance is exactly what the third step above claims: the shape comes from where a passage sits and not from what it says. That correction is a research technique rather than something you can apply through a hosted API, but it tells you the bias is a property of how attention is distributed over positions rather than a quirk of any one training corpus, which is why it shows up across model families.

The plain version of all of it is that the middle of a prompt is not out of reach. It is crowded. The model is not failing to see the passage at index 10; it is failing to pick that passage out of a crowd, and the crowd is the rest of the prompt you assembled.

Model, template and length all move the curve#

The probe's most useful output is not the U. It is the discovery that the U you get is specific to the configuration you measured, so someone else's placement rule is not transferable to your system.

The third row is the one worth dwelling on, because it is a change to the prompt template rather than to the model or the data. Repeating the question after the document block — a two-line edit — halves the spread on this task without improving the best case, because the model no longer has to carry the query across twelve thousand tokens of documents before it encounters the instruction that tells it what to look for. The fourth row moves in the other direction: serialising each document as a JSON object with id, title and text fields, which is a perfectly reasonable thing for a pipeline to do, both lowers accuracy across the board and deepens the U, plausibly because the structural tokens dilute the passages and lengthen the prompt for the same content.

Model B in the second row is not simply better. It is flatter, which is a different property and the one that matters when you are deciding whether placement is something you need to manage. A model with a 6-point spread lets you order chunks by relevance and stop thinking about it. A model with a 15-point spread makes ordering a design decision with a measurable payoff.

Competition rather than availability#

The probe measures one thing: whether the model's answer changes when a fixed passage moves, with everything else held constant. It says nothing about whether the passage was retrieved in the first place, which is a separate measurement on a separate object, and confusing the two is how that morning got spent on the retriever. It also says nothing about what happens when the fillers stop being neutral. Everything in this chapter used passages that are on-topic but plainly do not answer the question; when the surrounding passages are near-misses that look like answers, the failure mode changes from "the evidence was ignored" to "different evidence was used", and the magnitude is larger. That case has its own chapter later in this topic.

Neither is this a statement that the middle of a context window is unreadable. Ask the model to reproduce the passage at index 10 verbatim and it will, at very high rates, at every position — a point a later chapter turns into an argument you can put in front of a stakeholder holding a vendor chart. The probe is measuring use under competition, not availability, and those two capabilities come apart badly enough that entire evaluations disagree about whether long context works.

That gap is where the most expensive wrong belief in this topic forms, and it is tempting for a good reason. If the model can quote the passage back, the passage was evidently available to it, and the natural conclusion is that any later failure to use the passage must be a failure of reasoning rather than of placement — which sends the investigation to prompt wording, to the model choice, or to the retriever, and never to the arrangement. The step being skipped is that the two requests ask for different work. Reproduce index 10 supplies the answer to the only hard part: it says which position to use. Answer the question does not, and the model has to establish which of twenty similar-looking passages is the one that bears on it before any of the content matters. A verbatim reproduction therefore tests whether the tokens are reachable, and a question tests whether they win, and a system can pass the first everywhere while losing points on the second in exactly the region the probe identifies.

Finally, a curve is not a rule. The numbers above are a snapshot of one model on one task with one template, and the honest summary of the current state is that the effect is robust in direction and unstable in magnitude: it moves with the model, the template, the task and the length, and it moves again after a provider updates a model behind the same name.

Position bias is the cheapest of this topic's effects to demonstrate and the easiest to act on once demonstrated, because the intervention is free: the same chunks, ordered differently, cost the same tokens and the same latency. What the probe gives you is not a placement to adopt but a spread to weigh — a system whose best and worst positions differ by six points can order by relevance and move on, and one where they differ by fifteen has a design decision on its hands, worth as much as most model upgrades and available this week. The number belongs to your configuration, so it is measured, kept next to your other evals, and re-measured when the model or the template changes.

The probe assembled its prompt from a list of passages that arrived in a fixed order, and every technique in the rest of this topic manipulates that list — how many entries it has, how they are ordered, which ones get dropped. That list has a producer with its own vocabulary and its own separate accuracy metric, and the next chapter names those pieces precisely, because the rest of the argument depends on being able to say which of two numbers you are talking about.