Accuracy against example count

The count of demonstrations in a few-shot prompt is a free parameter, and the way it usually gets set is by picking a round number. Eight is a common choice, sixteen is common when somebody is worried about coverage, and neither figure comes from a measurement. The sweep that would settle it is cheap: hold everything else fixed, vary only the number of examples, and score each point several times so the run-to-run movement is known before any gap is read.

What the sweep returns is not a line that keeps rising. It is a curve with three regions — a steep climb, a flat top, and a decline — and each region has a different mechanism behind it. Knowing which region a proposed count sits in is the difference between a prompt that pays for its example block and one that pays for it twice, since the tokens are charged whether or not the accuracy column moves.

Reading the shape#

Here is the output of one such sweep. Same 400-case held-out set, same decoding, same model version, five runs at every point, repeat runs of an identical configuration varying by under half a point. The examples are drawn from the same pool of labelled tickets and appended in the same order, so that going from four to eight means keeping the first four and adding four more.

The first thing most people do with that table is read the top row against the bottom row, conclude that few-shot prompting is worth 9.4 points, and pick sixteen because more context is more information. The shape says something narrower and more useful. Almost the entire gain — 8.2 of the 11.1 points ever available — arrives in the first two examples. The next two buy 2.9 more at nearly three times the token cost per point. The four after that buy nothing that survives the noise floor — the size of gap that repeat runs of a byte-identical prompt produce on their own, below which a difference cannot be attributed to the change — since a change of 0.1 against a spread of 0.5 is a way of writing "no measured difference". And the last eight are the only ones with a clear sign on them, which is negative.

Each of those claims is a subtraction on the table, and they are worth doing rather than taking. The most any quantity of demonstrations ever delivers on this task is the highest accuracy in the column against the lowest, and the highest is 89.1 at four examples against a zero-shot 78.0, so the whole prize is 11.1 points. The first two examples take the score to 86.2, which is 8.2 of those 11.1 — nearly three quarters of everything that was ever available, bought before the block is a page long. Reading the top row against the last row instead gives 87.4 against 78.0, or 9.4 points, and that figure comes out smaller than the prize because the last eight examples have already handed some of it back.

The final column is a division, and it is the one that turns a shape into a decision. Tokens per point gained is the extra tokens a row costs over the row above it, divided by the extra points that row buys. Two examples take the prompt from 610 tokens to 1,540, which is 930 extra tokens, and they buy 8.2 points; 930 divided by 8.2 is a shade over 113. Four examples take it from 1,540 to 2,470, which is the same 930 tokens, and they buy 2.9 points; 930 divided by 2.9 is just under 321. The same quantity of text, at the same price, on every call, buying a bit over a third as much the second time. That one comparison is the shape of the curve stated as an exchange rate, and it is available before any argument about mechanism.

The last two rows have no entry in that column, and the blank is arithmetic rather than an omission: their change column is zero or negative, and a token cost divided by a gain of nothing does not produce a number. Going from four examples to eight adds 1,860 tokens — 4,330 against 2,470 — for a change of −0.1. Going from eight to sixteen adds three thousand seven hundred and twenty more, and takes 1.6 points off.

The instinct that reads the bottom row and picks sixteen deserves better than being called careless, because it is the correct instinct nearly everywhere else an engineer meets it. More labelled rows do improve a fine-tuned model. More documents in a retrieval index do improve recall. More cases in an eval do make its verdict more trustworthy, which is an argument this topic makes elsewhere. And a modern context window has ample room for sixteen tickets, so nothing about the request even looks expensive in the way a request for more training data does.

The fine-tune is the closest of those comparisons, and it is worth handling as a comparison rather than letting it operate unexamined. Adding one row to a training set of fifty thousand contributes one update among fifty thousand, and every row that disagrees with it pulls the other way, so the influence of any single row is its share of an average. Adding one demonstration to a prompt does something with no counterpart in that picture. It changes the document the model is being asked to continue, and nothing in a request averages one demonstration against another the way a training run averages one row against another — the model scores the next fragment given the whole of what sits in front of it, once — so a demonstration is not diluted by its neighbours. It is read alongside them. The comparison holds for the fact that both are examples with labels attached, and it breaks at exactly the property that decides this chapter, which is that a prompt is one document rather than a bag of independent rows.

The reason the first two examples are worth so much is that they are doing several different jobs at once, and each job only needs doing once. The first demonstration establishes the output shape, kills the preamble, fixes the label vocabulary to the strings you actually use, and sets the register. The second usually lands on a different class and so establishes that the labels vary rather than being a constant. After that, the structural work is finished, and every subsequent example is only worth whatever new coverage it brings.

Each of those jobs is a property of the document rather than a fact about a case, and that is what makes them one-time purchases. Take the output shape. The model is scoring the next fragment of a document in which a correctly shaped output already appears, immediately above where the next one is due and in the same position relative to an input. The shape is established the instant one instance of it exists there, and a second instance carries no information the first did not. The preamble goes for the same reason — a demonstration whose answer begins with the answer is evidence that answers in this document begin with the answer — and so does the label vocabulary, since the strings your team actually uses are now present in the document and the model is choosing among fragments it can see rather than inventing them.

The second demonstration is the one exception, and it earns its slot by ruling something out rather than by establishing something. A single example is perfectly consistent with the label being a constant: every case in this document, all one of them, came back billing. A second example landing on a different class is the first evidence that the label depends on the input at all. Once that is settled there is no third property of the same kind left to fix, which is why the structural work finishes at two and everything past it is coverage.

Coverage is the right way to think about the rest of the curve. Each example demonstrates a decision on one region of the input space, and it helps on a test case to the extent that the test case resembles it in whatever way the model finds salient. Your six queues, your two or three ticket lengths, your handful of common ambiguity patterns — there are perhaps five or six meaningfully distinct kinds of decision in the whole task, and a randomly drawn example set covers them the way a coupon collector fills a set. The first few draws are almost always new. By the eighth draw, the odds that the new example demonstrates something none of the previous seven demonstrated are poor, and what you are mostly buying is a second, third and fourth instance of "short single-issue billing ticket goes to billing", which the model was already getting right at 97%.

The coupon collector is an analogy, and the place it fails matters enough to name. In the textbook version the number of distinct coupons is known, every draw is equally likely to land on any of them, and the expected number of draws needed to complete the set can be written down in advance. None of those three holds for an example pool. You do not know how many meaningfully distinct kinds of decision your task contains — five or six is an estimate from reading failures, not a count — and your pool is not uniform over them, because it was drawn from traffic and traffic is dominated by the easy cases. What survives the comparison is the shape of the returns: early draws are almost all new, later draws are almost all repeats, and the crossover arrives sooner than people expect. What does not survive is any calculation of where the crossover sits, which is why the sweep gets run rather than derived.

Three mechanisms that bend the curve down#

Coverage accounts for the flat section. The decline needs a different explanation, because if extra examples were merely redundant the curve would plateau rather than bend down. Three mechanisms push it down, and they act together.

The first is that hand-assembled example sets contain internal disagreements, and the disagreement rate rises with size. Two annotators, or one annotator on two different afternoons, will resolve the double-charged-and-late ticket differently, and with sixteen examples the chance that your block contains one pair with opposite conventions on the same boundary is high. The model does not adjudicate between them; it becomes less decisive on exactly the inputs that sit near that boundary, which is the slice you most wanted help with. A later chapter, on what one confidently mislabelled demonstration does, takes this further, because a single wrong label does something worse than making the model hesitant.

The second is that a long example block carries a distribution, and the model reads it. If your sixteen examples happen to contain seven billing cases because billing is your largest queue and you sampled proportionally, the model's prior shifts toward billing, and it will resolve close calls that way regardless of the input.

Reading a distribution sounds like something the model would have to be doing deliberately, and it falls straight out of what the model is for. At the point where the label is due, the model is scoring every candidate next fragment against the whole document above it, and one property of that document is that the string billing has already appeared in it seven times, in exactly this position, immediately after a ticket body. Text repeats itself: a token that has occurred often in a document is more likely to occur again, and that regularity holds across essentially everything the scoring function was shaped on. So the score of billing at the decision point is raised by the block's composition, by an amount that owes nothing to the ticket being classified. On a case where the input leaves the top two candidates close together — which is precisely the case the examples were added to help with — a nudge of that size is enough to settle it.

The measured version of that is well documented — the calibration work that came out of Stanford in 2021 named majority-label bias and recency bias explicitly, showing that the class distribution of the demonstrations and the label of the last example both move predictions measurably — and it grows with the number of examples, because a longer block carries a sharper apparent distribution. The recency half of that finding is the same property read at a shorter range, since the demonstration nearest the point of decision is the one the next fragment most directly continues. Two examples do not look like a sample of anything. Sixteen do.

The third is positional. The user's actual input now sits at the end of 8,000 tokens of other people's tickets, all of which look like the thing the model is being asked to classify. That is not a context-window problem in the sense of running out of room, since 8,000 tokens is nothing for a modern model, and it is worth keeping the two apart. It is an interference problem: sixteen full ticket bodies give the model sixteen opportunities to latch onto a phrase from a demonstration rather than from the input, and the failures it produces look like a confident label for a ticket that is not the one being classified.

The difference between interference and running out of room is worth being concrete about, since the two get the same name in conversation. Running out of room is a hard failure carrying an error with it: the request exceeds what the model will accept and the provider rejects it, so you find out immediately and you find out from the API. Interference is a soft failure inside a request that fits comfortably. The model is scoring the continuation of one document, and that document now holds seventeen ticket bodies of which sixteen are decoration and one is the question. Nothing in the string marks which is which — the tags or headings you wrapped them in are text in the same document, read by the same scoring pass — so a phrase in a demonstration is competing on equal terms with a phrase in the input for whatever the model finds salient. When the demonstration's phrase wins, the reply is a well-formed label at ordinary confidence for a ticket nobody asked about, and the only way to see that is to read the demonstration it came from.

The knee, not the maximum#

The token column is the part that turns a curve into a decision, and it deserves a second look. Going from four examples to eight costs 1,860 additional input tokens on every call, forever, and buys −0.1 points. Nobody would approve that as a written proposal. It gets approved constantly as a default, because eight is a round number and the incremental cost never appears anywhere a human reads.

The habit that makes the sweep worth running rather than merely interesting is choosing the operating point from the knee rather than from the maximum. The knee is the place where the curve bends from climbing steeply to running roughly flat, borrowed from the way anyone reading a latency-versus-load graph picks the point past which more load stops being free. The maximum of the curve above is 89.1 at four examples, and 89.0 at eight is statistically the same number; a team optimising the accuracy column will report the higher of two indistinguishable values and take the more expensive configuration for free. Reading the knee instead means asking where the curve stops climbing faster than the noise, taking the smallest count at that point, and treating everything to the right of it as something that must justify itself against the cost column. On this task that is four, and it is four regardless of how the last three rows are ordered.

Finding the knee on your own sweep needs nothing beyond two columns you already have. Take the noise floor first, which is the spread across repeat runs of one unchanged configuration — half a point here. Then walk down the change column and find the last row whose gain clears that floor by a margin you would defend: 8.2 at two examples and 2.9 at four both clear half a point comfortably, and −0.1 at eight does not clear it in either direction. The climbing stops after four, so four is the knee. The reason it is four rather than eight is that 89.1 and 89.0 are not a maximum and a runner-up. They are one number measured twice, and a spread of half a point means a repeat of the four-example configuration could have come back a couple of tenths lower while a repeat of the eight-example one came back a couple of tenths higher, reversing the order of the column without a single example changing.

The interesting cases are the ones in the middle, where the trade is real. Suppose the sweep on your extraction prompt says six examples score 91.2 and two score 89.7, a difference of 1.5 points well outside the noise, and the four examples in between are 2,400 tokens per call. The right response is not to pick a side from the accuracy column alone, and it is also not to pick from the cost column alone. What the sweep has given you is an exchange rate — 1.5 accuracy points for 2,400 tokens per call — and the decision needs the other half, which is what those tokens actually cost in money and in latency at your volume, and what 1.5 points is worth in whatever the errors do downstream.

One sequence of examples is not example count#

A curve like the one above is a measurement of one particular sequence of examples, and it is easy to read it as a measurement of example count in general. Those are different things, and the difference shows up the moment you re-run the sweep with a different draw from the same pool:

import random, statistics

def sweep(pool, held_out, counts=(0, 2, 4, 8, 16), draws=5, seed=11):
    rng = random.Random(seed)
    for k in counts:
        results = []
        for _ in range(draws):
            block = render(rng.sample(pool, k))          # a fresh draw each time
            results.append(score(BASE + block, held_out))
        print(f"k={k:>2}  mean={statistics.mean(results):5.1f}  "
              f"min={min(results):5.1f}  max={max(results):5.1f}")

k= 0  mean= 78.0  min= 78.0  max= 78.0
k= 2  mean= 84.1  min= 79.6  max= 87.9
k= 4  mean= 88.3  min= 85.2  max= 90.4
k= 8  mean= 89.0  min= 87.6  max= 90.1
k=16  mean= 87.4  min= 86.5  max= 88.2

The mean curve has the same shape, and the columns either side of the mean are the ones to read. At two examples the luckiest draw scored 87.9 and the unluckiest 79.6, so the spread across draws is 8.3 points. Set that against what the rest of the curve is worth on the same table: the mean at sixteen is 87.4 against a mean at two of 84.1, so every example after the second is delivering three and a third points between them all. The choice of which two examples to use moves the score by about two and a half times as much as the choice of how many examples to have. Which two examples you pick matters more, at that point on the curve, than whether you have two or eight. The spread narrows as k grows, because a larger sample of examples is a more stable estimate of the pool, and that is a genuine argument in favour of larger blocks for anyone unwilling to curate: more examples is a way of buying robustness against a bad draw. Curating four good ones beats sixteen random ones and costs a quarter of the tokens, which is why the two chapters that follow the cost argument spend their time on which examples rather than how many.

Ordering has a similar effect at the small end and it is worth knowing about before you attribute a two-point swing to a content change. Permuting the same four examples can move accuracy by several points on some tasks, with the last example exerting more influence than the first, so a sweep that appends examples in a fixed order — as the opening table did — is holding one confound constant rather than eliminating it. If you need the count decision to be solid, average over shuffles as well as over runs.

None of this makes the curve unreal, and its general shape has been stable since the behaviour was first characterised. The GPT-3 paper in 2020 reported accuracy against the number of in-context examples across dozens of tasks, sweeping K from zero into the tens, and the pattern there is the same one you just measured: a steep early climb, a knee, and a long flat top that on several tasks turns downward. What has changed since is where the knee sits. Instruction-tuned models — base models given a further training pass on large collections of written instructions paired with correct responses, which is why a current model does something sensible when you simply tell it what you want — arrive already knowing how to follow a task description, so the zero-shot end of the curve has risen a great deal — the 78 in your table would have been far lower on a 2020 base model — and the knee has moved left. The practical consequence is that a habit formed on older models, where eight or sixteen examples were genuinely doing work, now buys the flat part of the curve at full price.

It is also worth being clear that this curve is not a scaling law and does not transfer. There is no universal knee at four; there are tasks whose knee is at one, because all the examples were ever doing was fixing the output shape, and tasks with a dozen genuinely distinct input kinds whose knee is further right than any block you would want to pay for. A task where the label set is large — a taxonomy of forty product categories, say — cannot be covered by demonstrations at all, since even one example per class is forty examples, and the flat-then-declining region arrives long before coverage is achieved. On tasks like that the sweep is still worth running, and what it usually tells you is that examples are the wrong instrument and the label definitions belong in the instructions.

So the count question has a defensible answer and it is a smaller number than the default. Sweep the counts you would plausibly ship, read where the climb stops outrunning the noise, take the smallest count at that point, and require every example past it to be argued for against what it costs. The curve tells you what quality each choice delivers. It says nothing about what each choice costs once it is running, and the flat section between four and eight examples is only a decision when both numbers are on the table.