Few-shot prompting for base models

Base checkpoints are cheaper to host, often materially so, and that is the whole reason anyone reaches for one in an application rather than as a fine-tuning starting point. What makes the saving hard to collect is that a base model completes documents and has never been shown the pattern of a request followed by a helpful reply, so the prompt that worked against an instruct model — an imperative instruction, addressed to a second person, expecting an answer to follow — arrives as the opening of a document rather than as a question.

Where that saving comes from is worth stating, because it is not where most people assume. The two checkpoints have the same architecture and the same parameter count, so generating a token costs a host the same arithmetic either way. The lower price is a pricing decision — thinner demand, a different licence, a cheaper serving tier — rather than a property of the weights, which makes it a fact about one provider's rate card on one day rather than a rule about base models in general. Read the two rates you are actually being quoted before doing any of the arithmetic below, since the whole calculation turns on the ratio between them.

The diagnostic worth carrying is the shape of the failure. A capability regression produces bad text: contradictions, wrong arithmetic, sentences that fall apart. A serialisation and prompting regression produces good text aimed at the wrong target, and an eval measures the target rather than the prose. This chapter is about the form of prompt a base checkpoint is actually for, the arithmetic that decides whether it saves money, and the things it cannot do at all.

Writing the prompt as a document#

The migration that works starts by giving up the register entirely. A base model completes documents, so the prompt has to be a document that is obviously unfinished at exactly the point where the answer goes, and the strongest way to establish what belongs there is to show the pattern several times. That is what few-shot prompting means: a handful of completed examples stand in the prompt as the specification, and the model completes the next one in the series.

Why that works is worth taking down to the level of the scores, because it is the same mechanism that made the instruct checkpoint work, reached from the other side. A model scores the next token given everything before it. After three completed Receipt: blocks, each followed by a JSON: line and an object, the document has an unmistakable genre, and in that genre what follows a JSON: line is an object. So the highest-scoring token at that position is an opening brace, and every token after it is shaped by the object already open. Instruction tuning made a direct answer the most probable continuation of a request. Few-shot prompting makes the next item in a series the most probable continuation of a document. Neither one installs a capability. Both are ways of arriving at a position from which the behaviour you want is the ordinary thing to do next.

FEWSHOT = """Receipt:
PRET A MANGER 11 BROAD ST
2026-03-14  14:22
FLAT WHITE            3.40
TOTAL                 3.40 GBP
JSON: {"merchant": "Pret A Manger", "date": "2026-03-14", "total": 3.40}

Receipt:
TESCO EXPRESS 42 HIGH ST
15/03/26  09:04
MEAL DEAL             3.90
LOYALTY BALANCE      12.00
TOTAL                 3.90 GBP
JSON: {"merchant": "Tesco Express", "date": "2026-03-15", "total": 3.90}

Receipt:
UBER *TRIP HELP.UBER.COM
2026-03-16
FARE                 18.50
TIP                   2.00
TOTAL                20.50 GBP
JSON: {"merchant": "Uber", "date": "2026-03-16", "total": 20.50}

Receipt:
"""

r = client.completions.create(
    model="meta-llama/Llama-3.1-8B",
    prompt=FEWSHOT + ocr_text.strip() + "\nJSON:",
    max_tokens=64,
    temperature=0,
    stop=["\n\nReceipt:", "\n\n"],
)
print(r.choices[0].text.strip())
# {"merchant": "Costa Coffee", "date": "2026-03-18", "total": 4.75}

Four decisions in that snippet carry most of the difference. The label JSON: at the end is the generation prompt — the position from which the only plausible continuation is an object — and it does the job the assistant header does in a chat template, with the important difference that it is ordinary text and therefore forgeable by anything inside ocr_text. The examples are separated by a delimiter that appears nowhere inside them, so the boundary between items is unambiguous. The examples deliberately include the awkward cases — a 15/03/26 date that must be normalised, a LOYALTY BALANCE line that is not the total — because in a completion prompt the examples are the only place instructions can live. And stop is not optional: without it the model finishes the object, writes \n\nReceipt:, and invents the next one, running to max_tokens every time.

That last point is worth a number. On 500 receipts, the same base prompt without stop sequences produced a median of 512 output tokens per call against 34 with them, so removing four characters of configuration multiplies the output bill by fifteen and the latency along with it. Instruct variants hide this, because <|eot_id|> is a learned stopping behaviour and the serving stack stops on it by default. Take the chat template away and stopping becomes your responsibility again.

That the document runs on is the price of the mechanism rather than an oversight in it. What makes the prompt work is that the document has a pattern, and the pattern does not stop after one more object. Three Receipt: blocks establish that a Receipt: block follows a completed object, so the likeliest token after the closing brace is the start of a fourth one. The very property being exploited to get an answer out is the property that keeps the document going once the answer has arrived, which is why the two cannot be separated and why the boundary has to be declared from outside.

There is also a decision not visible in the snippet: temperature=0. Few-shot completion prompts are more sensitive to sampling than chat prompts are, because the format is being held in place by pattern matching rather than by trained behaviour, and a single unlucky token that starts a sentence instead of a brace takes the rest of the output with it.

The asymmetry there is between two ways a format can be held in place. An instruct checkpoint's output format is held by training, so the weights make a well-formed reply likely across a wide band of sampling settings, because that is what the post-training data looked like every single time. A few-shot format is held by the prompt alone, and the margin between the right first token and a plausible wrong one is only as wide as three examples managed to make it. Sampling spends that margin. At temperature zero the highest-scoring token is taken every time and the width of the margin stops mattering; above zero, the margin is exactly what is being gambled with.

The arithmetic of the saving#

A rewritten prompt recovers most of the accuracy. Whether it saves anything is a separate question with a computable answer.

The 720-token few-shot prefix is the whole story. It is charged on every call, at a lower rate than the instruct model but on three times the volume, and at a task input of roughly 300 tokens per receipt that trade comes out eleven percent worse. The break-even is easy to compute and worth computing before a migration rather than after: with a fixed prefix of 720 tokens, a per-call task input of T tokens and comparable output lengths, the two paths cost the same when 0.20 × (T + 78) = 0.08 × (T + 754), which puts the crossover near T = 370. Below that the instruct model is cheaper despite costing more per token; above it the base model pulls ahead and keeps pulling, because the prefix is fixed while T is not. A service processing 8,000-token contracts would see the base path land at roughly a third of the cost.

Two levers move that crossover. Prompt caching, where the provider stores the attention keys and values for a shared prefix and bills it at a fraction of the input rate, applies almost perfectly here because the 720-token prefix is byte-identical on every call; at a typical cached rate of $0.02 per million the base path drops to about $0.44 per 10,000 calls, a genuine 42% saving. The other lever is shortening the prefix, which is worth testing rather than assuming. Neither lever is available while the prompt is still written in chat style, which is the real cost of the naive migration: it forfeits a workable option by producing evidence that the option does not work.

Choosing the examples is where the work goes#

In a chat prompt the instructions carry the task and examples are a supplement. In a completion prompt the examples are the entire specification, so how many and which ones is the design work rather than a tuning detail. Sweeping the count on the same 500 receipts:

The curve flattens after three, which is typical and worth finding rather than guessing: the sixth and twelfth examples together bought one point of accuracy for 138% more cost. Three examples at $0.59 per 10,000 calls beats the instruct baseline outright, which reverses the conclusion the six-example table reached.

That curve has its shape for a reason, and the reason also tells you which examples to keep. Each example does two jobs. It makes the genre more certain, and it demonstrates one particular way the task can be awkward. The first job finishes quickly: one example already establishes that this is a series of receipts followed by objects, and a second makes it unambiguous. The second job never finishes at all — it simply runs out of distinct awkwardness left to demonstrate. So an example beyond the third earns its tokens only if it shows something the earlier ones did not.

Which three matters as much as how many. The productive source is an eval's own failure list: run the current prompt, take the items it gets wrong, and turn the distinct failure kinds into examples — one for each date format that occurs, one for the receipt with a loyalty balance below the total, one for the merchant name split across two OCR lines. Examples drawn at random from the head of the distribution teach the model something it already does. Ordering has an effect too, smaller but real: shuffling the same three examples moved exact match by up to 0.8 points across seeds, and the ordering that worked best put the awkward case last, closest to the item being classified. That is small enough to be noise-adjacent and large enough that the order should be fixed in code and treated as part of the artifact rather than regenerated per request.

One serving detail catches people out. A base checkpoint has no chat template, so a chat request against it either fails or gets rendered with whatever default the stack falls back to — the wrong-template state from chapter 3, where a stack with no template of its own reaches for ChatML — and a provider that silently applies ChatML to a base model will make a correct completion prompt look broken. Use the completion route for base checkpoints, and if the only route available is a chat one, that provider is the wrong host for this plan.

Where the cheaper checkpoint is the wrong answer#

Few-shot completion prompting is not a general substitute for instruction following, and the tasks it suits are narrower than a cost table suggests. It works when the task has a fixed shape that can be demonstrated, the output is short and highly structured, and the same prompt runs thousands of times so the prefix is amortised and an eval is worth building. Classification, extraction, normalisation and reformatting all fit that description.

The things it does not do are structural rather than a matter of prompt craft. There is no multi-turn conversation, because there is no turn structure to hold one; you can fake a dialogue as a transcript, and the model will happily write both sides of it. There is no system prompt, so policy and constraints have to be smuggled in as patterns in the examples, and a constraint that only applies rarely will not appear in three examples at all. There is no tool calling in the trained sense, since the tool-call markers and the behaviour of emitting them come from post-training. There is no refusal behaviour and no safety tuning, which means a base checkpoint will complete text that an instruct one declines, and any product surface exposed to user input needs its own filtering rather than inheriting the model's. Long instructions with conditional logic — "if the total is missing, use the sum of line items, unless a service charge is listed separately" — are exactly what instruction tuning bought, and expressing them through examples requires one example per branch.

The pattern running through that list is worth naming, because it predicts the next item rather than merely enumerating the known ones. Every capability that survives the move lives in the shape of a document: a series can be demonstrated, a format can be demonstrated, a normalisation can be demonstrated. Every capability that does not survive lives in a marker, or in a trained response to one — a turn boundary, a system block, a tool-call token, a refusal. Prompting can change what a document looks like. It cannot add an entry to the vocabulary, and it cannot make a token likely that post-training never made likely. That line is where few-shot prompting stops, and it is a sharper line than a list of caveats suggests.

The historical framing helps, because this style is not new. Before instruction tuning was widely available, few-shot completion prompting was the only way anyone used a language model; the original GPT-3 work was a demonstration that a base model given a handful of demonstrations could do tasks it was never trained for, and prompt engineering at the time meant choosing and ordering those demonstrations. Instruct checkpoints made the technique unnecessary for most work, and the skill decayed accordingly, which is why a team reaches for a base model with a chat prompt today and reads the result as a capability gap. The technique still works. It is no longer the default, and the surrounding tooling — chat SDKs, message objects, structured-output helpers — is all built for the other path.

One case inverts the whole calculation. If you intend to fine-tune, the base checkpoint is frequently the right starting point rather than a cost compromise, because you are supplying the post-training yourself: your own examples, your own template, your own stopping token, with no instruction-tuned behaviour to argue against. That is a different project with a different budget, and it is the point at which the choice between variants stops being a prompting decision.

A base checkpoint is cheaper per token and is not automatically cheaper per call. The prompt has to be rewritten as a document with demonstrations and stop sequences before the weights can show what they know, the fixed prefix has to be amortised over enough task input to beat the instruct model's higher rate, and the capabilities that post-training supplied — turns, system messages, tool calls, refusals, conditional instructions — do not come back through prompt craft. When those three conditions hold, the saving is real and worth collecting; when they do not, the honest outcome of the experiment is that the instruct model was already the cheap option.

The receipt task got off lightly because it has one dimension that matters. A task with an image in it, a schema to satisfy and a latency budget has several, and they do not point the same way.