Zero-shot, few-shot, and example selection

Read these first

• Anatomy of a prompt

What you will know

• Get a task working zero-shot before adding examples

  Without a held-out test set and fixed decoding settings, later gains are indistinguishable from sampling noise or prompt drift.

• Choose examples that cover the edge cases rather than the obvious case

  Coverage should include ambiguous inputs, formatting violations, and adversarial cases, because polished happy-path demonstrations hide the errors users actually report.

• Measure the token and latency cost of few-shot examples on every call

  A 500-token example at high volume can dominate inference spend and shrink the room available for the user's input and the model's output.

• Show a case where a bad example actively taught the wrong behaviour

  The model can copy an incorrect label convention or unsupported claim, turning one annotation mistake into a systematic output bias.

• Decide when examples should be selected dynamically per input

  Similarity retrieval can select superficially related but semantically misleading examples, so the selector needs a fallback and evaluation against a fixed prompt.

Chapters

1. 1

   The zero-shot baseline

   An eight-example classifier scoring 91% on a 400-ticket held-out set says nothing until the same prompt with the example block deleted has been scored on the same cases with the same decoding. This chapter covers running that ablation, the run-to-run spread that decides whether a gap is real, and the four jobs — output shape, label-boundary convention, register, tie-breaking — that demonstrations do better than sentences, which is also why a three-class task with self-describing labels rarely needs them.

   Read chapter
2. 2

   Accuracy against example count

   Measured on a fixed 400-case set with repeats varying under half a point, accuracy can run 78.0, 86.2, 89.1, 89.0, 87.4 as examples go 0, 2, 4, 8 and 16. This chapter reads that shape: the structural work finishes in the first two demonstrations, coverage saturates after that, and internal disagreement, majority-label bias and interference bend the curve back down. Operate at the knee rather than the maximum, and treat everything to the right of it as something that must justify itself against the token column.

   Read chapter
3. 3

   Pricing the example block

   Five 500-token examples against a 400-token input and a 200-token output means the block is roughly 2,500 of 3,100 input tokens on every single call. This chapter prices that at real volume, counts the same tokens a second time as prefill work in time-to-first-token and p95, and covers the context budget they take from the user's input. It states what prompt caching actually gives you — a discount rather than a zero, at a hit rate bursty traffic never drives to 100% — and joins the price back to the example-count curve, so trading 1.5 points for four fewer examples becomes a decision with both numbers in front of you.

   Read chapter
4. 4

   Per-slice evaluation

   An eval printing 89.0 with 44 errors out of 400 supports no decision about which examples to keep. Cut the set into named groups — by language, by ticket shape, by class, by whatever your failures cluster into — read a run's errors into those groups instead of counting them, and score every later change per slice alongside the aggregate. This is the instrument that makes a concentrated bias visible while the mean barely moves, and the one that tells you what to look at before shipping a change that wins by three points overall.

   Read chapter
5. 5

   Coverage-driven example selection

   With room for four examples and failures clustering into multi-issue tickets, Portuguese tickets and one-line screenshot captions, the clean cases you already pass at 97% teach nothing. This chapter turns example selection into a coverage decision driven by your error analysis: one demonstration per failing kind, plus one clean case to anchor the output shape, deliberately including ambiguous, malformed and adversarial inputs — and shows what that block scores against random and class-stratified selection on the same set.

   Read chapter
6. 6

   Mislabelled demonstrations

   A single hand-annotated example labelling a clear refund request as a billing question does not produce a scatter of random errors. The model generalises the mislabel into a rule, so an entire slice of similar inputs routes the wrong way — 11 of 24 refund cases wrong and all 11 in the same direction — while the aggregate moves 2.3 points and gets attributed to a model version bump. This chapter covers that signature, the leave-one-out ablation that names the culprit, and the same failure on generative tasks.

   Read chapter
7. 7

   Nearest-neighbour example selection

   Retrieving the five or eight nearest labelled examples per input can move aggregate accuracy from 88 to 91 while one slice gets markedly worse. The mechanism is specific: near a class boundary the closest neighbours are superficially similar cases carrying the opposite label, so you demonstrate exactly the wrong answer on the hardest inputs. This chapter covers measuring the retrieved block's label purity, what a selector gives up in reproducibility, caching and pool hygiene, and the fallback to build before shipping one.

   Read chapter
8. 8

   Ageing example sets

   Twelve examples copied from production outputs that looked good one afternoon eighteen months ago will carry retired policies, an old customer mix and unaudited labels. This chapter traces how a stale assertion damages current answers, why deleting the single bad example still passes an eval drawn from the same era, and what provenance fields, lifecycle states and review triggers the example set needs to stop being a liability.

   Read chapter