Sampling

Read these first

• Autoregressive generation

What you will know

• Change temperature and predict the direction of the effect before running it

  The effect depends on logit gaps and truncation; extreme values can make output effectively greedy or nearly unconstrained.

• Run identical logits through top-k and nucleus filters and record cases where candidate count changes as mass shifts

  Nucleus sampling uses an input-dependent candidate count, while top-k imposes a fixed cutoff that may retain too much or too little tail probability.

• Run a repeated-output case with and without a penalty, then inspect whether the prompt or context creates the repetition before choosing the fix

  A penalty can hide contradictory or underspecified context while introducing unwanted changes to legitimate repeated terms.

• Choose sensible sampling settings for extraction, and different ones for ideation, and justify both

  A setting that increases variety can silently damage parse rates, schema adherence, and exact-match evaluation.

• Run the same seeded request across two supported execution paths and record whether output, logprobs, or only metadata diverge

  Seeds constrain random draws but do not standardize floating-point order, batching, kernels, or provider revisions.

Chapters

1. 1

   Logits and the next-token distribution

   At each position a language model emits one score — a logit — for every entry in its vocabulary, and a softmax turns those scores into probabilities that sum to one across the whole vocabulary. Every sampling parameter acts on that object. This chapter shows what it contains, how its shape changes from a landslide to a five-way tie between one position and the next, how nucleus sampling cuts it by probability mass, and why mass is never a share of the vocabulary.

   Read chapter
2. 2

   Sampling temperature

   Temperature divides every logit before the softmax, which stretches or flattens the gaps between candidates without touching their order. The direction of the effect is fixed by the arithmetic; the size of it depends entirely on how far apart the top logits already were, which is a number you can read from a log-probability dump before you change anything. This chapter covers the maths, how to predict a change, how temperature interacts with nucleus sampling, and the failures it cannot remove.

   Read chapter
3. 3

   Top-k and nucleus sampling

   Both truncate the candidate set before a token is drawn, and they cut on different axes. Top-k keeps exactly k candidates whatever their probabilities, so it retains a long junk tail on a peaked distribution and discards most of the model's proposals on a flat one; nucleus sampling keeps however many candidates it takes to reach mass p, so the count moves with the distribution. This chapter runs one set of scores through both filters, records where they disagree, and shows where a fixed cutoff still earns a place.

   Read chapter
4. 4

   Truncation settings under greedy decoding

   Setting temperature to 0 makes the sampler take the highest-scoring token, and the highest-scoring token is inside every nucleus and every top-k set, so the truncation parameters sitting beside it cannot change a single output. The misreading that keeps the pairing alive is that p is a share of the vocabulary rather than cumulative probability mass. This chapter proves the inertness, catalogues the other settings that are quietly doing nothing, and shows the two-request test that confirms it.

   Read chapter
5. 5

   Frequency and presence penalties

   Frequency penalty subtracts from a token's logit in proportion to how often it has already appeared; presence penalty is a flat one-off charge for having appeared at all. Both suppress repetition, and both do it by pushing the model away from words it has good reason to reuse — entity names, order numbers, figures — which is how a repetition metric improves several points while factual accuracy falls. This chapter covers the arithmetic, how to find the cause of a loop before suppressing it, and the workload shape where a penalty is the right tool.

   Read chapter
6. 6

   Sampling settings for structured extraction

   Structured extraction wants the single most likely continuation at every position, because the output has to parse and there is one right answer per field. A temperature copied from an SDK quickstart buys occasional wrong tokens at exactly the positions where a brace or a field name decides whether the document parses, and a per-position risk of 0.0007 compounds to a four percent document failure rate. This chapter covers which settings an extractor needs, how to classify parse failures before changing anything, and what a second-call repair step is really buying.

   Read chapter
7. 7

   Generating distinct options

   Raising temperature makes a single completion less predictable; it does not make ten items in one list differ from each other, and past a point it buys incoherence rather than variety. Tighten the nucleus to cut the low-probability tail that produces nonsense, keep enough temperature for movement, and get distinctness structurally through separate calls, an enumerated prompt or oversampling with a similarity filter. This chapter also covers which of these settings can safely be exposed outside engineering, and which cannot.

   Read chapter
8. 8

   Seeds and reproducibility

   A seed fixes the random numbers a sampler draws with and nothing else. Batch composition, kernel selection, floating-point summation order, hardware and a silent provider revision all change the logits themselves, so the same seeded request can return different text on two execution paths. This chapter covers what the seed controls, why a near-tie flips under greedy decoding too, how to send one request down two paths and read the three possible outcomes, and what to log instead of relying on replay.

   Read chapter