Logprobs and token-level confidence

Read these first

• Sampling

What you will know

• Request logprobs from an API and read the result

  Most providers can return per-token probabilities. Getting them once demystifies what 'confidence' can and cannot mean.

• Build a cheap classifier confidence score from a single token's logprob

  For a single-token answer like a label, its probability is a genuinely useful confidence signal at almost no cost.

• Measure how average logprob changes with output length on correct and incorrect answers, and diagnose why it fails as a confidence score

  Length normalization can make verbose and concise responses look comparable while hiding the few tokens that determine correctness.

• Identify a case where the model is confidently wrong and logprobs don't reveal it

  Fluent, high-probability text can be entirely fabricated. Probability measures typicality, not truth.

• Compare logprobs with a second-call or multi-sample confidence method on a fixed evaluation set, then choose the cheaper reliable option

  Provider support, sampling behavior, added latency, and answer correlation can make an apparently simple confidence signal misleading or uneconomical.

Chapters

1. 1

   Token logprobs

   Most providers will return the log probability of every token a model emitted, and often the top alternatives it considered, behind one flag on the request. This chapter covers how to ask for them, why they come back as logarithms, what the array actually looks like on a real completion — mostly positions the model was never in doubt about — and what the numbers cost to store. Seeing that shape is what keeps the word confidence from doing unearned work later.

   Read chapter
2. 2

   Single-token classifier confidence

   When the whole response is one label drawn from a closed set, that token's probability is a real statement about how the model spread belief across the options, and it costs nothing to obtain. This chapter builds the score — checking that each label is genuinely one token, renormalising over the valid keys — and reads it against two thousand adjudicated tickets to see what it orders and what it fails to promise.

   Read chapter
3. 3

   Mean logprob and length normalisation

   The mean of a completion's logprob array is the most common confidence metric in production, and most of the terms in it are commas, articles and conventional sentence openings. This chapter decomposes the average on a real answer, shows why dividing by token count trades a crude length dependence for a subtler one, measures correct against incorrect answers within length bands, and covers the two things the statistic is genuinely good for.

   Read chapter
4. 4

   Hallucination and token probability

   A fabricated case citation comes out with high probability on every token, because a plausible-looking citation is exactly what the surrounding text makes likely. Probability describes the typicality of a continuation under the model's distribution and has no channel to the world. This chapter traces a real fabrication token by token, compares it against a genuine citation, explains why the training objective contains no term for truth, and gives the procedure for finding a confidently wrong example in your own domain.

   Read chapter
5. 5

   Self-consistency sampling

   When the answer is prose rather than a label there is no decisive token to read, so the standard substitute is to sample the same prompt several times and measure how much the answers agree. This chapter covers where the method came from, what N samples cost in money and in tail latency, why a five-sample score has six possible values and a standard error of 0.18, and the test that tells you whether an apparently free-form decision is really a label decision wearing prose.

   Read chapter
6. 6

   Sampling temperature and answer equivalence

   A self-consistency score has two settings that nothing forces you to choose: the temperature the samples are drawn at, and what counts as two answers agreeing. Five samples at temperature 0 are one answer copied five times and score 1.00 on every case, correct or not. This chapter measures how the temperature sets the ceiling on observable disagreement, compares four definitions of agreement on the same five samples, and shows the extraction that puts the right case in front of a human.

   Read chapter
7. 7

   Confidence thresholds and calibration

   Logprob distributions differ between models, so the cutoff that gives 95% precision on one model is 0.83 and on another is 0.991, and comparing two models' confidence numbers is not a comparison of reliability. This chapter separates ranking from calibration, computes expected calibration error on real bins, gives the five-part procedure for deriving a threshold from a labelled set, and covers the canary that catches a model revision before the review queue drains.

   Read chapter
8. 8

   Choosing a confidence signal

   Which confidence signal to buy is a measurement problem, not a design-taste problem: score every candidate against one fixed labelled set at the review volume you can actually staff, and read off how much of the error each one catches. This chapter works the comparison for a clinical triage assistant on a doubled budget, measures the overlap between what two signals catch, and ends with the written statement of limits that has to accompany the number.

   Read chapter