A logprob scores the phrasing, not whether the passages said it

The sequence logprob of a generated answer is the sum of log p(token_i | tokens_<i, prompt) over the tokens the model emitted, and topic 1.7 covered how those values are produced. It is attractive as a confidence signal for three reasons that all hold: it costs nothing beyond a flag on a request you were already making, it is a real probability rather than a number the model chose to say, and it varies smoothly rather than landing on four round values.

What it measures is a different question from what it costs, and the answer is more specific than "confidence". Dividing by the token count removes the length artefact and leaves a per-token average, and that average has a plain reading: on this sequence, how unsurprised was the model by its own next word. Nothing in that quantity is about the world, the passages, or whether anything asserted is true. It is about the sequence being probable, and a fluent fabrication is exactly the kind of sequence a model finds probable.

What the number is measuring#

The raw sum has to be normalised before it can be compared across answers, and the reason is arithmetic rather than statistical. Every token's logprob is negative, since every probability is below one, so every additional token subtracts something from the total. A 20-token answer therefore beats a 120-token answer nearly always, and ranking by the raw sum is ranking by brevity with the appearance of ranking by confidence. Dividing by the token count removes exactly that dependence and leaves a per-token average that can be compared between answers of different lengths.

That quantity is a fluency measure, and the clearest way to see it is that language models were scored with it long before they generated anything anybody read. The statistic is perplexity, the exponential of the negative mean logprob, and it is the number that told you whether a model had learned English: it went down when text got more predictable rather than when it got more true. Nothing about the statistic changed when it was repurposed; only the question being asked of it did.

An analogy makes the mismatch vivid and it needs its limits stated, because it is a picture rather than a fact. Using a mean logprob as a correctness signal is like asking a spell-checker to fact-check: it will reliably flag the sentence that reads strangely, and a wrong sentence in confident prose reads perfectly well. Where the analogy breaks is that a spell-checker's verdict is a fact about a fixed lexicon, so it is at least stable — the same word gets the same verdict whatever surrounds it. A logprob's verdict is a fact about a conditional distribution that the retrieved passages themselves shift, so the same sentence can score differently depending on what was in the prompt, and the direction of that shift is the subject of the rest of this chapter.

Getting the value out of an API is straightforward where the API offers it.

import math
from openai import OpenAI

client = OpenAI()
r = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[{"role": "system", "content": SYSTEM},
              {"role": "user", "content": prompt_with_passages}],
    logprobs=True,
    top_logprobs=5,
)

tokens = r.choices[0].logprobs.content
seq_sum = sum(t.logprob for t in tokens)
mean_lp = seq_sum / len(tokens)
print(len(tokens), round(seq_sum, 2), round(mean_lp, 3))
# 47  -14.57  -0.310

The scaffolding problem, and how to not measure it#

The moment answers are returned as structured output, the mean over all tokens stops measuring the answer at all. Every character of the JSON envelope is fully determined by the schema, so those positions carry logprobs within rounding distance of zero, and they outnumber the content tokens.

Over the whole 41-token response, 27 tokens are scaffolding and the mean logprob comes out at −0.061, which would rank this answer among the most confident in the corpus. Restricted to the 14 content tokens the mean is −0.478. The two numbers describe different things, and only the second is about what the model said.

The split is exhaustive, which is what makes the problem tractable: twenty-seven scaffolding positions plus fourteen content positions is the whole forty-one-token response, so every position belongs to exactly one group and the mean over all of them is a weighted blend of the two group means with no third term. That framing is worth holding, because it says where the danger is. The scaffolding group's values sit within rounding distance of zero by construction — a schema-determined token has essentially no alternatives — so the blend is pulled toward zero in proportion to the scaffolding share, and the scaffolding share is set by the schema rather than by anything the model said.

That is the mechanism behind a bug that is otherwise baffling. Add two fields to the response object, and the scaffolding-to-content ratio moves; the blend moves with it; and every threshold tuned against the old ratio now silently cuts in a different place. That produces the particularly unpleasant case where an abstention rate changes after a release that touched no prompt and no model, and nothing in the diff explains it. The temptation to skip the masking is strong, because the unmasked number arrives free with a response you were already making, it looks like a probability, and it behaves plausibly on a handful of hand-checked examples: garbled answers do score low. What it will not do is behave consistently across a schema change. NeuralNine's How to Measure LLM Confidence: Logprobs & Structured Output walks through the same trap with a different schema if you want a second worked instance of it.

import json

def content_span_logprob(tokens, field="answer"):
    """Mean logprob over the tokens inside one JSON string field."""
    text, spans, cur = "", [], 0
    for t in tokens:
        spans.append((cur, cur + len(t.token), t.logprob))
        text += t.token
        cur += len(t.token)
    value = json.loads(text)[field]
    start = text.index(value)
    end = start + len(value)
    inside = [lp for (a, b, lp) in spans if a >= start and b <= end]
    return sum(inside) / max(len(inside), 1)

Even with the masking done properly, minimum token logprob usually separates better than the mean on extractive answers, meaning answers whose content is lifted out of the passages rather than composed, because a single fabricated entity buried in an otherwise-copied sentence moves the minimum and barely moves the average. That follows from what each statistic is: an average over fourteen positions dilutes one bad position by a factor of fourteen, while a minimum reports it undiluted. On a runbook corpus the mean over content tokens gives AUROC 0.61 and the minimum gives 0.64, which is a real but small improvement over a signal that was not good to begin with.

Why it fails on exactly the case RAG has#

The failure that matters is not the fluency artefact, which you could partly engineer around. It is that the logprob is computed inside the same forward pass that produced the answer, conditioned on the same context, including the retrieved passages the model has already decided are sufficient. When retrieval returns nothing relevant and the model falls back on what it absorbed in pretraining, that fallback is not a hesitant process. A confidently held parametric belief — parametric because it lives in the model's weights rather than in the retrieved text — is a peaked distribution, so the tokens expressing it are high-probability, and the mean logprob comes out excellent on precisely the answers with no support behind them.

Splitting 2,000 graded runbook answers by whether the gold passage was actually in the retrieved set makes the effect measurable.

The two subsets add to two thousand exactly, so this is the same graded set chapter 1 used, partitioned rather than resampled. There is a second arithmetic relationship in it worth noticing now and returning to in chapter 4: weighting the two AUROC figures by their subset sizes — fourteen hundred and eighty-six answers at 0.66 and five hundred and fourteen at 0.49 — reproduces the 0.61 that chapter 1's comparison table reports for this signal over the whole set. Two tables built from different splits landing on one number is a check that they describe the same population, and it says something about what the headline figure is: a fact about a retrieval hit rate of roughly three in four, not a property of the signal alone. AUROC does not decompose as a size-weighted average in general, so this is an identity that holds on these numbers rather than a theorem to rely on elsewhere.

On the half of the problem where the passages were there, the signal is weakly informative: wrong answers in that regime are usually the model misreading, hedging or stitching two passages together, and that shows up as hesitation. On the half where retrieval failed, separation collapses to chance, and the wrong answers score better than the average correct answer elsewhere in the corpus. This is the same failure the verbalized score has, arriving through a different route, which is why the two are correlated rather than complementary: both are functions of the model's internal state, and the retrieval gap is a state the model has no way to notice from inside.

That last phrase sounds like a claim about model capability and it is a claim about what the quantity is defined over, so it is worth walking the steps. The logprob at each position is a conditional probability given the context, and the retrieved passages are part of that context. Now ask what a passage that fails to mention a retry budget does to the probability of a sentence stating one. It does not lower it, because lowering it would require the passage to say something incompatible, and saying nothing is not saying something incompatible. It simply contributes nothing to the prediction. So the distribution at that position falls back to whatever the model would have produced with no passages at all — which is the parametric belief, peaked and high-probability. Absence of evidence has no representation in the conditional.

The consequence follows immediately. If you want a signal that responds to a passage failing to support a claim, the signal has to compare the claim against the passage as two separate objects and report on the comparison. No aggregate over the generation's own token probabilities can do that, however it is computed, because the generation's token probabilities are one distribution and the comparison needs two arguments.

Closed-set scoring, richer statistics, and what providers expose#

The argument above is against sequence logprob as a general correctness signal for free-form RAG answers. It is not an argument that token probabilities are uninformative, and there is a family of uses where they are the best thing available and reasonably calibrated.

Read that table as gaps from the first row rather than as four independent verdicts. The top row names a property — a constrained set of mutually exclusive continuations occupying one syntactic slot — and every row below is a measure of how much of that property survives. Extractive span selection keeps the mutual exclusivity and loses the single position, and it stays useful. Truncation detection keeps neither but asks a different question, about the shape of the distribution rather than about which continuation is right, and that question the statistic can answer. The last row keeps none of it: a free-form answer has unboundedly many continuations, they are not mutually exclusive, and they differ in fluency as much as in content.

The classification case is worth dwelling on because it is the strongest counterexample. When the model must emit one of four label tokens after a fixed prefix, the probability mass across those four positions is a distribution over a genuine set of alternatives, and there is no fluency confound because every alternative is a single token in the same syntactic slot. Routing thresholds built on that number behave well and recalibrate cleanly with the machinery from chapter 2.

The usual next move, once the mean disappoints, is to compute something richer from the same data: per-position entropy over the returned top-k alternatives, entropy being a measure of how spread the distribution over the next token is, the margin between the top token and the runner-up, or the fraction of positions whose entropy exceeds some cut. These are better instruments than the mean in the sense that they distinguish a position with one obvious continuation from a position with five plausible ones, and on a runbook corpus a mean-entropy score over content tokens reaches AUROC 0.63. They do not change the argument, and the reason is the closure argument from the previous section rather than a limitation of any particular statistic: every one of them is a function of the same conditional distribution, and no rearrangement of numbers computed from one object becomes a report about a second one. A model that is certain and wrong produces low entropy at every position, low margin nowhere, and a clean mean.

There is also a practical constraint that decides the question before any of the above matters. Token logprobs are a provider feature, and the providers do not agree. OpenAI's chat completions expose logprobs with up to 20 alternatives per position; Google's Gemini API exposes response logprobs with its own limits and model coverage; Anthropic's Messages API does not return token logprobs at all. Self-hosted inference through vLLM or llama.cpp gives you everything, including the prompt tokens. So a design that depends on this signal is a design that constrains which model you can run, and teams that adopted it in 2024 have generally discovered that the constraint bites the first time they want to evaluate a model from a different vendor. That list is a claim with a date on it rather than a standing fact — whatever a provider supports today can change — so the check belongs in your evaluation harness rather than in your memory, where it will be re-run alongside everything else that moves.

The retrieval-gap column is also what settles whether this signal can carry an abstention decision in a domain where wrong answers are dangerous. A clinician asking about an internal protocol document is in that regime whenever the protocol has been revised, whenever the phrasing of the question does not match the document's vocabulary, and whenever the collection simply does not cover the case. Those are the questions where a wrong answer does damage, and they are the subset where this signal measures at 0.49. A signal whose separation is concentrated on the easy half of the problem is worse than a weak signal that is uniform, because it will look acceptable in aggregate evaluation and be absent exactly where it is needed — and the aggregate is a mix, so it will keep looking acceptable as long as retrieval keeps working most of the time.

Two signals have now been measured against the same graded set, and both fail in the same place for the same underlying reason: each is computed from the model's own state, so neither can observe the difference between an answer read out of the passages and an answer recalled from pretraining. A third signal is available to every deployment regardless of vendor, needs nothing but the ordinary completions endpoint, and is the standard recommendation the moment somebody says the stated confidence is useless — sample the answer several times and see whether the samples agree.

It costs four extra generations per question and it improves on both of the numbers so far. It is also, on the retrieval-gap subset, a third function of the same conditional distribution, and the next chapter is about what that costs.