A model's confidence in itself measures fluency

The signal was three extra lines in the prompt and it had been in production for nine weeks. Escalation rate: 12.1%, with a weekly standard deviation of 0.4 points. Spend: down 38% against the always-strong baseline. Support tickets tagged bad-answer: flat. The quarterly review took forty seconds on the slide and the follow-up question was whether the threshold could come down to 0.6 and take more.

Here is what the signal is:

SYSTEM = """Answer the user's question using only the provided context.
Then rate your confidence that your answer is correct and complete.
Respond as JSON: {"answer": str, "confidence": float between 0 and 1}"""

resp = cheap.messages.create(
    model="small-8b",
    system=SYSTEM,
    messages=[{"role": "user", "content": f"{context}\n\n{question}"}],
    max_tokens=800,
)
out = json.loads(resp.content[0].text)

if out["confidence"] < 0.70:
    out = escalate_to_strong(context, question)

And here is one of its outputs, from a request where the retention policy in the retrieved context covered EU customers only and the question was about a customer in Brazil:

{
  "answer": "Your data is retained for 24 months after account closure, after
   which it is permanently deleted. You can request earlier deletion at any
   time through Settings → Privacy → Delete my data.",
  "confidence": 0.95
}

The answer is wrong, the policy it describes does not apply, and 0.95 is above 0.70, so no escalation happened, no strong model ever saw the request, and the trace records a successful 412-millisecond call.

The number is generated, not computed#

The confidence value comes out of the same forward pass, the same weights and the same sampling loop as the answer, conditioned on the answer the model has just written. There is no separate evaluation step in which the model compares its output against something it knows to be true, because it has nothing to compare against; the tokens 0, ., 9, 5 are predicted the way every other token is predicted, from what tends to follow text of that shape. What the model is estimating, as well as it can, is how a confident answer of this kind is usually annotated. That is a statement about the answer's form.

You can see the consequence in the distribution. Across 41,208 production requests the returned values looked like this:

Eighty-eight percent of the mass sits on six discrete values, and 72% on two of them. This is not a continuous score with a threshold in the middle of it; it is a nearly categorical variable with a handful of levels, most of which mean "fine". Moving the threshold from 0.70 to 0.60 does not tune anything gradually — it moves one bucket, 6% of traffic, across the line in a single step. Moving it to 0.86 would move 15% the other way. There are no intermediate settings, because the model does not produce intermediate values.

The stability of the escalation rate follows directly. That 12.1% ± 0.4 is a statistic of this distribution, and the distribution is a property of the model's decoding behaviour rather than of the workload's difficulty. A team tested this by injecting 500 deliberately unanswerable requests into the stream — questions whose answers were absent from the retrieved context entirely — and 91% of them came back with a confidence of 0.90 or higher. The escalation rate barely moved, which is exactly what you would expect from a number that tracks how a fluent answer is usually labelled. A signal that is stable against an injected 500-request block of guaranteed failures is stable for reasons that have nothing to do with being right.

The errors it cannot see are the ones you built it for#

Correctness and fluency come apart in a specific, predictable place. When the model is uncertain in a way it can express — the context is contradictory, the question is ambiguous, the answer requires arithmetic it is visibly struggling with — the hedging shows up in the prose and often in the number too, and that request escalates. That is the easy half, and self-reported confidence catches a fair amount of it.

The other half is the case where the model has a confident, wrong prior: a policy that changed, a fact that is true for one jurisdiction and not another, a plausible API method that does not exist, a citation with the right journal and the wrong year. In those cases nothing in the generation feels uncertain to the model, because the wrong answer is the high-probability continuation given everything it has learned. The output is fluent, complete, well-formed, and confidently labelled. The escalation signal is a measure of the property that failed to vary.

This is why the failure mode is invisible rather than merely unmeasured. A missed escalation on this signal is correlated with the answer being confident and plausible, which is also the property that stops a user from noticing. Someone reading the Brazilian retention answer has no reason to doubt it; they will doubt it in fourteen months.

The support-ticket evidence deserves the same scepticism. Suppose 4% of requests get a materially wrong answer, and suppose — generously — that one in twenty affected users files a ticket about it. At 40,000 requests a week that is 1,600 wrong answers and 80 tickets, which arrive in a queue of several thousand tagged as ordinary product questions, get answered by a human, and are closed. Nothing in that pipeline attributes a ticket to a model choice on a request that happened three days earlier. "No complaints" in this setting is compatible with a large absolute number of bad answers, and the ticket volume would have to roughly double before it cleared the noise in a weekly chart.

The prompt variations that do not rescue it#

The first instinct on seeing that distribution is that the prompt is at fault, and there are three standard repairs worth knowing the outcome of, because each one is tried in most teams and each one moves less than expected.

Asking for the confidence before the answer rather than after inverts the conditioning: the model commits to a number and then writes something consistent with it. Escalation rate on the same traffic moved from 12.1% to 14.8%, and the missed cases stayed missed, because the model still has no information about correctness at the point it produces the number — it now has less, since the answer does not exist yet.

Asking in a second, separate call — show the model the question and its own answer, ask whether the answer is right — is the most promising of the three, and it is a different design rather than a prompt change. It is a verifier that happens to use the same model, and it inherits the same blind spot as long as the same weights are asked about the same content: the model that generated a confident wrong answer will read it back and find it convincing, which showed up as a 71% agreement between the second call's verdict and the first call's confidence bucket. Using a different small model for the second call breaks that correlation and is the reason verifier cascades are usually built with a model from another family.

Replacing the number with a verbal category — "certain / probable / uncertain / guessing" — makes the distribution more legible and no more informative. Eighty-one percent of responses came back "certain", and the injected-unanswerable block came back "certain" 88% of the time. The output format was never the problem; the quantity being reported is unavailable to the reporter.

There is one variation that does help, and it helps because it changes what is being asked about. Requiring the model to quote the span of retrieved context that supports each claim, and then checking programmatically that the quoted span exists in the context, converts a self-assessment into a verifiable one. The model can still quote a span that does not support the claim, so it is not a correctness oracle, and it catches the entire class of answers built from parametric memory — whatever the weights absorbed during training — rather than from the documents in front of the model — which was the Brazilian retention case exactly.

What else could sit in that position#

The escalation signal is a function evaluated on the candidate answer, and there are several to choose from with genuinely different properties.

Two rows deserve emphasis. The deterministic check is the strongest signal available whenever the task admits one, because it is not an estimate at all: JSON that fails schema validation is definitively bad, SQL that errors against the schema is definitively bad, generated code that fails the unit tests is definitively bad. Cascades on structured tasks should exhaust these before reaching for anything probabilistic, and they cost microseconds. Their limitation is that they are one-sided — they catch malformed output and say nothing about output that is well-formed and untrue.

The last row is the one that quietly reintroduces chapter 1's problem. A difficulty classifier evaluated on the request is a routing decision that has been moved inside a system whose diagram says "cascade", and if it is the only signal, the cheap model's answer is still never examined. Difficulty-based dispatch of this kind — the design behind vLLM's semantic router, which decides whether a query warrants a reasoning model at all — is a reasonable thing to build, and it is a router, with the router's measurement obligations.

There is also a distinction hiding in the phrase "escalation signal" that is worth pulling out, because even a perfectly calibrated correctness detector would not be the right function. What you want to know at the decision point is not is this answer wrong but would the strong model do better on this request. Those diverge in both directions. A question whose answer is absent from the context is one the cheap model gets wrong and the strong model also gets wrong, so escalating it burns strong-model spend for nothing and returns the same failure 1.9 seconds later. A question the cheap model happens to get right by a coin flip is one where escalation would have been valuable insurance. Correctness is the tractable proxy because you can label it, and it is a proxy.

What has actually been established after nine weeks#

The evidence in the review slide supports one claim: routing 88% of traffic to a model that costs a twentieth as much reduces spend by 38%. That is arithmetic and it is not in doubt. Nothing in the slide bears on whether the 12% being escalated are the right 12%, because every number on it is derived from the model's own behaviour rather than from any external check on the answers.

Lowering the threshold to 0.6 takes 6% of traffic out of the strong model's hands, banking a further saving of a few points, in exchange for an increase in exactly the class of error that nothing in the current system can observe. The trade might be fine. The point is that it is being made without a denominator.

Getting a denominator means comparing the escalation decisions against outcomes labelled by something other than the model that made them, which turns one unmeasured signal into two measurable error rates that do not cost the same thing.