A reliability curve tells you what a 0.8 cutoff actually buys

A reliability curve is the elementary construction behind every calibration claim: sort scored items into bins by their score, and within each bin measure the fraction that turned out correct. A perfectly calibrated signal puts every bin on the diagonal, so items scoring 0.7 are correct seventy percent of the time. An operating point is a specific cutoff together with what it does — how much traffic it refuses and how accurate the remainder is. The distinction between the curve and any single point on it is what this chapter is about, because the statistic teams usually reach for lives at neither.

Take a claim of the form: stated confidence correlates with correctness at r = 0.31 with several hundred graded answers behind it, the association is real, so threshold at 0.8 and ship. Nothing in that sentence is about 0.8. A correlation coefficient is one number summarising the entire joint distribution — the one usually quoted here is the point-biserial coefficient, which is the ordinary Pearson correlation computed with one of the two variables being the 0/1 grade — and it is compatible with an enormous range of behaviours at any particular point in the range.

Two populations, one correlation#

The quickest way to see that is to construct two populations that agree on the coefficient and disagree on everything a product team would ask.

Both are plausible shapes for a model's stated confidence. A gets its correlation from a small, clean tail of low-confidence answers that really are wrong; B gets the same coefficient from a broad, gentle slope with wrong answers scattered across the whole range. Shipping the same rule against the two produces a product that refuses one question in five and one that refuses nearly half.

The reason the coefficient cannot separate them is worth stating as a property rather than as a surprise. A correlation is a functional of the whole joint distribution: it integrates over every score, weighting each region by how much data sits there. A cutoff is a functional of the distribution restricted to one side of a point: everything above the cut, and nothing else, determines what gets served. Two functionals that read different regions of the same object can agree on one and disagree on the other, and there is no arithmetic that recovers the second from the first, because the first has already summed away the information the second needs. To know which population you have, look inside the range instead of summarising across it.

That is also the stumble this chapter is built around, and it is tempting for a specific reason: r comes with a p-value, and the p-value answers a real question — whether the association could plausibly be noise. Having answered a hard-sounding question convincingly, it is natural to treat the harder question as settled too. The constant quantity is the coefficient; the dangerous property that varies underneath it is where in the range the wrong answers sit.

Bucketing, and what falls out of the buckets#

import numpy as np

def reliability(scores, correct, edges=np.linspace(0.0, 1.0, 11)):
    idx = np.digitize(scores, edges[1:-1], right=True)
    rows = []
    for b in range(len(edges) - 1):
        m = idx == b
        n = int(m.sum())
        if n == 0:
            continue
        rows.append({
            "bin": f"[{edges[b]:.1f}, {edges[b+1]:.1f})",
            "n": n,
            "mean_conf": float(scores[m].mean()),
            "accuracy": float(correct[m].mean()),
            "gap": float(scores[m].mean() - correct[m].mean()),
        })
    return rows

Run over 900 graded answers from a policy-document assistant, the populated bins look like this. Nothing below 0.4 appears, because the model almost never states a low number even when a follow-up call is explicitly permitted to.

Two things in that table can be checked without trusting anybody. The counts add to nine hundred exactly — twenty-two, forty-eight, ninety-six, one hundred and seventy-one, three hundred and thirty-six and two hundred and twenty-seven — so every graded answer landed in a bin and none was dropped or double-counted. And the gap column is a subtraction and nothing else: forty-five hundredths minus forty-one hundredths is four hundredths, fifty-five minus fifty-two is three, sixty-five minus fifty-eight is seven, and so on down every row. That column adds no measurement, which means anybody can rebuild it, and its being positive in every row is the single fact the rest of this section leans on.

The curve rises, which is the thing r was detecting, and it sits above the diagonal everywhere, which is the overconfidence chapter 1 described. What it now supports is the operating-point question, because the cutoff decision is a cumulative read of this same table: everything at or above the cut is served, everything below is refused.

Work the 0.8 row out of the bins, because seeing one row derived is what makes the rest readable. Serving everything at or above 0.8 means serving the top two bins, which is three hundred and thirty-six plus two hundred and twenty-seven, or five hundred and sixty-three of the nine hundred — sixty-two and a half percent, the figure in the first column. The correct answers among those are seventy-two percent of three hundred and thirty-six plus eighty-one percent of two hundred and twenty-seven, which comes to a little over four hundred and twenty-five; dividing by five hundred and sixty-three gives 0.756, the accuracy of what is served. The wrong ones are the remainder of that same set, about one hundred and thirty-seven answers, which scaled to a thousand questions is the 153 in the last column. And the correct answers refused are the correct ones in the four bins below the cut, a little under two hundred, which scales to the 220. Four cells, one bin table, no extra information. The 0.9 row's first column reproduces the same way: two hundred and twenty-seven of nine hundred is 25.2%.

Two of the cutoff rows do not reproduce as cleanly, and it is worth saying so rather than reading a closure into them. Serving everything at or above 0.7 means serving the top three bins, which is seven hundred and thirty-four of nine hundred — a shade over eighty-one and a half percent, not the 82.6% printed. The correct-answers-refused cell in the same row is further from what the bins give than its neighbours are, and the 0.9 row's version of that cell is a few units off as well. The two tables are close enough throughout to be describing the same nine hundred answers, and they are not reconcilable to the digit the way the 0.8 row is. Nothing below depends on the rows that do not close, so the safe reading is to derive from the bins and treat the cumulative table as a rounded presentation of them.

At 0.8 this system refuses 37% of its traffic and improves the accuracy of what it still answers from 0.687 to 0.756. Whether that is a good trade is not a question this table settles, and the fourth column is the one a ship date has not looked at: 220 questions in every 1,000 get a refusal despite the model having produced a correct answer. That column is what chapter 6 is built around.

The shape of that cumulative table is worth reading as a curve in its own right, because the interesting information is in the differences between adjacent rows rather than in any single one. Moving the cutoff from none to 0.7 costs 76 correct answers per 1,000 and removes 88 wrong ones — seventy-six minus zero, and three hundred and thirteen minus two hundred and twenty-five — which is roughly one good answer sacrificed per bad answer suppressed. Moving from 0.8 to 0.9 costs 264 correct answers and removes 104 wrong ones, since four hundred and eighty-four minus two hundred and twenty is two hundred and sixty-four and one hundred and fifty-three minus forty-nine is one hundred and four, a ratio of about 2.5 to 1.

That the exchange rate is a difference of two columns already printed is what makes it trustworthy: it is not a fifth measurement, it is the slope of the table, and anybody who accepts the rows accepts it. It degrades as the cut rises, and it degrades for a reason visible in the bin table — the top of the range is where most of the correct answers live, so the higher you cut, the more of the population you are cutting into and the larger the share of it that was right. Nobody needs a cost model to see that the last row is a different kind of decision from the second, and this is the table on which such a decision is made rather than argued.

Compressing the curve into one number, carefully#

Six rows are hard to track across model versions, so the curve gets summarised. Expected calibration error is the size of the gap column, weighted by how much traffic each bin carries.

def ece(scores, correct, n_bins=10):
    rows = reliability(scores, correct, np.linspace(0, 1, n_bins + 1))
    total = sum(r["n"] for r in rows)
    return sum(r["n"] / total * abs(r["gap"]) for r in rows)

# policy assistant, 900 graded answers, 10 equal-width bins
ece(scores, correct)          # 0.117

The 0.117 comes straight out of the table above and is worth reproducing once. Multiply each gap by the count in its bin: four hundredths on twenty-two answers, three on forty-eight, seven on ninety-six, eleven on one hundred and seventy-one, fourteen on three hundred and thirty-six, thirteen on two hundred and twenty-seven. Those products total a little over a hundred and four, and dividing by nine hundred gives a shade under twelve hundredths — the reported figure, to the third decimal place.

An ECE of 0.117 says that on average a stated confidence is about twelve points away from the empirical frequency it should match. It is a useful regression alarm — if a model upgrade takes it from 0.12 to 0.28, something changed — and it has three weaknesses worth holding onto.

Equal-width bins put 62% of this traffic in the top two bins, which follows from the same counts as before, so the estimate is dominated by two averages and the bins below 0.6 barely count; equal-mass binning, which puts the same number of answers in each bin, gives 0.104 on the identical data. The absolute value discards direction, so a model overconfident on half its range and underconfident on the other half can report a small ECE while being wrong in both directions.

And the statistic is blind to the property chapter 1 established as decisive. That is usually stated as an observation — shuffle the scores among the answers so that separation collapses to chance while the marginal distribution is untouched, and ECE moves hardly at all — but on this data it is a consequence rather than a coincidence, and the derivation takes one line. Every gap in the table has the same sign. When that holds, the absolute values in the ECE sum can be dropped, and a weighted mean of differences is the difference of the weighted means: ECE becomes the traffic-weighted mean confidence minus the traffic-weighted accuracy. Both of those are single numbers describing the whole population. So on a uniformly overconfident signal, ECE degenerates into exactly the two-means comparison chapter 1 warned against, and a comparison of two means cannot see ordering, because reshuffling which answer carries which score leaves both means untouched.

Working the two means out is also the check that the tables belong together. The traffic-weighted mean confidence across the six bins comes to 0.81 — and that is precisely the mean stated confidence reported below for this elicitation, arrived at from bin counts rather than from the raw column. The traffic-weighted accuracy comes out a fraction of a point above the 0.687 the cumulative table starts from, which is the same rounding-level residual as the cutoff rows that did not close, and the difference between the two weighted means lands on the ECE just derived.

The alternatives trade off differently, and it is worth knowing which question each one answers before quoting any of them in a ship review.

Read the second column as a partition rather than as a ranking. Each row answers a question the others do not, and the last two rows split the two properties of chapter 1 between them: Brier fuses level and separation, AUROC isolates separation, ECE isolates level. Brier score is the mean squared difference between the stated confidence and the 0/1 grade, and it decomposes into a calibration term and a resolution term, which is why it moves when either property changes and why a Brier improvement alone never tells you which one improved. Reporting AUROC alongside ECE keeps the two properties visible separately, and that pair is a reasonable default for a model-version dashboard.

The number moves when nothing about the answer does#

A reliability curve is a property of a specific elicitation — the exact way the question of confidence is put to the model, including where in the response the number is asked for and what units it is asked in — and the elicitation is more fragile than it feels. On the same 900 questions with the same retrieved passages and the same underlying model, four variants of how confidence was requested produced these figures.

The second row is the one every table above describes, and the two figures it shares with them — a mean of 0.81 and an ECE of 0.117 — both fell out of the bin counts a moment ago. Two tables built from different columns of the same data landing on the same pair of numbers is what licenses reading this table as a controlled comparison: four elicitations over one question set, rather than four separate evaluations whose differences might be differences in the questions.

Read the rows as gaps and the pattern is that the level column swings by nearly twenty points across the four while the separation column moves by nine hundredths. The verbal-percentage phrasing halves the calibration error without meaningfully changing separation, which is the pattern Just Ask for Calibration reports across several models and tasks: elicitation strategy is a lever on level and a much weaker lever on ordering. That is the asymmetry of chapter 1 arriving from the opposite direction — recalibration is a transformation you apply afterwards and it cannot touch ordering, and elicitation is a change you make beforehand and it barely touches ordering either.

The JSON variant is the worst of the four, and the mechanism is visible if you think about where the field sits. The confidence token is generated inside a structure the model is completing, before or alongside the answer content rather than after reading it, so whatever the number is a report on, it is not a report on a finished answer. Structured-output constraints narrow the token distribution at exactly that position as well, which pushes the value further toward whatever is most probable in that slot.

Two other sensitivities show up in the same corpus. Answer length correlates with stated confidence at r = 0.22 independent of correctness, so questions that happen to elicit longer answers get systematically higher numbers. And regenerating the confidence call with the same answer at temperature 0 produced a different value on 8% of items after an unrelated system-prompt edit that added a sentence about output formatting. A curve measured under one prompt is therefore not evidence about the same system under another, which means the calibration check belongs in the pipeline that runs on prompt changes rather than in a notebook someone ran in March.

Flat curves, thin bins and a question mix that moves#

A reliability curve does not make an unusable signal usable. Chapter 1's column would produce a curve too, a nearly flat one hovering around 0.71 across every bin, and reading a cutoff off a flat curve gives you a rule that refuses traffic uniformly at random. The construction assumes the ordering is worth something; it is how you choose an operating point on a signal that has already earned the right to have one, and running it first on a signal that has not is how a team ends up debating where to cut a coin.

Per-bin accuracy is also an estimate, and the bins at the edges are usually the thinnest. The [0.4, 0.5) bin above has 22 answers, so its 0.41 accuracy carries a 95% interval of roughly ±0.21 — wide enough that the bin is consistent with anything from a coin to a fairly bad signal, which are two very different diagnoses. Wilson intervals on each bin — an interval for a proportion that stays inside 0 and 1 and does not fall apart when the count is small, unlike the textbook normal approximation — printed next to the accuracy column, stop a five-answer bin from being read as a data point.

The construction is borrowed rather than invented here, and knowing where from helps set expectations for how much it can be pushed. Operational weather forecasting has used exactly this construction since the 1950s to check that "70% chance of rain" days rain about 70% of the time, on populations of tens of thousands of forecasts accumulated over seasons. The parallel is close and it breaks in one place, which is the place that matters: a forecast is graded by the weather at no marginal cost to anybody, and an answer is graded by a person at roughly two to four minutes each. So the resource that makes fine-grained bins meaningful — sheer volume of labelled outcomes — is precisely the resource an LLM team does not get for free. Nine hundred graded answers gives six usable bins with wide intervals. Budgeting for calibration means budgeting for grading, and a team that wants a curve it can trust at the tails is committing to a few thousand graded items rather than a few hundred.

And every figure here comes from graded historical answers, which are a sample of the question distribution as it was. A curve measured before a corpus expansion describes a system that no longer exists, which is the recheck condition chapter 6 makes explicit.

What none of this machinery cares about is where the score came from. Bucketing, ECE, the cumulative cutoff table and the sensitivity checks all take a scalar per answer and a grade per answer, and the same code runs against a number the model wrote, a probability computed from token statistics, or a score returned by a separate model reading the retrieved passages. That indifference is the useful part, because the results in this chapter are an argument for finding a different scalar rather than for tuning this one.

The next candidate is already sitting in the API response for most of these systems, unused. It costs nothing to collect, it is a real probability rather than a number chosen for its plausibility, and it has the appealing property that the model did not choose to say it.