Ninety-five percent agreement and a kappa of 0.31 are the same data

Two reviewers labelled the same 1,000 chat transcripts for a binary policy_violation flag. The quality report has two lines in it. Raw percentage agreement is 95%, which the product manager has already forwarded to the stakeholder channel with the word "clean" attached. Cohen's kappa is 0.31, which the data scientist who computed it has flagged as a blocker. Nobody has made an arithmetic mistake. Both numbers describe the same 1,000 rows, and the gap between them is not a disagreement about method — it is the single most important fact about this dataset, stated twice in two different units.

The reason is the prevalence. Roughly 96% of transcripts are negative, because most support conversations do not contain a policy violation, and that skew means the two reviewers spend the overwhelming majority of their effort agreeing about the easy negatives. Raw agreement counts those. Chance-corrected agreement asks how much of the agreement you would have got anyway from two people who knew the base rate and were otherwise guessing, and subtracts it.

Where both numbers come from#

Here is the full contingency table. Every claim in this chapter is derived from these four cells.

Observed agreement is the diagonal, (13 + 936) / 1000 = 0.949, which rounds to the 95% in the report. Expected agreement multiplies the marginals: reviewer A called 3.8% of items positive and reviewer B called 3.9% positive, so if the two of them were independently sprinkling positives at those rates they would land on the same item together 0.038 × 0.039 = 0.15% of the time, and both say negative 0.962 × 0.961 = 92.4% of the time. That is 92.6% agreement from nothing but the marginals.

def cohens_kappa(a, b, c, d):
    """a = both positive, d = both negative, b and c = disagreements."""
    n = a + b + c + d
    po = (a + d) / n
    p_a_pos, p_b_pos = (a + b) / n, (a + c) / n
    pe = p_a_pos * p_b_pos + (1 - p_a_pos) * (1 - p_b_pos)
    return po, pe, (po - pe) / (1 - pe)

po, pe, k = cohens_kappa(a=13, b=25, c=26, d=936)
print(f"observed {po:.3f}  expected {pe:.3f}  kappa {k:.2f}")
# observed 0.949  expected 0.926  kappa 0.31

The denominator is what makes kappa unforgiving here. There are only 7.4 percentage points of headroom between chance agreement and perfect agreement, and the reviewers captured 2.3 of them, so they earned 31% of the agreement that was actually available to be earned. The other 92.6 points were free.

The cleanest way to see that the free points are genuinely free is to invent a third reviewer who has never read the guideline, never opens a transcript, and marks every single item no violation. Against reviewer A, that reviewer scores 962/1000 = 96.2% raw agreement, which is better than the two real reviewers managed, and their kappa is exactly zero, because a constant labeller has no variance for the correction to credit. Any metric on which a reviewer who does no work outscores two reviewers who did the work is not measuring the work.

What the two reviewers actually did on the class that matters#

The dataset exists to find policy violations. Restrict attention to the items where either reviewer said violation, which is 13 + 25 + 26 = 64 transcripts, and the two of them concurred on 13. Positive-class specific agreement — the share of positive calls that both reviewers make, 2a / (2a + b + c) — is 26 / 77 = 0.34. On the only class the project cares about, they are agreeing about a third of the time.

That reframing is usually more persuasive to a product manager than the kappa is, because kappa is a number on an unfamiliar scale with folklore thresholds attached to it, whereas "of the 64 transcripts one of us flagged, we both flagged 13" is a sentence about transcripts. The right response to the request to ship on the 95% is to move the conversation onto those 64 items and off the aggregate entirely: read the 51 disagreements, cluster them the way the previous chapter's audit clustered the ticket disagreements, and find out whether they concentrate on a definable situation. If 40 of the 51 turn out to be transcripts where a user quotes an abusive message they received rather than sending one, that is a guideline defect with a known fix and a known cost. If the 51 are scattered with no pattern, the problem is different and worse, because there is nothing to write into the document.

The same skew hides inside an aggregate#

Prevalence skew in a binary flag is visible if you look, since there are only two classes and one of them is obviously rare. The harder version is a multi-class aggregate, where the skew is real but nothing in the reported number points at it.

An annotation lead reports Cohen's kappa of 0.82 across a 12-class support-intent dataset of 6,000 items, and the project is being signed off on that. The number is correctly computed. What it does not carry is the distribution of the 6,000 items across the 12 classes, and on this dataset two classes hold 80% of the volume.

The aggregate of 0.82 is almost entirely the first two rows. order_status and returns are the intents where the customer says what they want in the subject line, reviewers converge on them without effort, and together they contribute about 0.74 of the 0.82 before any other class has been considered. Every remaining class could sit at 0.45 and the headline would still read 0.82.

The two rows worth stopping on are fraud_report at 0.29 and accessibility_request at 0.21. Those are 150 items between them, 2.5% of the corpus, and they are the two intents with an escalation path and a regulatory tail. They are also, predictably, the two intents the model keeps getting wrong in production, which is the conversation that started the labelling project in the first place. A dataset that is excellent on the intents anyone could have routed with three keyword rules, and close to chance on the two intents that motivated the work, is not a dataset that has met its purpose, and the aggregate reports it as a success.

The mechanism is identical to the binary case. A weighted average over classes weights each class by how much of the corpus it holds, so rare classes are arithmetically unable to move it. There is no threshold on the aggregate that fixes this, because raising the bar to 0.9 only demands more agreement on order_status.

Breaking it down, and what the breakdown costs#

The operational rule that falls out is that agreement is reported per class and per example type before anything is signed off, and the aggregate is reported alongside as context rather than as the decision variable. Per class is the easy half — you have the confusion matrix already, so a one-vs-rest kappa per class is a loop.

Per example type is the half that gets skipped, because the types have to be defined by hand. On the transcript dataset the useful cuts were conversation length under and over 20 turns, transcripts containing a quoted third-party message, transcripts from the two locales served by machine translation, and transcripts truncated by the export job. Agreement on transcripts under 20 turns was 0.58; on the translated locales it was 0.19. Neither cut is visible in any per-class number, because both cuts run across the classes rather than along them.

Breaking down does cost something real, which is precision. The accessibility_request row rests on 60 items, and a kappa computed on 60 items has a bootstrap 95% interval roughly 0.2 wide, so 0.21 is really "somewhere between about 0.11 and 0.31" — bad either way, but not a number to rank classes by. The practical response is to report the interval alongside the point estimate and to treat rare-class agreement as a trigger for reading items rather than as a measurement to track week over week. Fifteen minutes reading the accessibility_request disagreements will tell you more than any tightening of the confidence interval, and the disagreements are already sitting in the annotation store.

What goes into the sign-off document, then, is not a number but a small table and a list: agreement per class with counts, agreement on each defined slice, the count of items in the unresolved queue, and the two or three specific situations that account for the bulk of the disagreements. A reviewer of that document can form an opinion about whether the dataset supports the decision being made on it. A reviewer of "kappa 0.82" can only decide whether 0.82 sounds like a large number, which it does.

Whether to compute Cohen's kappa at all is a smaller question than it is usually made out to be. The choice depends mostly on how many reviewers you have and whether every item is labelled by every reviewer.

The folklore thresholds attached to kappa — 0.6 as substantial, 0.8 as almost perfect — come from a 1977 paper by Landis and Koch that offered them as arbitrary benchmarks in a discussion of biomedical rater data, and they have been quoted as standards ever since. They are not calibrated to anything about your task, and the prevalence sensitivity above is exactly why they cannot be. A kappa of 0.65 on a balanced three-class problem and a kappa of 0.65 on a 96/4 binary flag describe very different amounts of reviewer skill.

None of this touches the question of whether the labels are correct. Every statistic in this chapter compares reviewers to each other, and the entire apparatus — aggregate, per-class, per-cut, chance-corrected or not — is silent on whether the convention they are both applying is the convention the project needs. There is a failure mode in which every one of these numbers comes back healthy, stays healthy across four rounds of data collection, and the model does not improve at all.