Positive specific agreement

Chapter 2, on chance-corrected agreement, left a gap that a coefficient cannot close. Raw agreement and kappa are both aggregates over a table in which one cell holds nearly all the mass, so between them they establish that agreement on a rare label is thin without saying what to ask for instead. Positive specific agreement is the quantity that fills the gap: the fraction of positive labels that are shared between two annotators, computed on the positive class alone and ignoring the enormous agreement on the negative class entirely.

Take 500 items labelled "contains an unsupported claim" by two annotators, arriving at a review meeting with 93.0% raw agreement and Cohen's kappa a shade over 0.2 attached to it, and meant to become the reference against which an automated judge will be validated.

The thirty-five disagreements have a shape#

Recomputing from this table gives 465 agreements, so 93.0%, and kappa 0.22. What the table adds is the count that the aggregates hid: 41 items were flagged by at least one annotator, and on 6 of them the two annotators picked the same item. Thirty-five times, exactly one of them saw an unsupported claim.

That ratio has a name and a formula. Positive specific agreement is computed as twice the joint positives over twice the joint positives plus the two disagreement cells. It is the same quantity as the F1 score — the standard single-number summary of a detector, balancing how many of its flags are correct against how many of the real cases it caught — that you would get by treating one annotator as the prediction and the other as the reference, and it is symmetric, so it does not matter which one you nominate.

Both of those claims are checkable on this table and worth checking, because the formula looks arbitrary until it has been derived once. Nominate annotator A as the detector and annotator B as the truth. Then A's correct flags are the 6 they both made, A's false alarms are the 21 A raised alone, and the cases A missed are the 14 B raised alone. Precision — the share of A's flags that were right — is 6 out of the 27 A raised, a bit over a fifth. Recall — the share of B's flags A found — is 6 out of B's 20, three tenths. F1 combines those two by a rule that comes out, after cancelling, as twice the correct flags over twice the correct flags plus the false alarms plus the misses, which here is twelve over twelve plus twenty-one plus fourteen. Twelve over forty-seven is 0.255.

The symmetry is visible in that last expression rather than something you have to take on trust. Swapping which annotator is called the detector swaps the false alarms with the misses — A's 21 become misses and B's 14 become false alarms — and the formula adds those two numbers together, so the sum is unchanged and so is the answer. That property is exactly what you want here, because neither annotator is the truth and any statistic that gave a different answer depending on which one you wrote down first would be reporting an artefact of your data layout.

def specific_agreement(both, a_only, b_only, neither):
    n = both + a_only + b_only + neither
    return {
        "union_positive_rate": (both + a_only + b_only) / n,
        "joint_positive_rate": both / n,
        "positive_specific":   2 * both    / (2 * both    + a_only + b_only),
        "negative_specific":   2 * neither / (2 * neither + a_only + b_only),
    }

for k, v in specific_agreement(both=6, a_only=21, b_only=14, neither=459).items():
    print(f"{k:22s} {v:.3f}")

union_positive_rate    0.082
joint_positive_rate    0.012
positive_specific      0.255
negative_specific      0.963

Agreement on the negative class is 0.963; agreement on the positive class is 0.255. Those two numbers are the 93% pulled apart, and they point in opposite directions. On the class the label exists to identify, roughly one in four positive labels is shared. If you framed this as an information-retrieval problem — annotator A retrieving unsupported claims, annotator B being the reference — you would be looking at an F1 of 0.26 and would not consider the retriever working.

The negative figure is computed the same way with the roles of the classes swapped — twice the 459 items both called supported, over that plus the same two disagreement cells — and it comes out at 0.963 for a reason worth naming. The two figures share a denominator term, the 35 contested items, and differ enormously in what they are being compared against: 35 disputed items set beside 6 joint positives is overwhelming, and the same 35 set beside 459 joint negatives is a rounding error. One table, one set of disagreements, two readings that differ by a factor of nearly four, and the raw agreement figure is a weighted blend of them in which the negative class supplies almost all the weight.

Positive specific agreement is the number to demand beside a raw-agreement-and-kappa pair on any skewed label, and it has one property that makes it more useful in a meeting than kappa: it is directly interpretable as a fraction of the flags, so nobody has to argue about what 0.22 means on a scale from -1 to 1. Its complement is equally readable. Thirty-five of the 41 flagged items are contested — thirty-five over forty-one is a little under six in seven — so if you adjudicated this set you would be re-examining 85% of everything either annotator thought was a violation.

How many positives a sample has to contain#

The second figure the decision needs is how much confidence the 0.255 itself carries, and with 6 joint positives the answer is very little. A bootstrap over the 500 items — resampling items with replacement and recomputing positive specific agreement each time — gives a 95% interval of roughly 0.13 to 0.41.

import random

def bootstrap_psa(labels, iters=5000):          # labels: list of (a, b) 0/1 pairs
    out = []
    for _ in range(iters):
        s = [random.choice(labels) for _ in labels]
        both   = sum(1 for a, b in s if a and b)
        only   = sum(1 for a, b in s if a != b)
        out.append(2 * both / (2 * both + only) if both or only else 1.0)
    out.sort()
    return out[int(0.025 * iters)], out[int(0.975 * iters)]

An interval that wide cannot distinguish "the annotators share a quarter of their flags" from "they share nearly half", and any threshold you set at 0.4 is inside the noise. The width is set by the number of positives, not by the number of items, which is the practical rule worth carrying: the effective sample size of a rare-label agreement study is the union-positive count, and 500 items at 8% union prevalence — prevalence being the share of items carrying the positive label — is a study with 41 usable observations.

Why the item count drops out is visible in the formula. Every quantity in positive specific agreement is one of the three positive cells; the 459 items both annotators called supported appear nowhere in it. Labelling another thousand items that nobody flags therefore changes the numerator by nothing and the denominator by nothing, and an estimate that does not move when you add data is an estimate that was never using that data. What labelling more items does is add union positives at whatever rate the traffic supplies them, and it is those that narrow the interval.

Getting the interval down to roughly ±0.08 needs somewhere near 100 union positives, and ±0.05 needs about 250. Those two figures come from one rule that is worth having rather than looking up: the width of an interval on a proportion shrinks with the square root of the number of observations behind it, so four times the observations halves the width and a hundred times narrows it tenfold. Check it against the interval already in hand. Forty-one union positives gave a spread of roughly 0.13 to 0.41, which is about fourteen points either side of the estimate. Going from forty-one observations to a hundred is a factor of about two and a half, whose square root is a little over one and a half, and fourteen points divided by that lands at about eight — the first figure. Going to two hundred and fifty is a factor of about six, whose square root is about two and a half, and fourteen divided by that is a little under six, which rounds to the second. The rule also delivers the bad news, which is that an interval seven times tighter than the one in hand would need roughly fifty times the positives this study had.

At this prevalence, 100 union positives means labelling around 1,200 items uniformly, since positives arrive at eight in every hundred items and a hundred of them therefore takes about twelve hundred items to accumulate, which at two annotators and 90 seconds an item is 60 person-hours. That last step is worth doing too, since it is the one that turns a statistical wish into a budget: twelve hundred items labelled twice is two thousand four hundred labels, each taking a minute and a half, which is three thousand six hundred minutes, or sixty hours. The alternative is to stop sampling uniformly. Enrich the agreement study with strata — slices of the traffic defined in advance and sampled at their own rates rather than in proportion to how often they occur — where positives are dense: outputs an automated check already flagged, outputs from the query types where unsupported claims have historically appeared, outputs users complained about. Measure agreement inside each stratum, and keep the uniform slice only as the estimate of prevalence. That buys the same interval for a fifth of the hours, at the cost of a number that describes agreement on hard cases rather than on traffic. Both are worth knowing and they are not interchangeable, so the report has to say which one it is quoting.

Which coefficient answers which question#

Three or more annotators, ordinal scales — scales whose values have an order, so that a 5 scored against a 4 is a smaller disagreement than a 5 scored against a 2 — and items that some annotators left unlabelled all break Cohen's kappa, which is defined for exactly two raters over a complete matrix. Krippendorff's alpha is the general form and handles all three, which is why it is the default in a mature labelling programme rather than a more exotic choice.

Each of those three is a structural break rather than an inconvenience, and naming which one you have is how you pick a coefficient without guessing. With three raters there is no single pair of marginals to build a chance model from, so the per-rater construction Cohen's kappa is defined by has nothing to operate on. With an ordinal scale the statistic has to know that some disagreements are worse than others, and kappa's arithmetic only ever asks whether two labels matched, which throws that ordering away. With a ragged matrix — some items labelled by two people, some by four, some by one — the table kappa expects, one row per item with exactly two entries, does not exist. Alpha is built to absorb all three: it works from a general notion of how far apart two labels are, pools across whatever raters happen to have seen an item, and reduces to something very close to kappa when the data happens to be two raters on a complete binary matrix.

Alpha will not rescue the table above. Run on the same 500 binary labels it lands within a couple of hundredths of the kappa, because it is correcting for the same skew. What it gives you is coverage of the cases kappa cannot express: a five-point helpfulness scale where being one apart should count as a smaller disagreement than being three apart, a pool of six reviewers who each did a partial overlapping slice, or a set where the third reviewer was only brought in on contested items.

Gwet's AC1 appears in the table because it will be proposed by someone who has found that it reports 0.90 on data where kappa reports 0.15, and that difference is not a fix. AC1 uses a chance model that assumes agreement on a rare category is largely non-random, so on skewed data it reports a much higher number by assumption. Trace where the difference enters and the point becomes concrete rather than a matter of taste. Every one of these coefficients has the same shape — observed agreement minus a floor, over one minus that floor — and they differ only in how the floor is computed. Kappa's floor on the table above sits at about ninety-one in a hundred, since one annotator flagged a twentieth of the items and the other a twenty-fifth and both therefore said "supported" almost always. That leaves a ceiling roughly nine points wide, and the pair's 93% is only two of those points above the floor, which is the whole of the 0.22. AC1 builds a floor that falls as the class gets rarer, on the reasoning that when almost nothing is a violation, two people both saying "clean" is mostly them both being right rather than both guessing. Lower the floor and the ceiling opens up, and the same observed agreement divided by a much larger available headroom returns a much larger number. Nothing about the annotators changed between the two figures. What changed is the assumption about what agreement is free. It is defensible in settings where the skew is a known property of the domain rather than a sign that the label is under-specified. Switching to it because the first number was uncomfortable is choosing the baseline that produces the answer you wanted, and the decision it feeds — whether these labels can serve as ground truth — is exactly the decision that baseline shifts.

The value that flips the decision#

With positive specific agreement in hand, the shipping decision becomes statable rather than arguable. For a reference set that a judge will be validated against, the thresholds worth committing to in advance are these.

The threshold is not a universal constant, and it should be set by what the labels feed. Labels used to rank two prompt variants by a difference of ten points tolerate more noise than labels used to certify that a safety filter catches 99% of violations. Write the number down before running the study, because a threshold chosen after seeing the result is not a threshold.

Three worlds the coefficients cannot tell apart#

Positive specific agreement is not chance-corrected, which means it cannot be compared across datasets with different prevalences without care, and it will read higher on a set where both annotators flag liberally. Reporting it alone repeats the mistake of chapter 2, on chance-corrected agreement, in the opposite direction. The pair to quote is prevalence, kappa or alpha, and positive specific agreement, because each of the three constrains the reading of the others.

More importantly, none of these numbers identifies who is right. The table shows 21 items A flagged alone and 14 B flagged alone, and it is entirely consistent with A being a careful reader catching real unsupported claims that B skims past, with B applying a stricter definition of "claim", or with both of them guessing on genuinely borderline text. Those three worlds call for three different responses — retrain B, rewrite the definition, or accept that the category has a fuzzy boundary and change the label schema — and no coefficient distinguishes them.

The state of the art on validating a human-versus-model labelling relationship has moved toward exactly this recognition. The Alt-Test procedure described in Validating Annotation Agreement between Humans and LLMs asks whether a model annotator agrees with each human at least as well as the other humans agree with that human, framed as a hypothesis test rather than a threshold on a coefficient. It is a better instrument than a single number, and it still tells you nothing about the content of the disagreements.

On a rare label the aggregates are dominated by a cell nobody disputes, so the figure to demand is the one computed on the positive class alone, quoted beside the prevalence that explains why the aggregates read as they do and beside the count of joint positives that says how far to trust it. Two thresholds set in advance turn that figure into a decision that survives a meeting: below 0.4 the annotators have not demonstrated that they are detecting the same thing, and above 0.65 they have.

The content of the disagreements is the next thing to look at. Thirty-five contested items is a small enough batch to read in an afternoon, and the pattern inside them — whether they cluster on a particular kind of claim, a particular length of passage, or a particular week of one annotator's work — determines which of the three worlds you are in. Chapter 4, on diagnosing a disagreement set, works through that diagnosis before anybody pays for a relabel.