Precision says nothing about what the detector let through

The procurement pack arrives with a benchmark page. The vendor's PII detection service reports 99.1% accuracy across twelve entity types, evaluated on 50,000 documents. Your platform team, being careful, does not take that on trust: they run the service over a labelled set of 400 of your own support transcripts and get 99.2% precision, which is better than the datasheet. Legal wants a signature saying that support transcripts can go to an external model provider, and both numbers appear to say yes.

Take the accuracy figure apart first, because it is the weaker of the two and it is the one that appears on every datasheet. Accuracy over a token-level classification task — the detector labelling each token, roughly each word or word-piece of the text, as personal data or not — is the fraction of tokens labelled correctly, and in a support transcript almost every token is not personal data. A transcript of 900 tokens containing a name, an email address, and an order reference has perhaps 6 tokens that should be flagged and 894 that should not. A detector that flags absolutely nothing, implemented as return [], labels 894 of 900 tokens correctly and scores 99.33% accuracy — above the vendor's headline. The metric is dominated by the negative class to the point where it cannot distinguish a working system from an empty function.

Precision is a real metric and it answers a real question, which is the wrong question here. Precision is the share of the spans the detector flagged that were genuinely personal data: of the things it pointed at, how many were right. Recall is the share of the personal data actually present that the detector found: of the things that were there, how many did it catch. A disclosure happens when something that was there was not found, so the rate a disclosure depends on is recall, and precision is silent about it by construction.

The first row is not a strawman. It is the exact behaviour of a detector whose thresholds were tuned to eliminate customer complaints about over-masking, which is a thing teams do under pressure and which improves precision monotonically as it destroys recall.

Where these two words came from, and why the asymmetry flipped#

Precision and recall were formalised in information retrieval, in the Cranfield experiments of the late 1950s and 1960s, where the problem was ranking documents for a librarian's query. In that setting precision was the metric that mattered most in practice, because a user scans the top of a result list and a false positive costs them a few seconds of reading. Recall was the harder number to even compute, since it required knowing every relevant document in the collection.

Both facts invert in PII detection. A false positive costs you a masked span — some lost context in a prompt, which chapter 4 will show can be expensive but is recoverable. A false negative costs you an irreversible disclosure to a third party. And recall is not merely hard to compute here; it is hard in a specific way that shapes the whole measurement design, because the denominator is "all the personal data in your traffic" and the only instrument you have for finding it is the thing you are trying to evaluate.

Producing a recall number on your own data#

The vendor's 99.1% was measured on the vendor's corpus, and the entity distribution, language mix, formatting conventions, and noise of real support transcripts are the entire difficulty. A detector scoring well on clean, well-formed entities can sit far lower on text full of typos, transliterated names, identifiers pasted out of screenshots with the artefacts optical character recognition leaves behind, and phone numbers written the way people actually write them. The only number that supports a sign-off is one measured on a reviewed, representative sample of your traffic.

The first obstacle is that random sampling cannot measure the entities you care about most. National identity numbers appear at roughly 4.1 occurrences per 1,000 documents, so a random sample of 400 transcripts contains between one and two of them. To bound recall at 95% confidence with a half-width of five percentage points around an estimate near 0.7, you need about 323 positive occurrences of that entity: the sample size for a proportion is z² · p(1−p) / w², which at z = 1.96, p = 0.7 and w = 0.05 works out as 3.84 × 0.21 / 0.0025 ≈ 323. At a prevalence of 4.1 per 1,000, collecting 323 occurrences means reading roughly 79,000 documents. Nobody is annotating 79,000 transcripts.

The way out is stratification. Build strata that concentrate the rare entities, sample within each, and weight the results back. A workable set of strata for a support corpus is: documents flagged by a deliberately loose net (the union of every regex you have, a second independent detector, and keyword triggers such as passport, NIE — the Spanish identity number issued to foreign residents — and sort code), documents flagged by the production detector only, documents flagged by nothing at all, and a stratum sampled purely at random regardless of flags. The last one is the important one and it is the one that gets cut for budget, because it is the only stratum that can find entities no detector recognises. If every document you review was surfaced by some detector, your recall estimate is conditioned on being findable, which is exactly the assumption under test.

Weighting the strata back together is where stratified estimates go wrong quietly. Each stratum has a known sampling fraction — you reviewed 200 of 4,100 loosely-flagged documents and 200 of 380,000 unflagged ones — and every occurrence found in a stratum stands for one over that fraction of occurrences in the population. Recall within a stratum can be read directly, which is what the report above does, and any population-level statement has to carry those weights or it will be dominated by the stratum you oversampled. The unflagged stratum is the expensive one because it has a tiny hit rate: 200 documents reviewed to find perhaps four occurrences nobody's detector saw. Those four are worth the two person-days, since they are the only direct evidence you will ever get about the entities that are invisible to your whole toolchain, and their absence is what lets a team believe a 99% figure.

Annotation is human work and it should be double-blind with adjudication. Two annotators over 400 transcripts at roughly six minutes each is about 80 person-hours before adjudication, and the agreement statistic goes in the report next to the recall figure, because a recall number derived from a corpus where Cohen's kappa is 0.55 is not a measurement of the detector. Kappa is the standard chance-corrected agreement score between two annotators: 0 means they agree no more often than two people guessing independently would, and 1 means they agree on everything, so 0.55 says roughly that half the agreement in the corpus is not attributable to the guideline. It is a measurement of the guideline, and chapter 2's tier definitions are what raise kappa into the 0.8 range where the recall number means something.

from math import sqrt

def wilson(hits: int, total: int, z: float = 1.96) -> tuple[float, float]:
    """95% CI for a proportion; behaves sanely at small n, unlike normal-approx."""
    p = hits / total
    denom = 1 + z * z / total
    centre = (p + z * z / (2 * total)) / denom
    half = z / denom * sqrt(p * (1 - p) / total + z * z / (4 * total * total))
    return centre - half, centre + half


def recall_report(labelled: list[Span], detected: set[SpanKey]) -> list[Row]:
    rows = []
    for entity, lang in sorted({(s.entity, s.lang) for s in labelled}):
        truth = [s for s in labelled if (s.entity, s.lang) == (entity, lang)]
        hits = sum(1 for s in truth if s.key in detected)
        lo, hi = wilson(hits, len(truth))
        rows.append(Row(entity, lang, len(truth), hits, hits / len(truth), lo, hi))
    return rows

Stratifying by language as well as entity is not optional in a European or Latin American support corpus, and the two axes interact. A name recogniser trained largely on English data will pick up Sarah Whitfield and miss Ngô Thị Bích Hằng, and the aggregate name recall hides that completely if 94% of your corpus is English. The output that a sign-off can use looks like this:

entity            lang  n_true  found  recall   95% CI
national_id       es       118     84   0.712   [0.625, 0.786]
national_id       en        47     39   0.830   [0.699, 0.912]
email             all      341    327   0.959   [0.932, 0.975]
person_name       en      1,204  1,151   0.956   [0.943, 0.966]
person_name       vi        63     38   0.603   [0.480, 0.715]
payment_card      all       55     51   0.927   [0.826, 0.972]

The Vietnamese name row is the kind of finding that only stratification produces, and it changes the remediation entirely: the fix is a language-specific recogniser or a different model, not a threshold change. The Spanish national identity row is the one from chapter 2's table, now with an interval attached, and the interval is wide enough that the true value could be 0.63 — which is worth knowing before anyone signs anything.

This corpus is an evaluation set, and everything from versioning and lineage applies to it unchanged: it gets a version, the detector version that produced each number is recorded against it, and re-measurements after a rule change are compared against the same pinned snapshot. Without that, every threshold tweak produces a number that cannot be compared to the last one, and the trend you think you are watching is mostly corpus churn.

What a recall figure still does not tell you#

Recall on a reviewed corpus is an estimate over the distribution you sampled from, which means it expires. Onboard a customer in a new market, add a channel where users paste screenshots that go through OCR, or change the transcript format so that agent and customer turns are interleaved differently, and the number no longer describes your traffic. A recall measurement is a snapshot with a date on it and needs re-running on a schedule, not a property the detector now has.

It is also a span-level metric, and detection is only the first half of the operation. A detector can find a phone number and the redactor can mask it badly, leaving 06-24 11 87 03 as 06-24 11 87 **, which counts as a true positive in every recall calculation and is still an identifying string. Boundary errors of this kind are common at the edges of entities — a masked address that leaves the house number, a masked name that leaves the surname because the recogniser scored it separately — and they need their own check on the output text rather than on the span list.

Finally, a good recall number says nothing about whether the redacted output is still useful. That is the other side of the ledger entirely, and the tempting response to the table above — push the thresholds down until recall clears 99% everywhere — is the subject of the next chapter, which prices it in the currency of a retrieval system that stops working.