Uncertainty sampling cannot tell hard from undecidable

A teammate ran the next 5,000 items through uncertainty sampling instead of drawing them at random, the retrained model went from 82% to 86%, and the proposal on Monday is to make it the default for the rest of the project. Four points is a large move on a metric that has been crawling, the mechanism is intuitive — spend labels on the items the model is least sure about rather than on the thousands it already gets right — and the saving argument is the one that makes budget holders lean forward, because if 20,000 well-chosen labels do the work of 60,000 random ones then three-quarters of the previous chapter's budget evaporates.

The experiment as run cannot support the claim, because it has one arm. Between the old model and the new one, two things changed at once: the sampler, and the existence of 5,000 additional labelled examples. A model with 5,000 more examples of anything will usually be better, and on a project moving from 30,000 to 35,000 items a four-point gain from volume alone is entirely ordinary. There are also changes nobody was tracking, since those five weeks contained a guideline revision, two new reviewers coming off ramp, and a data refresh that extended the corpus by a month.

The arm that has to be there#

The comparison that means something holds the label budget and the calendar fixed and varies only the selection policy.

Both batches are labelled in the same weeks, by the same reviewers, under the same guideline version, because a batch labelled after a revision is labelled to a different convention and that difference alone can be worth more than the sampler. Both are trained from the same checkpoint with the same recipe and the same number of seeds. The evaluation set predates both policies and does not move between arms.

The obvious objection is cost: the honest experiment labels 10,000 items to make a decision about 5,000, and doubles the arithmetic from the budget model. The answer is that the decision being made is not about 5,000 items, it is about the remaining 25,000 and about whether the team adopts an infrastructure commitment it will keep for a year. Against that, one extra batch is cheap, and the alternative is adopting a policy on evidence that is consistent with the policy doing nothing at all.

Why agreement fell to 0.55 on the selected batch#

The 40-class support-intent version of this project ran the two-arm experiment, and the first thing that came back was not an accuracy number. Reviewer agreement on the uncertainty-sampled batch was 0.55, against 0.87 on the random batch labelled in the same fortnight by the same four people.

There is a reading of that which is half right and therefore dangerous: the sampler is working, it is selecting exactly the ambiguous boundary cases the model needs, low agreement is what boundary cases look like, so adjudicate what you can and keep going. The reason it is only half right is that predictive entropy is high for two entirely different populations of item, and the acquisition function cannot distinguish them, because from inside the model they look identical.

Reading 300 items from the uncertainty batch on this project sorted them into 38% hard-but-decidable, 41% undecidable, and 21% ordinary items the model happened to be unsure about for no interesting reason. The undecidable 41% breaks down further into 19% truncated by the export, 13% messages containing two genuine intents that the single-label schema cannot represent, 6% non-English, and 3% spam and test tickets. Accepting the batch whole puts 4,100 items of that into training with labels that were, at best, a reviewer's guess between two defensible options.

There is a cost consequence as well as a quality one. Adjudication on the random batch ran at the pilot's 9%; on the uncertainty batch it ran at 34%. At 4.5 senior minutes an item, adjudicating 10,000 uncertainty-sampled items costs 255 senior hours against 68 for a random batch, which is about $17,700 of senior time that the "fewer labels, same accuracy" pitch did not include. Active learning shifts cost from reviewers to adjudicators, and adjudicator time is the scarcest line in the budget.

Triage the batch instead of accepting it#

The fix is to put a gate in front of the acquisition function and a triage step behind it.

def acquire(pool, model, k, min_chars=200):
    eligible = [x for x in pool
                if not x.truncated
                and len(x.text) >= min_chars
                and x.lang == "en"
                and not x.is_duplicate]
    scored = [(margin(model.predict_proba(x.text)), x) for x in eligible]
    scored.sort(key=lambda s: s[0])              # smallest margin first
    return diversify([x for _, x in scored[: k * 4]], k)

def triage(batch, labels_a, labels_b, gold_owner):
    """Split a double-labelled uncertainty batch three ways."""
    agreed, adjudicate, quarantine = [], [], []
    for item in batch:
        a, b = labels_a[item.id], labels_b[item.id]
        if a == b:
            agreed.append(item)
        elif item.flag in {"TRUNCATED", "MULTI_INTENT", "NO_INTENT"}:
            quarantine.append((item, item.flag))   # never enters training
        else:
            adjudicate.append(item)                # goes to gold_owner
    return agreed, adjudicate, quarantine

The min_chars and truncated filters remove most of the undecidable mass before a reviewer ever sees it, which is the cheapest possible intervention and the one that gets skipped because the acquisition function is where the interesting mathematics lives. diversify matters for a different reason: the lowest-margin 5,000 items in a pool frequently contain hundreds of near-duplicates from one broken automated flow, and labelling the same item 300 times buys one item's worth of information at 300 items' cost.

Quarantine has to be a first-class outcome with a reason code rather than a deletion, for two reasons. The counts are the input to a fix somewhere else — 19% truncated is a bug in the export job worth more than any sampling policy, and 13% multi-intent is an argument for changing the schema. And a quarantined item removed silently biases the corpus in a way nobody can later reconstruct.

Why 79.4% does not settle it either#

The batch went in, the model retrained, and accuracy on the existing test set moved from 79.0% to 79.4% on a 40-class problem. Three separate things prevent that number from deciding anything.

Run-to-run variance swallows it. Retraining the baseline five times with different seeds on this project gave 78.6%, 79.0%, 79.3%, 79.8% and 80.1%, a spread of 1.5 points from nothing but initialisation and data order. A single run landing 0.4 points above a single baseline run is inside that spread, and the fix is to report the mean and spread over seeds for both arms rather than a point estimate for each.

The test set was drawn under the sampling policy being replaced. It is a random sample of production, so it is dominated by the items uncertainty sampling deliberately stopped labelling — the head classes the model already handles. If the new labels sharpened boundaries that occur on 3% of production traffic, the largest plausible effect on this test set is about 3 points and the realistic one is a few tenths, which is exactly what was observed and is equally consistent with the policy working well and with it doing nothing.

The aggregate hides the classes the labels were aimed at, in the way the earlier chapter's 12-class kappa hid its rare classes. The uncertainty batch concentrated on eleven of the forty intents; per-class accuracy on those eleven went from 51% to 58%, while the head classes lost 0.3 points to the reduced share of head-class training data. That is a real and useful result which the aggregate reports as noise. Deciding on the aggregate means deciding on a number that the intervention was structurally unable to move.

What to ask for, then, is the delta on the targeted classes with seed spread on both arms, on a test set refreshed under production-like sampling, and ideally a metric tied to the decision the model serves — on a router, the share of tickets landing in a queue that cannot resolve them, which is what the 51% to 58% actually buys.

What the triaged experiment actually returned#

Running the two arms with the gate and the triage in place, on 5,000 accepted items per arm, gave a different picture from either the enthusiastic 86% or the discouraging 79.4%. Averaged over five seeds, the random arm went from 79.0% to 79.6% and the filtered uncertainty arm to 80.9%, a gap of 1.3 points that sits outside the 1.5-point seed spread only because the spread narrows when you compare paired seeds on the same test set. On the eleven targeted classes the gap was 6.4 points. Adjudication on the filtered uncertainty arm ran at 19% rather than 34%, because the gate removed most of the undecidable mass, which brought the senior-time premium down from roughly $17,700 per 10,000 items to about $6,700.

That is a real result and a modest one, and it is the shape most honest active-learning evaluations have. The policy is worth adopting on this project, for the targeted classes, with the gate, at a cost premium that has to be written into the adjudication line rather than assumed away. It is not the three-quarters budget saving that opened the discussion, and the distance between those two conclusions is entirely a matter of which arms were run.

What active learning does not fix#

It does not fix a guideline. A sampler that concentrates the reviewer pool on the most ambiguous items in the corpus amplifies every defect in the document, because ambiguous items are where the document is silent, and a pool with a shared wrong convention will apply it more often and more consistently on an uncertainty batch than on a random one.

It does not produce a corpus you can compute statistics on. The resulting dataset is a biased sample by construction, so class priors estimated from it are wrong, and it cannot supply a test set. Test data has to keep coming from production-like sampling regardless of how training data is chosen, which is a standing cost the "fewer labels" pitch tends to forget.

It also has a cold start and a serialisation cost. Below a few thousand labels the model's uncertainty is not informative enough to select on, and every batch requires a training run to complete before the next batch can be chosen, which turns a labelling operation that could run continuously into one that stops for a day every fortnight. On a project where reviewer capacity is booked in advance, idle reviewers are a real line in the budget.

Where the field has largely moved is model-assisted labelling rather than model-assisted selection: the model pre-labels every item with a confidence, reviewers verify the high-confidence ones at a fraction of the time cost and label the rest from scratch, and the saving shows up as seconds per item rather than as items. The arxiv tutorial "Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop" and the AAIF workshop "Automating Data Annotation with LLMs" both work through where that holds and where it fails, and the failure mode they converge on is the one this topic keeps returning to — a pre-label that reviewers accept without reading is a systematic error applied at machine speed, and it will not show up in any agreement number.

Running an annotation operation well is mostly a matter of doing these things in an order that lets each one inform the next: write the guideline out of a disagreement audit, measure agreement per class and per slice rather than in aggregate, build a small gold set with authority over the convention, pilot with everyone who will work the project, budget adjudication and rework as first-class lines with a named owner, and treat any change to how items are selected as an experiment that needs a control arm. The next dataset you are handed is an opportunity to check which of those the people who produced it actually did, and the answers are usually visible in an afternoon of reading disagreements.