A pilot exists to produce throughput, defects and adjudication load

The requisition on the table asks for 120,000 labelled items and quotes six weeks. Nobody in the room has labelled one of these items. The six weeks came from a per-item time that someone in the meeting produced by labelling a handful of examples on their laptop while the taxonomy was being discussed, and the handful were the examples the taxonomy had been designed around, so they went quickly. Approving the requisition commits the budget, the vendor contract and the model team's Q3 plan to that number.

The alternative is to spend 300 items first. At the rates this project eventually measured, 300 items costs about $1,270 all in, against roughly $97,000 for the full single-labelled run, so the pilot is a little over 1% of the thing it is estimating. What comes back is four rates that the estimate needs and cannot otherwise obtain: how long an item actually takes, how often reviewers get it wrong against a reference, how often they disagree with each other, and how much senior time the disagreements consume.

Run it like a load test, not like a demo#

A demo runs clean inputs through the system in front of an audience, operated by the people who built it. A load test runs production traffic through the system, operated by whatever is going to be operating it in production, and its entire value comes from the fact that nobody has curated the inputs. A pilot that is really a demo produces numbers that are all biased in the same direction, and the direction is optimistic.

The sample has to be drawn the way the corpus will be drawn. If the production selection is "every ticket created in a given week", draw a week of tickets, not a stratified sample with the classes balanced and the confusing ones dropped. Drawing 300 items this way on the intent project turned up 14 conversations truncated by the export job's 4,096-character limit, 6 tickets not in English, 4 exact duplicates from a retry bug, and 2 pieces of spam. Those 26 items are 8.7% of the sample, and none of them would have appeared in a curated pilot. What they cost is disproportionate to their number, because no reviewer had been told what to do with them: the median time on those 26 items was 150 seconds against 82 seconds on ordinary items, and most of that was reviewers reading a truncated transcript three times looking for an intent that had been cut off.

The people have to be the people. All four reviewers who will work the run labelled all 300 items, independently, which produces 1,200 annotations and costs four times as much as having one person do it. That is the expensive part of the design and it is what buys three of the four outputs, since agreement, adjudication load and the spread of throughput across individuals all require more than one person to have seen the same item. The reviewers must not be the people who wrote the guideline. An author labels faster because they are not consulting a document they are recalling, and their defect rate against a reference they themselves authored is close to definitionally zero, so both of the numbers they produce are unusable.

Instrument it so the numbers are measured rather than remembered#

Self-reported timings are consistently wrong, usually low, because people do not count the twenty seconds spent scrolling back to re-read the guideline. The labelling tool logs events, and the two clocks it needs to distinguish are active time and wall time.

{"event":"served","item":"tk-104882","reviewer":"R3","ts":"2026-04-02T09:14:22Z"}
{"event":"blur","item":"tk-104882","reviewer":"R3","ts":"2026-04-02T09:15:58Z"}
{"event":"focus","item":"tk-104882","reviewer":"R3","ts":"2026-04-02T09:17:31Z"}
{"event":"guideline_open","item":"tk-104882","reviewer":"R3","section":"returns","ts":"2026-04-02T09:17:40Z"}
{"event":"submit","item":"tk-104882","reviewer":"R3","label":"returns",
 "active_ms":96400,"wall_ms":214000,"flag":null}

Wall time on that item is 214 seconds and active time is 96, because the reviewer went to lunch mid-item. Throughput planning uses active time, because that is what scales with volume; wall time matters separately for scheduling, since a reviewer who is nominally on the project for four hours a day delivers considerably less than four hours of active labelling. The guideline_open event is worth logging on its own, because a section that gets opened on 40% of items is a section that is not doing its job, and that is a guideline defect you can find without reading a single disagreement.

The analysis is a handful of aggregations, and the percentiles matter more than the mean.

import numpy as np

active = np.array([a.active_ms for a in pilot]) / 1000
print(f"n={len(active)} median={np.median(active):.0f}s "
      f"mean={active.mean():.0f}s p90={np.percentile(active, 90):.0f}s "
      f"p99={np.percentile(active, 99):.0f}s")
# n=1200 median=82s mean=104s p90=240s p99=511s

per_reviewer = {r: np.median([a.active_ms for a in pilot if a.reviewer == r]) / 1000
                for r in {a.reviewer for a in pilot}}
print(per_reviewer)
# {'R1': 71.0, 'R2': 78.5, 'R3': 88.0, 'R4': 96.5}

The mean is 27% above the median, which is the tail doing work: the slowest 10% of items take 240 seconds or more and contribute about a quarter of the total labelling time. Estimating with the median and then discovering the mean is the most common arithmetic error in this whole area, and it is worth about 20% of the budget on its own. The per-reviewer spread is a second planning input, since a pool median of 82 seconds assembled from people ranging 71 to 97 will move as the roster changes.

Adjudicate the disagreements, because that is where the fourth number is#

Of the 300 items, 41 had at least two reviewers giving different labels, which is 13.7%. Those 41 went through the adjudication loop from the first chapter, and the senior reviewer took a median of 4.5 minutes on each, which is 184 minutes of senior time for a 300-item sample. Per item of corpus that is 37 seconds of senior attention against 82 seconds of reviewer attention, and senior time is priced at three times reviewer time, so on this project adjudication was heading toward 58% of the labelling cost before anyone had written it into an estimate.

The rulings had a second effect. Nine of the 41 produced a written ruling, three of those changed the guideline, and the changes invalidated 22 of the 300 items already labelled, which had to be redone. That ratio — a guideline revision touching 7% of the corpus labelled so far — is itself an estimate you now own, and it is the input to the rework line that most estimates omit entirely.

There is a precision limit here worth being honest about. At n = 300 and an adjudication rate near 0.14, the standard error is about 2 percentage points, so the true rate is somewhere in 10–18% at 95% confidence, and on 120,000 items that spread is worth roughly 9,600 items of senior adjudication either way. The per-class numbers are worse: with 12 classes and a corpus where the rarest class is 1.5%, a 300-item pilot contains four or five instances of it, which supports no agreement estimate at all. A pilot gives you a solid median time, a usable adjudication rate, a rough defect rate, and essentially nothing about the rare classes. Knowing which of your four numbers is soft is part of the deliverable.

The tool is part of the rate you are measuring#

A pilot run in a spreadsheet measures spreadsheet throughput, and that is not a rounding error. On this task the interface accounts for a large share of the per-item time, because the labelling decision itself is often quick and everything around it is not: finding the ticket, scrolling to the last turn, remembering which of 12 classes is which, moving the hand to the mouse, clicking a dropdown, scrolling the dropdown, clicking again, saving, waiting for the next row. Replacing that with a single-key binding per class and automatic advance took the median on the pilot from 82 seconds to 61 in a second pass over the same items, which is a 26% change in the largest line of the budget, obtained without touching the guideline or the people.

That is the practical argument for building a small purpose-made labelling interface rather than adopting a general-purpose platform, which Hamel Husain's "Intro To Error Analysis: Creating Custom Data Annotation Apps" walks through end to end. A single-purpose app that shows exactly the fields this task needs, binds exactly the classes this taxonomy has, and logs exactly the events described above is typically a day or two of work, and it is a day spent against a line worth tens of thousands. The corollary for the pilot is that whatever tool the run will use has to be the tool the pilot uses, because a rate measured on a different interface is a rate for a different project.

The exception path has to exist before the pilot rather than being discovered during it. Reviewers need a flag value for each way an item can be unworkable — truncated, wrong language, duplicate, spam, no intent present — and a one-line instruction on what to do with each. Without them the pilot measures the cost of confusion instead of the cost of the exception, and the 150-second median on those 26 junk items is mostly reviewers being conscientious about an item nobody had told them to abandon.

Three pilot designs, and what each one can tell you#

Design A is what gets run when the pilot is understood as a demonstration that labelling is possible, which was never in question. Design B is the common compromise and it does produce an honest time-per-item, which is the single most valuable number, so it is much better than nothing. Everything that concerns more than one person — agreement, adjudication, the rework rate — requires design C, and those are precisely the lines that the estimate is missing.

The extra $1,000 buys the difference between an estimate with two inputs and an estimate with five. On a $97,000 run with a committed date attached, that is not a close call, and the reason it does not happen more often is that design C looks like it is spending four people's time to label 300 items that one person could have labelled, which is exactly what it is doing and exactly what it is for.

What a pilot is not#

It is not a quality gate on the guideline. The 41 disagreements will surface guideline defects, and they should be fixed, but a pilot that ends with "agreement was 0.86, so the guideline is fine" has substituted a rate for the disagreement audit and will miss the shared-convention failure entirely, since a pool that has already converged on a wrong rule will produce excellent pilot agreement.

It is not a feasibility test for the model. Three hundred labelled items train nothing, and the temptation to fit a small model on the pilot data and read something into the accuracy should be resisted, because whatever number comes back is dominated by the sample size.

And it is not a forecast of steady state. A pilot measures the first 300 items ever labelled on this task, by people in their first days on it, under a guideline at version one. Every one of those conditions changes, and they do not all change in the direction of speed. Turning four pilot rates into a number and a date requires taking a position on how each of them drifts, and on the lines the pilot never measured at all.