Adjudication and rework are the lines that blow the estimate

Last quarter's estimate fitted on one slide. Forty thousand items, a pilot median of 90 seconds, so 1,000 reviewer-hours; at $28 an hour that is $28,000, and with five reviewers at 35 hours a week it is six weeks. The project delivered in week eleven and the invoices came to $61,000. The post-mortem found nothing wrong with the 90 seconds — the reviewers hit it, and the median for the run came in at 93. Every hour of the overrun was in work the slide did not have a row for.

The same arithmetic is now being applied to a 60,000-item project whose pilot came back with 200 items, two reviewers, a median of 75 seconds, agreement of 0.88, and 9% of items requiring adjudication. The one-line version says 60,000 × 75 s × 2 reviewers = 2,500 hours, or $70,000. Building the estimate properly, from the same pilot, produces $150,000 and a plan for eight reviewers over sixteen weeks. The naive figure is 47% of it, which is the ratio that shows up again and again on this kind of work.

Every line the slide is missing#

N = 60_000
MEAN_S = 96          # pilot median 75 s, mean/median ratio 1.28 from the tail
ADJ_RATE, ADJ_MIN = 0.09, 4.5
REV_RATE, SNR_RATE = 28.0, 95.0

def hours(items, seconds):
    return items * seconds / 3600

first_pass   = hours(N, MEAN_S)
second_pass  = hours(N, MEAN_S)                      # double-labelled at 100%
adjudication = N * ADJ_RATE * ADJ_MIN / 60           # senior
rework       = hours(7_200 * 2, MEAN_S)              # 6 revisions x ~5% of corpus
ramp         = hours(8 * 1_000, MEAN_S * (1/0.6 - 1))
qa_seeding   = 0.03 * (first_pass + second_pass)     # 3% gold items in the queue
calibration  = 8 * 1 * 16                            # 1 h/week/reviewer, 16 weeks
guideline    = 4 * 16                                # senior, 4 h/week

reviewer_h = first_pass + second_pass + rework + ramp + qa_seeding + calibration
senior_h   = adjudication + guideline
print(f"{reviewer_h:.0f} reviewer-h  {senior_h:.0f} senior-h  "
      f"${reviewer_h * REV_RATE + senior_h * SNR_RATE:,.0f}")
# 3807 reviewer-h  469 senior-h  $151,151

Laid out as a budget, the interesting column is the last one.

Adjudication is a quarter of the budget on its own, and it is a quarter that has to be bought in a different currency: 405 senior hours over sixteen weeks is 25 hours a week of somebody with authority over the taxonomy. That is not a person who fits adjudication around their existing job, and staffing it as though it were is the specific mechanism by which adjudication queues grow to four weeks deep and reviewers start guessing rather than flagging, which quietly destroys the measurement the double-labelling was paying for.

The rework line rests on the softest assumption in the model, six guideline revisions each invalidating around 5% of what has been labelled so far. That figure came from a pilot that produced three revisions in 300 items, extrapolated with a decay because revision rate falls as the document matures. It could plausibly be half or double. The mean of 96 seconds is the input to distrust second, since the pilot measured two people in their first days on a version-one guideline; the adjudication rate of 9%, measured on 200 items, carries a standard error of about 2 points, so the true rate is somewhere in 5–13% and the adjudication line ranges from $21,000 to $56,000. The item count of 60,000 is the only input that is genuinely known.

Cutting double-labelling from 100% to 10%#

The proposal to halve the cost by double-labelling only 10% of items is roughly right about the money. The second pass falls from 1,600 to 160 hours, and adjudication falls with it, since disagreements can only be detected where two people labelled the same item: 9% of 6,000 rather than 9% of 60,000, so 40 senior hours instead of 405. Together that is about $75,000, and it is half the budget because adjudication is attached to double-labelling rather than being a separate line.

What it spends is the error correction and the measurement. On the 90% of items labelled once, the label is whatever one reviewer decided, so the corpus inherits the reviewer defect rate directly — around 10% on the gold-set measurements from the earlier project, which on 54,000 items is 5,400 wrong labels going into training with nothing to catch them.

The third row is the answer worth arguing for. Double-labelling has diminishing returns exactly where agreement is already high, which is the two head classes carrying most of the volume, and it has its full value on the rare classes and the known-hard slices. Stratifying by class and by the slice cuts from the earlier chapter recovers most of the measurement for two-thirds of the price. Three conditions come with it: the sampling has to be random within each stratum and invisible to reviewers, or effort concentrates on the audited items; the gold seeding stays at 3% regardless, because it is the only correctness signal left once most items are single-labelled; and the reduction does not start until the guideline revision rate has flattened, since early items are the ones most likely to be labelled under a convention that is about to change.

The median moved from 75 to 110 seconds#

Six weeks in, the throughput dashboard shows the median at 110 seconds, which is 47% above the pilot and puts the delivery date somewhere in week 23. Two explanations account for almost every case of this.

The first is composition: the remaining work is harder than the work already done. This happens whenever the queue is not randomly ordered — sorted by date and the earlier months came from a channel with cleaner text, or the reviewers were allowed to skip items and the skipped ones accumulated, or the head classes were exhausted early and what is left is the long tail. The second is process weight: the guideline has grown from one page to nine across five revisions, reviewers now open it on 47% of items rather than 18%, and QA feedback has made everyone slower and more careful. Reviewer turnover belongs in this second family, since a new joiner labelling at 60% of steady rate drags the pool median without anything about the corpus having changed.

Telling them apart takes an afternoon and a held-back sample. Reserve 100 items at the start of the project, drawn from the same distribution as the week-one work, and serve them again in week six mixed into the queue. If the median on those items is now 108 seconds, the corpus is not what changed and the cause is in the process or the people. If it is back near 78 seconds, the corpus got harder and the schedule needs re-forecasting against the remaining composition rather than fixing.

The event log supports a second pass over the same question without any new labelling. Per-reviewer medians separate a pool-wide slowdown from one or two ramping joiners. The guideline_open rate per item, up from 18% to 47%, is direct evidence of lookup burden. Median time per class, compared against the changing class mix in the queue, quantifies how much of the drift is composition arithmetic: if order_status was 55% of week-one items at 48 seconds and is 20% of week-six items, the mix alone accounts for a large part of the move.

Quote a range, and book the re-forecast#

A single number written on a slide becomes a commitment the moment it is read, regardless of what the person who wrote it intended, so the estimate should arrive as a distribution. Sampling the three soft inputs over their plausible ranges — mean seconds per item 85 to 115, adjudication rate 5% to 13%, guideline revisions three to nine — and running the cost model a few thousand times gives a median of $151,000, an 80th percentile of $186,000, and a 95th of $214,000. What gets committed is the 80th percentile, and what gets reported alongside it is which input the spread is coming from, because that tells you what to go and measure to narrow it. On this model the adjudication rate contributes about half the variance, so 200 more double-labelled pilot items are worth more than any refinement of the timing data.

The other half of quoting a range is booking the moments when it gets replaced by measurement. At 10% of volume — 6,000 items, roughly week two — every rate in the model exists as an observation rather than an extrapolation, and the re-forecast either confirms the plan or surfaces a problem while there is still time to change the design. A second checkpoint at 30% catches drift that the first one was too early to see. Both dates go in the plan as deliverables with the same weight as the dataset itself, because the failure being guarded against is not that the estimate was wrong, which it always somewhat is, but that nobody found out until the deadline. The weekly report that supports this is four lines: items completed, rolling defect rate against gold, adjudication queue depth in days, and median seconds per item. Three of those four are leading indicators, and the one everyone watches — items completed — is the only lagging one.

Leftover capacity is not a staffing plan#

There is a version of this project that has no estimate at all. Labelling is done by three people on the support team when they have capacity, the dataset is six weeks late, and its quality has never been measured. Every specific problem in that sentence has a specific cause, and none of them is fixed by asking the three people to go faster.

Capacity that is left over is the first thing an incident takes. Two support escalations in a week and the labelling stops, with no one to notice, because nobody's week is planned around it. There is no owner, so when a reviewer hits an item the guideline does not settle there is no adjudicator and no queue, and the item gets a guessed label. Quality has never been measured because measurement is work, and work that nobody is accountable for does not happen. The slip lands on the model team, who own the deliverable that depends on the data and have no authority over the people producing it, which is the worst possible allocation of responsibility.

The structural fix is to make it funded work with one named owner accountable for both a delivery date and a quality bar, with reviewer hours committed up front rather than drawn from residual capacity. Concretely: hours booked in the support team's plan the way an on-call rotation is booked, an owner who is not the model team's tech lead, a stated quality bar expressed against a gold set rather than as agreement, and a weekly report of items completed, defect rate and adjudication queue depth. That is more overhead than three people doing it in the gaps, and it is the overhead that makes the date real.

Everything so far has assumed the 60,000 items are drawn at random from production, and that assumption is doing a great deal of work in the cost model. If a smarter selection could reach the same accuracy with 20,000 labels, three-quarters of this budget disappears, which is why the proposal to choose items by model uncertainty is so appealing and why it needs to be checked more carefully than it usually is.