Without a mechanical oracle the human grades in batches

The experiment ran for five working days on one harness and two queues. Queue A was failing tests in the notifications service, 41 tasks. Queue B was rewriting user-facing error strings to be clearer, 38 tasks, graded against a four-criterion rubric — a written scale a person applies by hand to turn a judgement into a score — that a product designer had written and applied herself. At the end of the week Queue A had a 66% acceptance rate and Queue B had 63%, and the product manager's reading was that the agent handles copy about as well as it handles code, so the copy queue should be expanded.

The acceptance rates are the same number measuring two different things, and the difference shows up the moment you look at what arrives when a task is rejected. This is what the agent received on a Queue A rejection:

FAILED tests/test_digest.py::test_quiet_hours_suppresses_push
  assert 1 == 0
   +  where 1 = len([<PushEvent id=8821 sent_at=02:14 tz='Europe/Madrid'>])
notifications/digest.py:118: in _should_send
    return now.hour not in QUIET_HOURS

And this is what it received on a Queue B rejection, 31 hours later:

{
  "batch_id": "strings-2291",
  "strings_changed": 8,
  "scores": {"clarity": 4, "tone": 3, "actionability": 2, "voice_consistency": 4},
  "weighted": 3.2,
  "threshold": 3.75,
  "comment": "Too apologetic in places, and 'contact support' isn't an action
              most users can actually take at 2am."
}

The first names a file, a line, a function, and the two values that disagree, and it arrived 8 seconds after the patch existed. The second is a scalar and a sentence covering eight rewritten strings at once, and it arrived a day and a half later because the designer grades on Tuesday and Thursday afternoons.

Credit assignment is the thing that broke#

The agent's next attempt on the Queue A task has somewhere to go. _should_send is the function, now.hour is the expression, and the timezone in the failing event is a strong hint that the comparison is happening in UTC. Whatever it changes, the change is aimed.

On the Queue B task, nothing in the feedback says which of the eight strings dragged the score down. "Too apologetic" might describe one string or five. The rewrite that follows will touch all eight, including the three the designer thought were an improvement, and those three are now at risk of getting worse in a way the next batch score cannot distinguish from the first problem being fixed. A scalar over eight items carries, at best, enough information to say the batch is bad; localising the cause takes roughly three more rounds of bisection, at 31 hours a round, which is nine days to learn something the Queue A loop learns in eight seconds.

The row that constrains the whole enterprise is the verdicts-per-week ceiling. The model can generate 38 rewrites in an afternoon, and the loop cannot advance faster than one human's grading throughput, which means adding model capacity to Queue B changes nothing at all. The 1.4 attempts figure is the visible consequence: the agent is not converging in 1.4 attempts, it is being accepted or abandoned in 1.4 attempts because a second round costs another day and a half, and the designer would rather fix the string herself than wait. That behaviour looks like efficiency in a dashboard and is a queue quietly failing.

The partial verifier is real and does not cover the graded property#

Error strings are not entirely uncheckable, and the checkable part is worth building because it is cheap and it removes a whole category of rework from the designer's plate.

# checks/strings.py — runs in 0.3 s over the full catalogue
BANNED = {"oops", "whoops", "something went wrong", "please try again later"}
ICU_ARGS = re.compile(r"\{(\w+)\}")

def check_entry(key: str, new: str, old: str) -> list[str]:
    errs = []
    if len(new) > 120:
        errs.append(f"{key}: {len(new)} chars, limit is 120 (truncates on mobile)")
    if ICU_ARGS.findall(new) != ICU_ARGS.findall(old):
        errs.append(f"{key}: placeholder set changed {ICU_ARGS.findall(old)} -> ...")
    if new.lower() in BANNED or any(b in new.lower() for b in BANNED):
        errs.append(f"{key}: matches banned phrase list")
    if key not in load_catalogue("es") or key not in load_catalogue("de"):
        errs.append(f"{key}: missing translation, will fall back to English")
    return errs

That check has every property from the opening chapter: per-output, deterministic, sub-second, and mostly built already if you have an internationalisation pipeline, the machinery that keeps one message catalogue per language and swaps them at render time. It catches a mangled ICU placeholder — one of the {month}-style slots that the message-formatting library fills in with a real value — that would have rendered Your invoice for {month} is ready as literal braces in production, and it catches a string that overflows on a 375-pixel viewport. It says nothing whatsoever about whether the string is clearer than the one it replaced, which is the property the ticket was about. The partial verifier moves a slice of the work into the fast, self-correcting regime and leaves the graded property exactly where it was, and the mistake to avoid is letting a green string-lint create the impression that the batch is in good shape.

The same test, applied to a docs build#

A product manager who has watched the coding agent work will eventually propose the API reference pages: the agent reads the source, writes the reference, and the site build runs in CI, where a broken code sample fails the build. The observation is correct, and the argument that this makes documentation as verifiable as code needs the check applied item by item.

The docs build verifies that the samples compile against the current API, that internal links and anchors resolve, that every page has valid frontmatter, which is the small metadata block at the top of each source file naming its title and slug, and that no page references a symbol the extractor cannot find. Those are real, deterministic, per-page checks costing about 90 seconds, and they will catch the class of error where the agent writes client.fetch_order(id) for a method that is now client.get_order(order_id).

What it leaves standing is everything the prose asserts. A sample that compiles and returns a value can sit underneath a paragraph saying the call retries three times on a 502 when the client retries twice and only on connection errors. A page can be complete, well-formed, entirely resolvable, and describe a timeout parameter as seconds when the implementation passes it to a library that expects milliseconds. The build has no opinion about the sentences, and the sentences are what the reader acts on. So the docs domain has the shape of Queue B with a thin, genuinely useful mechanical shell: the compile-and-link layer behaves like code, and the accuracy layer behaves like a rubric, with a human oracle grading a page at a time, late and in batches.

Recovery is where the gap is widest#

Run the comparison deliberately and it is worth scoring the two queues on three axes rather than one, because acceptance rate collapses all three into a number that happens to match.

Correctness means different things on each queue, and the difference is not a matter of rigour. On Queue A, correct denotes a property of the artifact — the suite is green, the behaviour matches the assertion — and two people looking at the same patch will agree on whether it holds. On Queue B, correct denotes a relation between the artifact and a reader's response, mediated by a rubric that a second designer would apply differently. When the same eight strings were re-graded by a colleague as a control, the weighted score moved from 3.2 to 3.7, which crosses the 3.75 threshold's neighbourhood and would have flipped one of two batches from reject to accept. A verifier whose verdict depends on who is holding it is not a gate, it is an opinion with a number attached, and the number invites more trust than the opinion would have got on its own.

Feedback quality is the axis already visible in the two payloads above, and the way to score it is to ask what the next attempt can condition on. A pytest failure is a pointer into the artifact. A rubric score is a summary statistic over the artifact. Everything about iteration follows from that, including the fact that Queue A's second attempts succeed 61% of the time while Queue B's succeed 34% — a second attempt aimed at nothing in particular is close to a fresh draw.

Recovery is the axis nobody measures and the one that decides where the work actually lands. Track the share of accepted outputs that a human subsequently edited before shipping: on Queue A it was 9%, and those edits were mostly naming and comments. On Queue B it was 71%, and the designer's own estimate was that she spent about seven minutes per accepted string adjusting it, which is close to what writing it from scratch costs her. Acceptance on Queue B is therefore not the end of the work; it is a handoff of a draft, and the week's 24 accepted strings represent perhaps 40% of the effort saved that the acceptance rate implies. The corresponding number on Queue A — patches merged and then reverted or hot-fixed within seven days — was 4%.

Those three axes are the rubric worth writing down before you run this comparison on your own queues, because all three are collectible in a week and only the first one has an obvious place to look.

What this does not argue#

None of this says to keep agents away from subjective work. It says that the loop you get there is a different machine, and it should be designed as one: larger batches so a grader's session covers more ground, the constraints pushed forward into the prompt as examples and a style guide rather than discovered through rejections, a fixed grading slot on the calendar so latency is predictable, and an explicit acceptance that the output is a draft for a human to finish rather than a patch that converges. Queue B produced 24 accepted strings in a week, which is real work, and it produced them the way a fast writer with an editor produces work.

The mistake is applying the code-loop's economics to it — expecting the throughput to scale with model capacity, expecting acceptance rate to be comparable across queues, or expecting that adding attempts improves quality. When a team reports that their agent programme works equally well across domains, the number they are usually comparing is acceptance rate, which is the one number that hides the entire difference laid out above.

There is one move that appears at this point in every version of this conversation, and it appears because the human grader's slowness is so obviously the bottleneck: have a strong model apply the rubric instead. It is fast, it is cheap, it can grade every string individually rather than in batches of eight, and it fails in a way that is worth an entire chapter.