A guideline is written out of the disagreements it resolves

Two reviewers each labelled the same 400 support tickets as billing, technical, or account, working from a one-page guideline that gave each category a one-sentence definition and two examples. They agreed on 364 of them, which is 91%, and the annotation lead put that number in the weekly update as evidence that the taxonomy was working and the full 40,000-ticket run could start. Someone asked to see the 36 disagreements before signing off, which took about an hour to read, and they were not spread evenly across the label space at all. Twenty-six of the thirty-six were the same ticket: a customer who cannot log in, whose subscription lapsed because a card expired.

Here is one of them, lightly trimmed.

Ticket #48812
Subject: can't log in
Body:
  I've been trying to get into the dashboard since Friday and it just
  bounces me straight back to the login screen every time. My card
  expired last month, I did go and update it in the billing portal on
  Saturday, but nothing changed. Can someone reset whatever needs
  resetting on my account?

Reviewer A labelled it account, because the customer is describing a login failure and the guideline said account covers "access, login, permissions and profile issues". Reviewer B labelled it billing, because the guideline said billing covers "invoices, payment methods, subscriptions and refunds" and the ticket says the card expired. Both of them applied the document correctly. The document has two rules that fire on the same ticket and says nothing about which one wins, so the label depends on which sentence the reviewer read first, or which part of the ticket they anchored on, and neither reviewer was ever aware they were making a choice.

The disagreements have a shape, and the shape is the work#

Grouping the 36 disagreements by the pair of labels involved took a spreadsheet and twenty minutes, and it converts a vague sense that "the guideline is a bit loose" into a short list of specific decisions somebody has to make.

Twenty-six items is 6.5% of the sample, and if the same rate holds over 40,000 tickets that is roughly 2,600 items whose label is decided by a coin flip inside a reviewer's head. It is worse than a coin flip in one respect, because the two reviewers are not flipping independently over time. Reviewer A anchors on the symptom and will keep labelling lockouts as account; Reviewer B anchors on the cause and will keep labelling them billing. Whichever of them ends up doing more of the volume determines what the class means in the finished dataset, and nothing in the process records that this happened.

The other clusters are smaller and are not less important. The five password-reset tickets are a real question about where the boundary between infrastructure and account state sits, and the three checkout 500s are a question the taxonomy was never designed to answer, since a payment-flow bug is genuinely both a billing surface and a technical fault. What the audit gives you is not the answer to any of these; it is the guarantee that these are the four questions and that there is not a fifth one hiding in the 91%.

What the document was missing#

The one-page guideline was a taxonomy: three names, three definitions, six examples chosen because they were clean. Taxonomies are tempting to write because they read as complete — every ticket has to be one of the three, the categories look mutually exclusive when you say them out loud, and a page that fits on a page feels like a well-designed page. The thing a taxonomy does not contain is any account of what to do when two definitions both apply, and that is the only situation where a guideline is doing any work at all. On the tickets where the answer is obvious, the document is decoration; the reviewers would have agreed with no guideline whatsoever.

An annotation guideline is closer in kind to an API specification than to a glossary. A glossary tells you what a term means. A specification tells you what happens on the inputs that are awkward: what the endpoint returns when two fields conflict, which validation runs first, what error the caller gets when both a retry and a timeout are in flight. The value of the document is concentrated entirely in its handling of the awkward inputs, and the awkward inputs are not knowable in advance from the design — you find them by running traffic through and reading what broke. The 36 disagreements are that traffic.

The revised section for billing and account looks like this. It is longer than the sentence it replaces, and it is longer specifically in the places where the sentence failed.

## billing

Tickets whose resolution requires a change to money, entitlement, or
subscription state.

### Included
- Invoice questions, refunds, proration disputes
- Failed or expired payment methods
- Plan changes, cancellations, renewals
- **Access failures whose proximate cause is a lapsed or unpaid
  subscription**, even when the customer describes them as a login
  problem

### Worked boundary case — lockout after a failed payment (#48812)

> "I've been trying to get into the dashboard since Friday and it just
> bounces me back to the login screen. My card expired last month, I
> did go and update it in the billing portal on Saturday, but nothing
> changed."

Label: `billing`.

The customer describes a login symptom, which makes `account` the
natural first read. It is `billing` because the resolution path runs
through the subscription: someone has to confirm the payment settled
and re-issue entitlement. Nothing about the account record itself is
wrong.

Counter-case: if the same customer's card is current and payments are
settling, an identical login-loop message is `technical`, because the
resolution path is now a session or auth bug.

### Precedence

When a ticket satisfies more than one category, label it with the
category that owns the *resolution*, not the one that matches the
symptom. Resolution ownership breaks ties in this order:

1. `billing` — if entitlement or payment state must change
2. `technical` — if a system is behaving incorrectly for a valid account
3. `account` — everything else about the account record itself

Ruled by: Priya Raghavan (support-ops), 2026-03-14. Guideline v4.

Three things in that block do not exist in a taxonomy. The worked case reproduces the actual ticket text rather than a sanitised paraphrase, because reviewers pattern-match against surface features and a paraphrase strips exactly the surface features they were matching on. The counter-case pins down what the rule does not cover, which is what stops it being over-applied to every ticket that mentions a card. The precedence rule is a total order over the categories, so any conflict between any two of them resolves without a further ruling, which is what makes it worth more than a case-by-case answer.

The other clusters do not all get the same treatment#

Writing the precedence rule closes the 26-item cluster and, because precedence is a total order, it also closes the five password-reset tickets without a separate ruling: a reset email that never arrives is a system misbehaving for a valid account, so technical wins over account. That is the return on writing an ordering rather than a case-by-case answer, and it is why the ordering is worth the argument it takes to agree on.

The three checkout tickets are a different kind of problem and the rule does not honestly resolve them. A payment page returning a 500 during a card update is a technical fault whose resolution genuinely sits with two teams, and forcing it into billing because entitlement is involved produces a label that routes the ticket to people who cannot fix it. The options are to add a fourth class, to allow a secondary label, or to accept the mis-route and record that you accepted it. Three items in 400 is 0.75% of volume, which on 40,000 tickets is 300 tickets a quarter arriving in a queue that will bounce them, and whether that is worth a schema change is a decision for the team that staffs the queue. What the audit contributes is the number, since the alternative was an argument about whether this "ever really happens".

A guideline that absorbs every ruling grows, and the growth has a cost that shows up as reviewer time rather than as a complaint. The document on this project went from one page to four over the first three weeks, and reviewers began opening it on nearly half of items instead of skimming it once a morning. Two disciplines keep that in check. Rulings that generalise become precedence rules or definition edits, and only the ones that cannot be generalised stay as worked cases, which keeps the case list from becoming a growing pile of anecdotes. And each class gets its examples ordered with the boundary cases first, because a reviewer consulting the document under time pressure is almost never checking whether an obvious ticket is obvious.

Somebody's name has to be on it#

The precedence rule above did not fall out of the data. Somebody decided that resolution ownership beats symptom, and that decision is a judgement about how the labels will be used downstream — in this case, because the classifier's output routes tickets to queues, and the queue that can actually fix a lapsed-entitlement lockout is billing operations. A different downstream use would justify a different rule. If the labels were feeding a report on what customers complain about, symptom would be the right key and the same ticket would be account.

Because it is a judgement, it needs a person rather than a process. A named adjudicator with authority over the guideline is the difference between a question that gets answered in a day and a question that gets discussed in every reviewer's head separately for the rest of the project. The person needs three things: the authority to decide, enough context on the downstream use to decide well, and an obligation to write the decision back into the document. The write-back is the part that gets dropped, and dropping it means the adjudicator answers the same question eleven times over the project while the reviewers who did not happen to ask keep guessing.

The loop that keeps a guideline alive runs like this.

The flag state matters as much as the ruling. Reviewers need somewhere to put an item other than one of the three labels, because the alternative is that they pick the closest one and move on, and a guessed label is indistinguishable from a confident one once it is in the dataset. An UNRESOLVED value costs a little throughput and buys the visibility that the entire loop depends on.

The version bump matters for the same reason. Once a ruling changes what billing means, every already-labelled item that matches the ruling is now labelled under an old convention, and the dataset silently contains two definitions of the class. Argilla's write-up on maintaining data quality through guideline changes is worth reading on this specifically; the practice they land on is to treat each guideline version as an attribute of the label, so that you can always ask which convention a given annotation was produced under and re-label the affected slice rather than the whole corpus. On this project, the v4 ruling forced a sweep of 26 pilot items plus whatever had been labelled in the two days between, which is a small number caught early and would have been 2,600 items caught late.

What this does not do#

A guideline of this kind does not make the ambiguous items unambiguous. Ticket #48812 is genuinely both a billing matter and an access matter, and the precedence rule does not discover a hidden truth about it — it imposes a convention so that every reviewer imposes the same one. That distinction matters when you get to the point of asking whether the labels are right rather than merely consistent, which is a different question with a different instrument.

It is also not reviewer training, though it is easy to spend the budget as if it were. Sending the two reviewers on a calibration session would have raised agreement on lockout tickets, because they would have converged on somebody's convention in the room, and the convention would have lived in their heads where the next reviewer to join cannot read it. Anything that improves agreement without changing the document has bought you a result that decays the moment the roster changes.

And it does not eliminate disagreement. It relocates disagreement out of reviewers' private judgement and into a queue where it is counted, which has a cost per item that somebody eventually has to put in a budget.

What the audit could not tell us is whether 91% was a good number in the first place. On three classes with this distribution it happens to be informative, but the same headline figure on a different label distribution can be produced by two reviewers who are barely paying attention, and there is a standard way to be misled by it.