Majority-class and keyword baselines

Two arms cost almost nothing and belong in every classifier comparison. The first is the majority-class predictor, which ignores its input entirely and always returns the most common label; it is not a candidate for shipping, it is the floor that says how much of a headline accuracy figure comes from the model and how much from the traffic mix. The second is a keyword or regular-expression rule set, written in an hour by somebody who has read two hundred inputs, which is a lower bound on what pattern matching alone can do on this task.

Neither requires a training run, a labelling campaign or a new dependency, and both are scored by the same harness that scores the model. What they need is the thing an LLM classifier already needed: a fixed set of inputs with labels assigned independently of any prediction, and a written policy resolving the cases where two labels are both defensible. That shared requirement is what makes the objection this chapter is mostly about collapse.

The objection contradicts the number in the same document#

The objection arrives phrased almost identically every time: there is no labelled training data for an alternative, so there is no baseline to compare against. It is worth taking apart carefully rather than waving off, because the person saying it is usually not being lazy. They have an intuition that supervised learning is a heavyweight commitment and prompting is not, and the intuition has some truth in it. The conclusion drawn from it does not follow.

A design document that reports 93% accuracy on 400 hand-labelled complaints has already disproved it. Those 400 labels are labelled data. They are not enough to fit a good classifier — a few hundred examples across three classes is thin — but the claim that none exists is false on the document's own evidence, and the difference matters because it changes the question from "can we get labels" to "do we have enough, and what would more cost". Those are answerable with numbers attached, whereas "there is no labelled data" is a door that closes the conversation.

The position has to resolve one way or the other. If a team genuinely has no labels, the 93% has no basis and the honest statement is that quality is unmeasured. If the 93% is real, labels exist and the alternative is not blocked, only under-resourced. There is no third arrangement in which the model has a defensible accuracy number and nothing else can be evaluated at all, because both arms need exactly the same instrument.

The few-shot examples make the point sharper. Each of the twelve worked pairs pasted into the prompt is an input with a correct label, chosen by a person who read the input and applied the policy, which is the definition of a hand-labelled example. The prompt is a labelled dataset with a peculiar delivery mechanism. Whatever it costs to produce the thirteenth example is what it costs to produce the two-thousandth, modulo tedium, and the team has already demonstrated it can pay that cost twelve times over.

Two arms that need no training run#

The constant predictor is the one people skip and the one that reframes a headline. If delivery_issue is 58% of complaints, a predictor that always returns it scores 58%, and a model at 93% has bought 35 points over doing nothing. That framing is uncomfortable in exactly the right way when the traffic mix is more skewed: on a task where one class is 88% of traffic, any accuracy figure below the low nineties is actively bad news.

Why the constant predictor's score is exactly the majority share deserves a line of reasoning rather than a line of assertion. The predictor returns delivery_issue on every complaint. It is therefore right on precisely those complaints whose true label is delivery_issue, and wrong on every other one, so its accuracy is the count of that class divided by the size of the set — which is the definition of that class's share of the set. Nothing about the inputs enters the expression at any point, which is why this is the one arm whose score can be worked out before it is run. Subtracting is then the right operation because both figures are shares of the same rows: 93 and 58 count complaints out of the same four hundred, so the difference reads directly as a net thirty-five complaints in every hundred that the model gets right and the floor does not.

The same identity is what makes the skewed case alarming, and it is worth walking rather than asserting. If one class carries 88% of traffic then the floor sits at eighty-eight by the argument above. A model reporting ninety has bought two points. A model reporting eighty-five is losing to a function that reads none of its input. The accuracy figure has not changed between the two worlds; the quantity it should be read against has. That is the reason for insisting on this arm even where nobody would consider shipping it — an accuracy number is uninterpretable without the floor under it, the floor is a property of the traffic rather than of the system, and a traffic mix remembered from a different quarter is not the one the score was measured on.

There is a second reason to run it that has nothing to do with rhetoric. A constant predictor exercises the whole harness — the loader, the label policy, the per-class report — against a system whose answers are known in advance, so a harness bug that inflates every arm equally shows up here as a floor that does not match the class share you can count by hand. It takes five minutes and it is the only arm whose correct score you already know.

The regex arm is the whole of the second one.

import re

RULES = [
    ("billing_dispute", re.compile(
        r"\b(refund|charged?|invoice|double[- ]?bill|overcharg|"
        r"credit card|payment|subscription)\b", re.I)),
    ("delivery_issue", re.compile(
        r"\b(deliver|courier|tracking|dispatch|parcel|shipment|"
        r"never arrived|late|lost in transit)\b", re.I)),
    ("product_defect", re.compile(
        r"\b(broken|faulty|defect|cracked|doesn'?t work|stopped working|"
        r"damaged|missing part)\b", re.I)),
]

MAJORITY = "delivery_issue"

def classify(text: str) -> str:
    scores = {label: len(rx.findall(text)) for label, rx in RULES}
    best = max(scores, key=scores.get)
    return best if scores[best] > 0 else MAJORITY

Twenty lines including the imports, no dependencies beyond the standard library, and about 40 microseconds per complaint on a laptop. The tie-breaking is crude — highest keyword count wins, ties broken by dictionary order — and the crudeness is deliberate, because the arm is a lower bound rather than a competitor anyone is trying to make win.

The function is worth reading line by line, because its most consequential behaviour is on the last line and it shows up in the table below. For each label, findall returns every place in the complaint where that label's pattern matched, and len reduces that to a count of keyword hits, so a complaint mentioning both a refund and an invoice scores two for billing_dispute. max(scores, key=scores.get) then takes whichever label ended up with the highest count. The return line decides what happens when nothing matched: if the winning count is zero — no pattern in any of the three rule sets fired — the function returns MAJORITY rather than admitting it has no opinion. A complaint written in words nobody put in a pattern is therefore not recorded as an abstention. It is recorded as delivery_issue.

That one line explains the shape of the arm's per-class results without anyone needing to open a confusion matrix. Recall on the majority class is propped up by the fallback, because a complaint the rules cannot read at all is guessed correctly whenever it happened to be a delivery issue. Recall on the two minority classes absorbs the whole of the damage, because a billing dispute phrased without any of the eight billing keywords lands in delivery by default. The fallback is part of what makes this a fair floor rather than a flaw in it: a rule set that abstained instead would score lower, and it would not be a floor for a system whose job is to return a label on every complaint.

Running all three arms over the same 400 complaints, under the same label policy, produces a table a design review can argue about. Recall, in the column below, is the share of a class's complaints that reached the right label, which is the number a team downstream of one class cares about and the one an aggregate hides.

The differences down that table are the sentence this chapter exists to produce, and they have to be taken column by column because the columns are not in the same currency. The aggregate column subtracts cleanly: the rules are worth twenty-six and a half points over the floor, and the model another eight and a half over the rules, all three being shares of the same four hundred complaints. The recall column is on a nought-to-one scale rather than a percentage, so a "point" there is a hundredth of it — 0.90 against 0.71 is a gap of nineteen hundredths, which is where the 19 points of billing_dispute recall in the takeaway comes from. Those two point counts must never be added or averaged. One is counted over complaints of every kind and the other over billing disputes only, and the entire reason to report both is that they disagree about how big the gap is: on the whole traffic the model is worth single digits, and on the class that turns into a payment dispute it is worth roughly twice that.

The last three columns price that gap, and each is quoted per unit for a reason worth knowing. Cost per 1k is the provider's charge for a thousand classifications, so $0.42 works out at roughly a twentieth of a cent per complaint; it is quoted per thousand because per complaint it rounds away to nothing and stops being comparable between arms. Latency is quoted at the ninety-fifth percentile rather than as an average, because a queue backs up against its slow calls rather than its typical ones and an average hides exactly the tail that causes a timeout. Forty microseconds against most of a second is a ratio of roughly twenty thousand to one, and whether that ratio matters is a property of where the classifier sits rather than of the classifier: it is invisible behind a message queue that already tolerates minutes, and decisive in front of somebody waiting for a form to submit.

Everything in that table came from one evaluation harness run three times with a different predict function. The harness is the reusable artifact, and it should take a callable and a labelled set and know nothing about which arm it is scoring, because the moment the LLM arm has its own bespoke evaluation path the arms stop being comparable.

def evaluate(predict, dataset):
    y_true = [row["label"] for row in dataset]
    y_pred = [predict(row["text"]) for row in dataset]
    return classification_report(y_true, y_pred, labels=LABELS,
                                 zero_division=0, output_dict=True)

results = {name: evaluate(fn, EVAL_SET) for name, fn in ARMS.items()}

Holding the label policy constant is the part that quietly breaks. If the prompt says "when a complaint mentions both a late delivery and a broken item, choose the one the customer is asking you to fix" and the rule set resolves ties by keyword count, the two arms are answering different questions on the ambiguous cases, and any gap between them partly measures the policy mismatch rather than the systems. Write the tie-break rule once, implement it in both arms, and note in the results which rows it applied to.

The chain behind that is worth laying out, because the conclusion is stronger than the warning suggests. The evaluation set's labels were fixed by a person applying one tie-break rule. Each arm applies whatever tie-break its own implementation happens to encode. On an ambiguous complaint, the arm whose tie-break agrees with the labeller's is scored right and the other is scored wrong — and neither outcome says anything about how well either arm reads text. The measured gap on those rows is therefore the policy mismatch, and it is signed: it favours whichever arm was built by somebody who had read the policy. The prompt usually was, because the policy was pasted into it verbatim. The rule set usually was not, because it resolves ties by counting keyword hits. So the mismatch biases the comparison toward the model, on precisely the rows a reviewer will ask to see. Implementing one tie-break in both arms is not tidiness; it is the difference between measuring two systems and measuring which system was told the rules.

The results that keep the model, and the one that does not#

A clean per-class failure against a stated bar is the tidiest way for a baseline to lose, and there are three other shapes that also keep the model, worth naming because they come up more often than the clean one.

• The rule set reaches parity, but only at 380 lines, with a change every time marketing renames a product, so maintenance runs at a person-day a month and the failure mode is a silent drop in one class.
• The label set is scheduled to change. A taxonomy the policy team is about to split from three classes into seven makes a trained artifact obsolete on the day of the change, and a prompt edit is a smaller event than a relabelling campaign.
• The classes are separated by something a bag of words cannot see. Sarcasm, negation and conditionals — "I'm not complaining about the delivery, it arrived fine, but the unit is dead" — are where the published comparisons consistently favour the model; Adaptable and Reliable Text Classification using Large Language Models is a reasonable entry point into where zero- and few-shot classification genuinely outperforms a trained classifier.

Equally, the result can go the other way, and the outcome should be allowed to be boring. Plenty of routing tasks are keyword-separable because the humans writing the tickets are choosing from a template in their heads, and a rule set that lands within a point of the model on every class is a rule set that ships.

What the harness keeps doing after launch#

A baseline is not error analysis, though it hands you the cheapest version of it available. The inputs where the rule set and the model disagree are a pre-filtered queue of hard cases, typically 10–15% of the set, and reading fifty of them tells you more about the task than another accuracy point. Where both arms agree and both are wrong, you have almost certainly found a label policy problem rather than a model problem.

The reason that queue is worth reading is a property of disagreement rather than of either system. Wherever two arms built on different principles return different labels, at least one of them is wrong, so the error rate inside the disagreement set is far above the error rate of the set as a whole. Filtering to it costs nothing — it is one comparison per row on results you already have — and it concentrates the mistakes into a tenth of the reading. The corollary is the sentence above it: agreement is not evidence of correctness, because two arms can agree by both being misled by the same wording, and when they do the fault is almost always in what the policy says a label means.

Another property gets assumed rather than checked, and it is worth narrowing before somebody leans on it. Turning sampling all the way down makes the model take its highest-scoring token at every step, which removes the run-to-run variation a sampler introduces and is genuinely what lets an evaluation be rerun on an afternoon and compared. It does not make the arm a fixed function. The weights behind a hosted model name change when the provider updates them, and nothing in the response announces that it happened, so the arm that scored 93.0% in one quarter is not guaranteed to be the same system a quarter later. A rule set has no such property and neither does a trained artifact sitting in a bucket. That asymmetry is not a reason to distrust the number — it is the reason the exact model version belongs in the results table beside the result, and the reason a six-month-old score is a claim about a system that may no longer be the one in production.

A baseline is also not a one-time gate. The arms drift apart as traffic changes, which means the harness has to keep running after launch, on a refreshed sample, with every arm still in it. A rule set that has not been executed in nine months is not a baseline; it is a file. Teams that skip the comparison entirely are usually not avoiding work but avoiding a number that might be uncomfortable, and the discomfort arrives later with interest when somebody else runs it.

None of this obliges anyone to ship the winner. A baseline that loses on every metric has still earned its hour, because the sentence it produces — worth this many points, over this much code, at this much cost — is the one a reviewer six months from now can check. The sentence a comparison without a baseline produces is that the model scored 93%, which is consistent with the task being trivial, and nothing in the document distinguishes the two.

The regex arm costs an hour and gives you a floor. The arm that actually competes on quality — a supervised classifier fitted on a few thousand labels — is a different kind of object, with a training run, a stored artifact and a retraining schedule, and you cannot credibly argue for or against it in a design review until you have built one yourself. That is the next chapter.