Choosing an enum over free text

An enum is a schema keyword declaring the complete list of values a field may hold. The alternative that keeps appearing in review is a plain string whose description lists the same values in prose, and the argument for it is not silly: the model follows the description almost all the time, the enum has to be kept in sync with the union type on the consumer side, and nobody has seen a bad value in the fixtures.

That last observation is the one to take apart first, because it is real evidence being read backwards. At the rate measured below, roughly one response in seventy carries an off-vocabulary value. A fixture set of twenty cases will contain one about a quarter of the time; a fixture set of fifty is close to a coin flip. So "nobody has seen a bad value in the fixtures" is exactly what a one-in-seventy failure rate looks like from a fixture set, and it is also exactly what a rate of zero looks like. The observation cannot separate those two, which is why the argument needs a count over traffic rather than a show of hands.

The argument as a whole is answerable with two numbers rather than an opinion. The first is the rate at which a described vocabulary is violated, which is a property of decode-time mechanics and can be measured on any traffic you already have. The second is the rate at which your vocabulary fails to cover the input at all, which the enum does not fix and which decides how much damage the enum does quietly. This chapter is about both, in that order, because a migration approved on the first without the second is approved on a claim it does not support.

At decode time these are two different mechanisms#

The difference is not one of emphasis. With category: string, the value position is constrained only by JSON string syntax — anything that forms a legal string is a legal continuation, and the space of legal continuations is effectively the whole vocabulary at every step. The description shifts the probability mass towards the six words. It shifts it a long way, and it leaves the rest of the space reachable.

With an enum, the six members compile into a finite automaton over token sequences — a small state machine that knows, at every position, exactly which tokens could still lead to one of the six and which could not — and the machinery of topic 5.2, on constrained decoding, applies the automaton as a mask on the logits at each step. Tokens that cannot begin or continue a legal member have their probability set to zero before sampling, so Billing with a capital B is not unlikely, it is unreachable.

At S0 the mask permits exactly three first tokens. Once b has been emitted the mask permits two continuations. By the third step the string is determined. A capital B never had a nonzero probability at any point, so a capitalisation failure cannot occur, and neither can a whole sentence, because there is no path through the automaton that reaches a space character.

Two details in that machine are worth making explicit, because the guarantee rests entirely on them. The first is that its alphabet is tokens rather than letters. The six members were compiled against this model's tokenizer, so every edge carries a token id and a token covering several characters at once takes a single edge. That relabelling is what lets the mask be applied straight to the logit vector with no character-level bookkeeping at decode time. The second is that reachability is decided by the member list and by nothing else. A capital B as a first token has no outgoing edge from S0, because none of the six members begins with a capital, so Billing is not an unlikely answer — it is not an answer. The same argument disposes of the whole sentence: a space appears in none of the six members, so no state has an edge labelled with a token containing one, and no path runs from S0 to any string with a space in it. Every guarantee here is a claim about paths through a graph you could draw by hand from the enum in a couple of minutes.

The size of the gap is measurable on traffic you have already paid for. One team holding a category field as a plain string with the description One of: billing, account, bug, feature_request, abuse, other checked 3,000 production calls from the previous fortnight. Forty-one of them, 1.4 percent, held something that was not one of the six: Billing with a capital B (nineteen), Billing / Payments (seven), billing_and_payments (four), Refund (six), bug_report (three), and two responses where the value was the sentence The category is billing. A description has a failure rate for the class "value outside the vocabulary", and an enum has none. The measured rate will vary with the model, the clarity of the words, and how often the input is genuinely ambiguous, but the shape of the comparison does not vary. The guide to structured outputs and function calling with LLMs works the same distinction through concrete JSON Schema and Pydantic definitions if you want the provider-specific mechanics.

Those 41 responses had raised nothing. The consumer was a switch with a default branch that mapped anything unrecognised to unknown, so the only trace was an ops dashboard whose category totals were 41 short of the call count, which nobody had ever reconciled.

The six counts add back to 41, and checking that is worth the ten seconds because it says the failures were sorted rather than sampled: nineteen, seven, four, six, three and two, with nothing left over in a residual bucket. The shape of the list is as informative as the total. Every one of those values is a near miss — a capitalisation, a punctuated pairing of two words, a compound built from the right word, a synonym, one different word, and in two cases the right answer wrapped in a sentence. Not one of them is a category from some other taxonomy or an invented concept. That is what a described vocabulary looks like when it fails: the model is aiming squarely at the set and landing beside it, which is why the failures are trivial to repair by hand and impossible to prevent by asking more firmly.

Measuring the switch honestly#

Replacing a free-text field with an enum and measuring the improvement is a two-line change and a three-number measurement, and the measurement people usually run captures one of the three. Run the same 500 inputs through both schemas, same model, same prompt, and count all of them.

The first is legality: how many outputs are exactly one of the allowed values, with no normalisation. This is where the enum wins by construction, and it is the number that gets reported.

The second is accuracy against labels. Hand-label the 500 inputs with the correct category, and compute per-class precision and recall for both schemas, precision being the share of responses assigned to a class that genuinely belong to it. On one subscriptions team's cancellation data this was where the enum's win got smaller: overall accuracy rose from 79 percent to 84 percent, driven almost entirely by the capitalisation and formatting cases, while precision on too_expensive fell from 0.88 to 0.71 because it had become the sink.

Reporting the first number without the second is how a change like this gets approved on a claim it does not support. A jump from 98.6 to 100 percent legality reads as a fix, and on that data it coexisted with one member's precision dropping by seventeen points, because legality asks whether the value is in the set and accuracy asks whether it is the right member of the set. Those come apart precisely when the vocabulary is the thing at fault, so they have to be reported together or the migration looks free.

The mechanism behind that coupling settles the equivalence question the pull request opened with. An input the vocabulary does not cover has no right answer in the set. Under free text the model writes something outside the set, which fails the legality check and shows up as an odd string somebody reads. Under an enum the same input yields a legal member, because a legal member is the only thing the grammar can emit, and it fails the accuracy check instead. The error was neither removed nor created. It moved from a column that is computed automatically on every response into a column that can only be computed against hand labels. A migration reported on legality alone is reporting the half of that transfer that looks like a win, and the two numbers are not two views of one improvement — they are the two ends of the same move.

The third is the escape rate: what fraction of the labelled set your vocabulary does not actually cover. This is the number that predicts how much silent damage the enum will do, and it cannot be computed from the model's outputs at all.

# The three numbers, computed against a hand-labelled set.
legal = sum(1 for r in runs if r.value in ALLOWED)                 # 500/500 with an enum
accurate = sum(1 for r in runs if r.value == r.gold_label)         # 420/500
uncovered = sum(1 for r in runs if r.gold_label == "NOT_IN_VOCAB") # 46/500  <-- the one that matters

Each of those three lines stands in a different relationship to its denominator, and that is what keeps them from substituting for one another. Legality is counted against the schema, so an enum sends it to 500 out of 500 by construction and the figure carries no information about the model at all. Accuracy is counted against a human label, and 420 out of 500 is the 84 percent quoted above — it cannot be computed unless somebody has labelled the inputs, which is the step that gets skipped. Coverage is counted against the vocabulary rather than against the model: it asks how many of the 500 inputs have no correct answer available anywhere in the set, and nothing the model emits can move it up or down.

Forty-six out of 500 is 9.2 percent. Those cases have to go somewhere, and with eight members and no eighth-and-a-half, they go into whichever member reads closest.

Where the description is the right call anyway#

The author of that pull request is right in more situations than the argument above suggests, and it is worth being able to name them, because "always use an enum" is a rule that will eventually cost you an outage.

The third row has a number attached that surprises people. Grammar compilation is per unique schema, and providers cache it, so a stable schema pays it once. A schema whose enum is rebuilt from a database on every request is a new schema every time; teams that have measured this report first-use compilation in the low hundreds of milliseconds for enums in the low thousands of members, on top of losing the prompt-cache discount on the whole schema block. For a 400-SKU enum that changes per tenant, the right shape is usually a free-text field plus a fuzzy match against the SKU list in code, where a near-miss is visible and correctable.

The enum is the right default, and the reason is narrow and worth keeping narrow: it removes one class of failure, the value outside the vocabulary, completely rather than mostly. It does not make the model better at choosing, it does not tell you whether the chosen member is right, and it converts a loud failure into a silent one wherever your list is short of the world. Silent failures accumulate at whatever rate your vocabulary is incomplete, and that rate is not observable from the responses.

The response to that is neither to abandon the enum nor to keep adding members until the tail is empty, because the tail never empties. It is to design the vocabulary so that the uncovered cases land somewhere countable — which requires first noticing that a field like reason was carrying more than one kind of distinction to begin with.