Deriving the language label

A language label is the field a pipeline stores on every request to say what language that conversation is in. It is the column a per-language breakdown groups by, and in most stacks it is also the switch that selects which system prompt gets assembled, sometimes which corpus partition is searched, and occasionally which model is called. It is therefore load-bearing twice, and a wrong value does not produce a missing number — it produces a confident wrong one, filed in a bucket somebody trusts.

Nearly every product populates that field on the first pass from Accept-Language, the HTTP request header a browser sends to say which translations of a page it would prefer. That header is the wrong source, and the rest of this chapter is about how wrong, in which direction, and what to replace it with. The replacement is a resolver: a small piece of request-time logic that reads the user's text, runs a language-identification classifier over it, applies a confidence floor, and is allowed to answer "unknown".

The header was designed to pick a document, not to describe a user#

Accept-Language comes out of HTTP content negotiation, and its job in the original design was to help a server choose between translations of a resource it already had — send the French copy of the manual to a client that prefers French. The value is populated from the operating system or browser locale, which the user set when they installed the machine, and it describes which interface they want back. It was never a claim about what language the person is going to type into a text box, and there is no field in HTTP that makes that claim, because HTTP has no idea what a text box is.

POST /v1/assistant/messages HTTP/1.1
Host: api.example.com
Accept-Language: en-US,en;q=0.9,pt-BR;q=0.8
Content-Type: application/json

{"conversation_id":"c_8812","text":"Oi, comprei o plano anual em janeiro e
quero cancelar. Consigo reembolso proporcional?"}

The header says English. The body is Portuguese. A pipeline that reads the header stores language = "en-US" for that request, so any failure that follows is counted against English, where it joins several hundred other cases and moves nothing.

That mislabelling does two things rather than one, which is why its effect is larger than the case count suggests. It removes a case from the bucket where it belonged, so Portuguese is measured on a sample that no longer contains it. And it adds that case to a bucket where it does not belong, so English is measured on a sample containing conversations English was never asked to handle. A missing label costs one observation. A wrong label costs one observation and buys a false one, in two different rows, and the two errors point the same way.

There is a short way to say what the header actually measures. Accept-Language is populated from the operating system's locale, and the operating system's locale was set by whoever provisioned the machine. On a consumer laptop that is usually the user, and the header is a decent proxy for them. On a corporate fleet it is an IT department in another country, and the header is a fact about procurement. Nothing anywhere in that chain consults the person typing, which is why no amount of care in reading the header recovers the language of the text.

In practice the header diverges from typed language in both directions, and both run in the same product at once. A large share of Brazilian professional users work on machines installed in English — corporate images, English-language Windows, phones bought with a default locale nobody changed — and they type Portuguese. Meanwhile a smaller group with pt-BR at the top of the header types English, because they are engineers pasting stack traces, or because they have learned that the assistant is better in English. Splitting the buckets by the detected language of the user's own text is what a dashboard should show:

The Portuguese row is propped up by fifty-one English conversations that the model handles almost perfectly, and the forty-four hardest Portuguese conversations have been filed under English, where they are 6% of a large bucket. Pool the 126 cases whose text is actually Portuguese and the real rate is 74.1%, seventeen points below what the dashboard reported and the worst locale in the product by a wide margin. The 91% was not a rounding error or a sampling artifact; it was a number computed correctly over a set of conversations that had little to do with Portuguese.

Both dashboard rows reconcile, and that is what makes them dangerous rather than merely wrong. The pt-BR bucket holds 82 Portuguese conversations at 86% and 51 English ones at 99%; weight each rate by its own case count, add the two products, divide by the cases involved, and 91% comes back out. The en-US bucket holds 716 English conversations at 93% and 44 Portuguese ones at 52%, and the same operation returns 91% again — the 44 are diluted by a bucket sixteen times their size, which is what being 6% of a large bucket means in practice.

The pooled Portuguese figure is that same operation applied to the cases that belong together: 82 at 86% and 44 at 52%, over 126 cases, which gives 74.1%. The step worth being careful about is the one people take instead, which is averaging the two rates directly. That returns an answer several points lower, and it is wrong because the two groups are not the same size — the 82 deserve nearly twice the say the 44 do. A rate is one count divided by another, and two rates can only be combined by going back to the counts they came from.

The label does not only measure, it routes#

If the label were used for reporting alone, this would be a measurement bug with a measurement fix. In most stacks the same field decides what the model is asked to do:

PROMPTS = {
    "en": "system/support_en.md",   # ends: "Always answer in English."
    "pt": "system/support_pt.md",   # ends: "Responda sempre em português do Brasil."
    "ja": "system/support_ja.md",
}

def build_request(req):
    lang = req.headers.get("Accept-Language", "en").split(",")[0][:2]
    return {
        "system": load(PROMPTS.get(lang, PROMPTS["en"])),
        "messages": [{"role": "user", "content": req.body["text"]}],
    }

Feed the earlier request through this and the Portuguese question about a plano anual arrives under a system prompt whose last line instructs the model to always answer in English. Topic 4.1, on how a system prompt's trailing instructions carry disproportionate weight, explains why that placement matters; here the weight is being spent on an instruction that directly contradicts the user's evident language. Some of the time the model follows the user and answers in Portuguese anyway, some of the time it answers in English, and some of the time it produces the third outcome that generates support tickets, which is a Portuguese-shaped answer to an English-shaped reading of the question — plano anual treated as an HR term because the surrounding instructions have primed an English support context with no product glossary for Brazilian plan names. The header is not merely a bad label. It is an input that actively degrades the requests it mislabels, which is why the fix belongs at request time rather than in the analytics job.

Deriving the label from the text the user wrote#

The signal that actually answers the question "what language is this conversation in" is the conversation. A per-turn detector over the user's message is cheap — fastText's lid.176, an off-the-shelf language-identification classifier that takes a string and returns a language code with a confidence score, covers 176 languages, loads in about 130 MB of RAM, and classifies a short string in well under a millisecond on one CPU core, which is nothing next to a model call. What it is not is reliable on every input, and the way it fails matters more than its headline accuracy.

Detection accuracy is a function of string length. On strings of twenty or more tokens, lid.176 sits in the mid-nineties for the languages a consumer product actually sees. On strings under ten characters it degrades sharply, and support traffic is full of those: "obrigado", "ok thanks", "PT-4410?", a bare order number. Certain pairs stay confusable at any length because they genuinely overlap — Portuguese against Galician, Spanish against Catalan, Indonesian against Malay, Croatian against Serbian in Latin script, Norwegian Bokmål against Danish. A detector that returns gl with 0.51 confidence for a Portuguese sentence is not broken; it is reporting real ambiguity, and the design question is what you do with it.

The dependence on length is not a quirk of one classifier. A language-identification model scores a string by the character sequences it contains, and the confidence it reports is the weight of evidence for the leading language set against the weight for every other. A long sentence supplies many sequences, most of them shared with only a handful of languages, so the ratio comes out sharply. A four-character string supplies a few sequences, and a few short sequences are shared with dozens of languages, so the ratio sits close to even and whichever side it falls on is close to arbitrary. The confusable pairs are the same effect arriving from the other direction: Portuguese and Galician share so much of the available evidence that adding length does not separate them, because the extra sequences are shared too.

The answer that keeps the dashboard honest is to let the resolver abstain. A label of unknown is a row you can go and look at, while a wrong label is a row that silently corrupts a bucket you trust.

Three properties of that resolver are worth naming. It is sticky at the conversation level, so a long Portuguese thread does not flip to unknown when the user sends "ok" as turn nine, and stickiness also means the label is stable enough to route on without the system prompt changing under the user mid-conversation. It uses Accept-Language as a tiebreaker rather than throwing it away, because for a genuinely ambiguous string the interface preference is a real prior and better than a coin flip. And it produces an explicit abstention that flows into the unknown bucket a per-language breakdown otherwise leaves open, which turns "we cannot tell" from an invisible failure into a countable one; when that bucket is 6% of traffic and its success rate is fifteen points below everything else, you have found a population, not a data-quality nit.

Two further columns pay for themselves immediately. Record the detected language of the model's reply alongside the user's, because answering a Portuguese question in English is a distinct failure that a task-success rubric often scores as a pass, and it is invisible unless you measure it. And record the raw header next to the derived label rather than overwriting it, so the divergence rate between the two is itself a metric.

Checking that the new label is any better#

Replacing one automatic label with another automatic label is not obviously progress, and the claim that detection is more accurate than the header deserves evidence rather than assertion. The measurement is small: pull 300 conversations stratified across the buckets, meaning each bucket contributes a fixed share of the sample regardless of how much traffic it carries, give a native or fluent speaker the user's first two turns with no metadata attached, and ask them what language it is. Agreement between the human labels and each candidate source is then a number you can put next to the change.

The per-language breakdown of the disagreements is the useful part rather than the headline agreement rate. A residue concentrated in Portuguese-Galician and Indonesian-Malay confusions plus romanised Hindi is a remediation list; "the detector is 96% accurate" is not. A confusion matrix, which is the grid counting how often each human label was paired with each machine label, over 300 cases has thin cells and will not support fine claims, and it is enough to tell you whether your errors are spread evenly or piled into two language pairs you could special-case.

Thin is worth quantifying, because it decides which claims the grid supports. Nine human labels crossed against nine machine labels gives more than eighty cells, and 300 cases spread over them average fewer than four apiece — with most of the mass sitting on the diagonal, so the off-diagonal cells that carry the interesting information hold ones and twos. A cell holding two cases cannot distinguish a small confusion rate from a slightly larger one. It can distinguish a pair that turns up repeatedly from a pair that never turns up at all, and that is the only question worth putting to it.

The other check costs nothing because the conversation text is already stored. Re-derive labels for the last twelve weeks of eval results and recompute the per-language history, which gives you a corrected baseline rather than a discontinuity — without it, the Portuguese line on the dashboard drops seventeen points on the day the change ships and every subsequent conversation about Portuguese begins by explaining that the drop was not a regression. Backfilling makes the series comparable and turns the label change into a one-off correction with a date on it.

What the backfill defeats is that a time series has no way to represent a change of definition. Every point in it is a number, and a reader compares this week's with last week's on the assumption that both were computed the same way. Change the definition without recomputing the history and the series shows a step, and a step is precisely what a regression looks like — so the chart is now making a claim the data does not support, and it goes on making it every time anybody scrolls back. Recomputing the history hides nothing. It moves the correction from inside the series, where it is indistinguishable from a quality event, to outside it, where it is an annotation with a date attached.

Language is not locale, and neither is script#

A detector that says pt cannot tell you whether the user is in São Paulo or Lisbon, and the difference decides which product names, tax rules, price formats and second-person pronouns are correct; você against tu is not a stylistic nicety in a support answer that quotes statutory rights. Region has to come from the account, the billing country, or the request path, and it belongs in a separate column from the detected language.

Script is a third axis. Chinese arrives as zh-Hans or zh-Hant, Serbian in Cyrillic or Latin, and users routinely write languages in scripts the model saw far less of during pretraining — romanised Hindi, romanised Arabic, Greeklish. A detector will usually resolve these to something, and often to the wrong thing, because romanised Hindi looks statistically closer to Indonesian than to Devanagari Hindi. If the product has a meaningful share of such traffic, script belongs in the label too, since the model's behaviour differs sharply between the two forms of the same language.

Code-switching, which is moving between two languages inside a single message, resists all of this by design. A single message that opens in Portuguese, quotes an English error string, and closes in Portuguese has no single correct label. Per-turn detection with a confidence floor handles it acceptably by picking the dominant language, and the residue lands in unknown, which is the right place for it.

None of this makes any answer better. Correct labels move failures into the buckets they belong to and stop the pipeline routing a Portuguese question through an English system prompt, and both of those were real defects, but the 74.1% that Portuguese was actually running at is still 74.1% after the fix — now visible, and now attached to a locale small enough that a traffic-weighted aggregate will never react to it.

That sentence needs its condition attached, because it reads as more pessimistic than it is. The 74.1% was measured on conversations already served under the old routing, so it is a fact about traffic that is over, and no relabelling improves a reply that has been sent. What the resolver changes is where conversations go from here — and some part of that seventeen-point gap was the English system prompt rather than the model's Portuguese, since a plano anual answered as annual leave is a routing failure and not a coverage one. How much of the gap the routing fix closes is its own measurement, taken on the same set after the change. What it will not do is close all of it, and whatever is left is a multilingual-quality problem that no amount of labelling work touches.

Making a release react to that residue means a gate, and a gate on 126 cases has to survive the objection that small locales are too noisy to gate on.