Senior · 9 chapters
Multilingual prompting and behavior
Quality degradation in lower-resource languages, evaluating per language rather than in aggregate
Aggregate quality numbers routinely ship broken locales, so multilingual behavior needs its own evidence and release gates.
Read these first
What you will know
Evaluate a prompt separately per language rather than in aggregate
A strong mean can hide a catastrophic tail, especially when traffic is uneven or language labels are missing.
Show quality degrading in a lower-resource language
Translation artifacts, script variation, and weaker pretraining data can turn a valid instruction into omissions or unsafe answers.
Decide whether to translate in, translate out, or prompt natively
The choice changes terminology fidelity, politeness, refusal behavior, and the prompt-injection surface.
Reproduce a language-specific failure with a fixed test set and trace whether it comes from instruction following, retrieval, translation, or the model
Without isolating the failing stage, teams often patch prompts for what is actually a corpus, tokenizer, or routing problem.
Budget for per-language eval sets
Native-speaker annotation and maintenance become recurring release costs rather than a one-time localization task.
Chapters
- 1
Aggregating quality across locales
A traffic-weighted mean bounds every locale's influence by its weight, so Vietnamese at 4% of traffic can fall from 85% to 20% and move an 88.1% headline by 2.6 points — inside the range the metric wanders in anyway. Work the arithmetic, see why no threshold on the aggregate recovers it, and replace the single number with a rate and a sample size per language.
- 2
Deriving the language label
The label on each request usually comes from the browser's Accept-Language header, which records an interface preference rather than the language of the text the user wrote. In one product that divergence ran at 11% of requests and made Portuguese read 91% when the conversations actually in Portuguese ran at 74.1%. Derive the label from content instead, let the resolver abstain, and validate it against human labels before routing on it.
- 3
Confidence intervals and per-language gates
A per-language dashboard is half the change; the other half is a gate that can actually block a launch. On 80 cases a 95% Wilson interval spans fifteen points, so an 85% floor decides releases by coin flip. Size the gate to the regression you mean to catch, compare paired cases rather than rates, and keep the absolute floor away from the deploy path.
- 4
Structural compliance in low-resource languages
A five-section report prompt achieves 99% compliance in English and 61% in Swahili, while native reviewers rate the Swahili prose natural 94% of the time. Fluency is a pretraining property and instruction following is a post-training one, so they thin out at different rates — and the tokenizer charges Swahili 1.92× the tokens, truncating the sections at the end. Measure structure deterministically, because the fluent surface fools every judge you have.
- 5
Pivot translation and native prompting
Pivoting a request through English preserves a tuned pipeline and pays for it in normalised product and legal terms, a politeness register chosen by the translator, and two extra stages that each need their own evidence. Prompting natively removes the pivot and promotes cross-lingual retrieval into the hard problem when the corpus is English. The middle arrangement — translate inbound, generate the answer natively — removes about 80% of the translation bill and is usually the one to ship first.
- 6
Refusal rates and the injection surface
Guardrail training is thinnest where pretraining coverage is thinnest, so the same request can be answered in English and declined in Thai, or the reverse: 3.8% unsafe-answer and 7.9% over-refusal in Thai against 0.7% and 2.1% in English. A translation pivot then rewrites clumsy in-language injection attempts into well-formed English imperatives, taking compliance from 3.3% to 14.2% on the same attack set. Both rates need native-authored evidence per language.
- 7
Cross-lingual retrieval
When the query is German and the corpus is English, recall@10 falls from 0.91 to 0.58 and the assistant says it could not find the information three times as often — on questions the corpus does answer. Run retrieval alone against gold document ids before touching the prompt, because softening the grounding instruction converts a countable failure into a silent wrong answer.
- 8
Tracing a pivoted answer
A detect, translate, retrieve, generate, translate stack told a Japanese customer they had thirty days to request a refund when the tier's real window is fourteen. Judging each stage against its own input — not against the customer's experience — puts exactly one row in the no column: inbound translation flattened a product tier into a generic adjective. Which is also why no line added to the English prompt can recover it.
- 9
Machine-translated eval sets
Running an English eval set through a translation API is a competent launch smoke test and a bad quality measurement: on topic-matched cases the machine-translated questions score 88% against 71% for native-authored ones. It also produces no baseline, so no paired regression gate can exist. Prices what native-authored per-language evaluation costs, and how to tier twelve locales when you can fund four.
External resources
- MMLU-ProX: A Multilingual Benchmark for Advanced LLM Evaluationarxiv.org
Quantifies the accuracy cliff from English to lower-resource languages.
- Why LLM Performance Drops in Non-English Languageslilt.com
This is a substantive research-driven explainer detailing multilingual failure mechanisms such as tokenizer inefficiency, translated constraints, coreference errors, cultural mismatch, and benchmark design issues, with concrete examples relevant to multilingual prompting behavior.
- Lower-Resource, Higher Scores: Language Bias in LLM Evaluatorsarxiv.org
Why LLM-judge scores are inflated in low-resource languages.
- Multilingual Prompting for Improving LLM Generation Diversityarxiv.org
Evidence on English vs native-language prompts per resource level.
- MMLU-ProX project pagemmluprox.github.io
Per-language result tables you can read before picking a model.
- Adapting LLMs to low resource languagesyoutube.com
Specific low-resource language adaptation mechanics