Structural compliance in low-resource languages

Structural compliance is the share of answers that contain every part the prompt asked for — five named sections, three bullets, a citation marker, a word limit. It is a separate quantity from whether the prose is any good, and in English the distinction rarely earns its keep, because a model that writes well in English also does as it is told in English. As pretraining coverage thins, the two come apart. The prose stays convincing and the compliance collapses.

The direction of that split is what makes it dangerous rather than merely annoying. The surface of the output is the last thing to break and the control over the output is the first, so an answer that is fluent, well-organised in appearance, and missing 40% of what was asked for passes a casual review in a way that a garbled answer never would. Human reviewers read the surface, and so does an LLM judge — a second model prompted to read the output and score it.

Two capabilities with different training provenance#

Fluency in a language is a pretraining property, pretraining being the long first pass over raw text in which the model learns only to predict what comes next. A model that saw a great deal of Swahili text — news, Wikipedia, forums, translated fiction — learned the distribution of it, which is enough to produce sentences a native speaker will accept.

Following a multi-part instruction is largely a post-training property, acquired in the much smaller curated stage that comes afterwards: supervised fine-tuning on written demonstrations of good answers, and preference data in which annotators rank one candidate answer above another. Those datasets are overwhelmingly English with a long tail of high-resource languages. Whatever instruction-following ability exists in Swahili got there by cross-lingual transfer from an English-trained capability, and transfer is partial in a way that is measurable: it carries the general disposition to be helpful and to answer in a structured-looking way, and it carries less of the precise behaviour of holding five named constraints in play for six hundred words. "MMLU-ProX: A Multilingual Benchmark for Advanced LLM Evaluation" is a useful reference point for how steep the underlying cliff is, since it holds the questions constant across languages and reports per-language accuracy; the gap between the top-resource and bottom-resource languages there is large enough that treating a model as uniformly multilingual is not a defensible modelling assumption.

Putting those two paragraphs together gives a rule that predicts new cases rather than describing this one. A capability's availability in a language tracks the stage of training it was learned in. Pretraining is enormous and reaches a long tail of languages, so what is learned there — the shape of a sentence, the register of an internal memo, idiom — is available wherever that tail reaches. Post-training is small, curated and expensive to produce, so it exists in the languages somebody paid annotators to write, and everywhere else it arrives only by transfer. That ordering is especially inconvenient for evaluation, because the capability that survives is the one every reviewer can see and the capability that does not is the one nothing in an ordinary review looks at.

The tokenizer takes a second bite#

The other half of the effect is mechanical and has nothing to do with what the model knows. The tokenizer is the fixed vocabulary that chops text into the units a model reads, emits and charges for, and the byte-pair vocabularies in common use are fit to a training mixture dominated by English and a handful of European languages, so a language whose morphology or script is poorly covered gets shredded into many more tokens per unit of meaning. Measured over the same 1,200-document parallel corpus with the model's own tokenizer:

The right-hand column is the one that explains short answers. A service that sets max_tokens — the hard ceiling on how many tokens a single response may contain before generation is cut off — to 800 because that was comfortable for the English report is giving Swahili roughly 250 words, which is not a five-section incident report. In the run measured below, 22% of Swahili responses hit that cap against 4% of English responses, and a response that hits the cap loses whatever was at the end, which is the recommendations and the next steps.

The middle column is one division: 480 tokens per thousand characters set against English's 250 is a little under twice, which is the 1.92. The right-hand column takes one further step and needs a fact the table does not state. Eight hundred tokens at 480 per thousand characters buys a bit under seventeen hundred characters, and turning characters into words requires an average word length, which differs by language. That is why the word column falls faster than the token ratio: Swahili loses more than 1.92 times the words, which is the table telling you Swahili also packs more characters into each word. The token ratio is a floor on the damage rather than the whole of it.

The same relationship shows up from the other end in the compliance table further down. The mean output tokens for the two languages sit within a few per cent of each other while the mean output words differ by roughly a factor of two. Nothing changed about how much the model chose to write. What changed is how much meaning a token carries, and the factor between the two word counts is the tokenizer ratio arriving on the invoice.

Truncation does not account for all of it, because plenty of incomplete answers finish well under the cap with a natural closing sentence. Both mechanisms push in the same direction and on the same sections. Constraints stated once at the top of a prompt have to survive the whole generation, and the further into a long sequence the model is, the more the local coherence of what it has already written dominates the instruction it was given six hundred tokens ago — a sequence that is twice as long in tokens for the same content gives that drift twice the distance to act over.

One sentence of mechanism sits under that. The model produces each token by attending over everything written before it, and the weight it puts on any one region is set against every other region in the sequence. A constraint stated once at the top is a fixed quantity of text; the answer growing in front of it is not. So the constraint's share of what is being attended to falls as the answer lengthens, and it falls twice as fast in a language whose tokenizer spends twice as many tokens on the same content. Nothing was forgotten. The denominator grew faster.

Measuring structure without asking a model#

The check has to be deterministic, because the point is to detect a failure an LLM judge is unusually bad at spotting. A judge asked "does this answer follow the instructions?" reads fluent, plausible Swahili and tends to say yes; the evidence in "Lower-Resource, Higher Scores: Language Bias in LLM Evaluators" is that judge scores skew upward in exactly the languages where quality is lowest, which makes the judge worse than useless as the gate on the thing that is actually breaking.

There is a sharper reason to keep a judge out of this, and it is not about bias. A judge is a model, and it carries the same provenance asymmetry as the model it is judging: its fluency in Swahili came out of pretraining, and its ability to hold a five-part checklist against a document came out of post-training, so the second thins in Swahili for precisely the reason the first does not. The instrument and the fault are made of the same material. A parser is not — a regular expression either matches or it does not, and its accuracy in Swahili is a property of the alias list somebody wrote rather than of anything a model learned.

Structure is checkable with a parser:

import re

SECTIONS = {
    "summary":         [r"summary",         r"muhtasari"],
    "findings":        [r"findings",        r"matokeo"],
    "risks":           [r"risks?",          r"hatari"],
    "recommendations": [r"recommendations", r"mapendekezo"],
    "next_steps":      [r"next steps",      r"hatua zinazofuata", r"hatua zifuatazo"],
}

HEADING = re.compile(r"^\s*(?:#{1,4}\s*|\*\*)?([^\n*#:]{2,60})", re.M)

def sections_present(text: str) -> dict[str, bool]:
    heads = [h.strip().lower() for h in HEADING.findall(text)]
    return {
        key: any(re.search(p, h) for h in heads for p in pats)
        for key, pats in SECTIONS.items()
    }

def compliance(text: str) -> bool:
    return all(sections_present(text).values())

The alias lists are the part that gets missed, and missing them turns the metric into a different bug. Any assertion written against English output — a heading string, a JSON key, a date format, a "Yes"/"No" answer, a citation marker like [1] — becomes a language-dependent assertion the moment you run the same eval in another locale, and it fails in the direction that makes the locale look worse than it is.

The one-directional part is what makes it dangerous rather than merely irritating. An assertion written against English strings can only ever mark non-English output wrong. It has no mechanism for marking a bad answer right by accident, so its errors never cancel — they accumulate, all pointing the same way, against the locale you already have the least evidence about. A metric whose errors are symmetric is noisy and averages out. A metric whose errors all point one way is a false finding waiting to be cited.

The counts worth recording per language are the ones that separate the mechanisms, since they lead to different fixes:

The last row is what makes this failure survive review. Prose quality is nearly intact, so every human signal you have says the locale is fine.

Not every constraint degrades at the same rate#

Section headings are one kind of instruction and they are not the most fragile kind. Running six single-constraint variants of the same task over 200 cases per language separates them, and the ordering is stable enough across models to be worth designing around:

Constraints that the output format makes self-evident hold up best, which is why answering in the right language survives almost intact — every token the model emits reinforces the constraint, so there is nothing to forget. Constraints requiring a running count, a budget, or suppression of a topic degrade fastest, because they demand that the model hold state about what it has already produced across a sequence roughly twice as long in tokens as the English equivalent. Negative constraints are the worst value for money in a low-resource language: "do not mention pricing" needs the model to notice an omission it is making, and there is no partial credit — one sentence about price fails the case.

That ordering has a shape once you ask what evidence of compliance is available in the text written so far. Answering in the user's language is self-evidencing: every token already on the page is evidence for continuing the same way, so the constraint is re-established at every step and there is nothing to hold in mind. Producing exactly three bullets is checkable from the text but not self-evident, because the model has to count what it has written, and counting is a computation over the context rather than a continuation of it. Not mentioning pricing is the worst of the three because compliance leaves no trace at all — an absence cannot be attended to, so nothing in the context reminds the model the rule exists. Self-evident, countable, invisible. That is the order the table is in, and it is the order to expect for a constraint the table does not contain.

Position in the prompt matters more here than it does in English. Moving five section names from the top of the prompt to the final line before the user's question lifted Swahili all-five compliance from 61% to 69% and left English unchanged at 99%, which is the same recency effect that exists in English, showing up larger because the margin in Swahili is not already saturated. Prompt-engineering advice calibrated on English tends to describe effects too small to bother with; several of those effects are worth several points once the language stops carrying the model.

Saying it again does not add budget#

The intervention everyone tries first is to say it louder: repeat the five section names at the end of the prompt, add "you MUST include all five sections", append a compliance reminder. On this prompt that moved Swahili all-five compliance from 61% to 66%, inside the noise of a 200-case run. Inside the noise is a claim worth checking rather than asserting: at a rate near 61% measured over 200 cases, the band of true rates that could plausibly have produced the reading runs close to seven points either side of it, so a five-point move sits well within a range the same measurement would wander across on its own. The honest report of that experiment is not that emphasis helped a little. It is that the experiment could not have detected a change of that size in either direction. It is tempting because it appears to work in English, where compliance is already at 99% and any nudge seems to hold. It fails because the instruction was never misunderstood: the model is not confused about what five sections means, it is running out of budget and losing constraint adherence over distance, and neither of those responds to emphasis.

Section presence is also a floor rather than a definition of quality. An answer can carry all five headings and be wrong under every one of them, which is why structural compliance belongs next to a correctness rubric rather than instead of it. What it buys is a signal that is cheap, deterministic, and uncorrelated with fluency, and therefore able to fire in exactly the cases where every human and model judge in the loop is telling you things are fine.

None of this is about translation quality. No translation stage exists in the pipeline above: the prompt is Swahili, the answer is Swahili, and every failure described here is native-language behaviour under a tokenizer that charges Swahili nearly twice per character and a post-training set that barely contains it. Measure structure per language, with the headings aliased into that language, and you get a number that moves when the locale breaks even though nobody reading the output can tell.

That is native prompting, where the language of the request and the language of the answer are the same and the only question is what the model does with them. Most teams do not start there. They start by translating into English at the edge, running the pipeline they already trust, and translating back — and that arrangement moves the failures somewhere else entirely.