Tokeniser fragmentation is the overlooked cause of a broken locale

Everything a retrieval system does to a query happens after two transformations that almost nobody inspects. The text is normalised — lower-cased, stripped of some characters, recomposed — and then split into subword pieces from a fixed vocabulary. Both steps run before the encoder, before the index, and before any of the instruments in the previous chapters can see anything, and both of them behave differently for different languages while reporting success identically for all of them.

This chapter follows one failing locale through those two stages. The tokeniser is the component that turns a string into the integer pieces a model consumes, and fertility is the number of pieces it needs per word of running text — a quantity no model card publishes and that varies by more than a factor of two across the languages a single multilingual model claims to support. Fertility is where most of the damage in this chapter comes from, and it is not a coverage problem: every input tokenises into something, which is exactly why nothing errors.

Four transformations before the encoder sees anything#

A retrieval service typically normalises every query with three lines that were written in 2024 and have never been touched, because they work.

import unicodedata, unidecode

def normalise(q: str) -> str:
    q = q.lower()
    q = unidecode.unidecode(q)          # "helpful" diacritic stripping
    return unicodedata.normalize("NFC", q).strip()

Run a Turkish query through it and print code points rather than glyphs, because the glyphs will lie to you in a terminal. The query below was typed by a customer in capitals, and means "I want to cancel my invoices".

>>> q = "İPTAL ETMEK İSTİYORUM FATURALARIMI"
>>> q.lower()
'i̇ptal etmek i̇sti̇yorum faturalarimi'
>>> [hex(ord(c)) for c in q.lower()[:3]]
['0x69', '0x307', '0x70']          # i + COMBINING DOT ABOVE + p
>>> normalise(q)
'iptal etmek istiyorum faturalarimi'

Two separate things went wrong in three lines, and both are worth taking apart because both are design decisions rather than bugs.

Python's str.lower() implements the default Unicode casing rules. Those rules are a single table, defined once for all locales, and Unicode ships them that way deliberately — a library that guessed a locale from the text would be unpredictable, so the standard defines one default and offers locale tailoring as something you ask for.

Under that default, İ (U+0130, capital I with dot above) lowercases to i followed by U+0307 COMBINING DOT ABOVE, because the default algorithm has no way to know that in Turkish the dot is part of the letter's identity rather than an accent to be preserved separately. The same rules lowercase I to i when Turkish requires ı (U+0131, dotless i), so the last word came out as faturalarimi where the correct form is faturalarımı. Turkish is one of a small set of locales — Turkish, Azerbaijani, Lithuanian — where case mapping is locale-dependent, and a pipeline that calls .lower() without a locale silently corrupts every capitalised query from them.

unidecode then finished the job, and its failure has the same shape. Stripping diacritics is a habit inherited from Latin-script search, where folding café to cafe genuinely helps recall, because in French the accent is a mark on a letter and the unaccented form is a plausible typing of the same word. In Turkish the mark is the letter, and there it collapses distinctions that carry meaning: ş→s, ğ→g, ç→c, ı→i, ö→o, ü→u. That turns şifre (password) into sifre, değil (not) into degil, and kâr (profit) into kar (snow). The resulting strings are not Turkish words, and the encoder has never seen them in training in any quantity.

Now feed both versions to the tokeniser and the consequence becomes concrete.

>>> tok = AutoTokenizer.from_pretrained("intfloat/multilingual-e5-base")
>>> tok.tokenize("iptal etmek istiyorum faturalarımı")
['▁iptal', '▁etmek', '▁istiyorum', '▁fatura', 'lar', 'ımı']
>>> tok.tokenize(normalise("İPTAL ETMEK İSTİYORUM FATURALARIMI"))
['▁i', 'pt', 'al', '▁et', 'me', 'k', '▁ist', 'iyor', 'um', '▁fat', 'ura', 'lar', 'imi']

Six tokens became thirteen, and the difference is not simply that there are more of them. In the first line the pieces are words and suffixes a Turkish speaker would recognise — ▁iptal is "cancel", ▁fatura is "invoice", lar is the plural. In the second, ▁fat, ▁ura and imi are fragments that appear in the vocabulary because they occur in other languages, so the vector that comes out is an average over pieces whose learned meanings were formed somewhere else entirely. That is why the query retrieved an article about exporting billing history rather than one about cancellation: nothing in ▁fat, ▁ura or lar carries more information about cancelling than about exporting.

Fragmentation as a budget problem#

The same mechanism has a second, larger effect on the document side, and it is the one that survives after normalisation is fixed.

Subword vocabularies are fitted to a corpus, and the fitting procedure decides which languages get short representations. Byte-pair encoding and unigram SentencePiece both start from characters and repeatedly merge the most frequent adjacent pair, stopping when the vocabulary hits its size limit. So a sequence of characters earns its own entry by appearing often in the fitting corpus, and one that appears rarely stays split. When the fitting corpus is a web crawl, English contributes an outsized share of the text even after the sampling temperature that multilingual projects apply to upweight smaller languages, so the vocabulary spends its 250,000 pieces where the text was: common English words end up as single pieces while Turkish inflected forms end up assembled from three or four.

Agglutinative morphology multiplies that, because it builds a single word by stacking suffix after suffix onto a stem. Turkish encodes with suffixes what English encodes with separate words, so a Turkish word carries several English words' worth of content — and each distinct combination of stem and suffixes is a different string, appearing correspondingly less often in the crawl, and therefore less likely to have earned a merge. The two effects compound: a language gets less of the vocabulary and needs more of it per word.

Nobody chose this and no model card reports it, and it is fully measurable on your own text in about five minutes, which makes tokens-per-word a better first question about a candidate encoder than almost anything on the card. Measured over the same 2,000-article content set and its translations, with a shared multilingual tokeniser:

Only the first numeric column is measured. The second is the first multiplied by 400, and it closes in all six rows, which means you can drop your own article length into it and get your own answer — the table is a rule rather than a set of observations. Read the first column as gaps and the picture is not a gentle slope: English to Spanish costs about a tenth of a token per word, Spanish to German about two tenths, and German to Finnish nearly a whole token, after which the tail languages sit together. The languages that break are not spread along a continuum; they cluster past a threshold.

The English article fits inside a 512-token encoder window with 36 tokens to spare. Its Turkish translation is 964 tokens, and the fraction discarded is the overflow divided by the whole — the amount past 512, over 964 — which is 47% of it. Discarded silently, because every mainstream embedding API and library truncates at max_seq_length by default rather than raising. Nothing in a log anywhere says that half of KB-2291-tr was never embedded.

The reason this survives review is that chunking is almost always written against characters or words. A chunker set to 1,200 characters with 200 characters of overlap produces chunks that comfortably fit the window in English, marginally fit in German, and overflow in Turkish by roughly a factor of two — which is the same factor the article row gives, 964 against a 512-token window. The overflow concentrates in the second half of every chunk, which is where the specific content usually sits, since articles open with context and close with instructions. On this corpus, 38% of Turkish chunks exceeded the window and lost an average of 44% of their tokens, and the lost portion was disproportionately the numbered steps at the end.

Fixing the two upstream stages is cheap and the effect is measurable. Locale-aware case folding through PyICU, the Python binding for the ICU library that knows Turkish lower-cases I to ı, dropping unidecode for languages where diacritics are phonemic, meaning the mark changes the sound and therefore the word, and normalising to NFC — the Unicode form that recombines a base letter and its accent into one code point — after the case change rather than before, moved the tr→en question cell from 0.093 to 0.31. Re-chunking against the tokeniser — budgeting 480 tokens of content and measuring with len(tok.encode(chunk)) rather than len(chunk) — moved tr→tr from 0.41 to 0.74. Neither change touched the model.

def chunk_by_tokens(text, tok, budget=480, overlap=60):
    ids = tok.encode(text, add_special_tokens=False)
    step = budget - overlap
    return [tok.decode(ids[i:i + budget]) for i in range(0, len(ids), step)]

Both changes also want a permanent instrument behind them, because they are the kind of fix that regresses the next time someone rewrites the ingestion path. Log the token count of every chunk at index time and alert when the 95th percentile for any language exceeds 90% of the encoder's window, which turns silent truncation into a page. The same applies to queries: record the tokens-per-word ratio per locale on a sample of live traffic, and a normalisation regression shows up as Turkish jumping from 2.4 to 4.1 within an hour of the deploy, long before it shows up in a weekly retrieval metric.

Arguing for the sprint#

A finding like this has to survive someone asking why a sprint should go to 2% of users, and that question deserves a real answer rather than an appeal to fairness. Four arguments do the work, and they are stronger together than separately.

Two percent of queries is not two percent of the business. Turkish queries come from 340 accounts carrying €4.1M of ARR, the annual recurring revenue those contracts are worth, of which €1.6M renews in the next two quarters, because the locale was opened for a specific enterprise segment that is heavy on seats and light on self-service usage. The traffic share measures how often those accounts type into the assistant; the renewal book measures what happens if they conclude the product does not work in their language. Pull the account list before the meeting, because "2% of traffic" and "11% of the renewal book" are the same population described by two different metrics, and only one of them is on the dashboard.

A recall of 0.15 is a broken product rather than a degraded one. At 0.85 the assistant is occasionally unhelpful and users adapt by rephrasing; at 0.15 it retrieves a confidently wrong article five times out of six and generates a fluent answer from it, which is worse than returning nothing, because the answer is trusted. The stale-translation case from chapter 3 is what that looks like when it reaches a customer, and it produced a compliance question rather than a support ticket.

The aggregate is now known to be uninformative for every thin stratum. Eight of the fourteen locales are under 2% of traffic and 9% of traffic sits in cells the headline metric cannot move — chapter 4 put the sensitivity numbers on that. The request is not "spend a sprint on Turkish"; it is "we currently have no ability to detect total failure in nine percent of our traffic, and here is one instance of it that ran for four months".

The fix is bought once for a group of locales, and the fertility table is the evidence. Locale-aware case folding and token-accurate chunking are properties of the ingestion and query paths, and they apply unchanged to Finnish, Estonian, Hungarian, Czech, Greek and Japanese, all of which have fertility above 2.0 or locale-specific casing or both. Running the same trace over Finnish before the meeting takes an hour and gives you a second locale to name; the projected recovery there was 0.29 to 0.63 on the monolingual cell from the chunking change alone. That is a gain of about a third of a point, which is very close to what the same change bought Turkish — 0.41 to 0.74 — and two locales recovering the same amount from one change is the strongest available argument that the cause is shared. One sprint against a shared root cause is a different proposal from one sprint per locale, and it is the proposal that gets funded.

Coverage against fertility#

Raising max_seq_length is the obvious response to the budget table and mostly does not work. An encoder trained with a 512-token window has learned position embeddings — the vectors that tell it where in the sequence each token sits — for 512 positions, so extending it requires interpolation and usually fine-tuning to be worth anything, and attention cost grows quadratically with length, so a 2,048-token window costs roughly 16× the compute per document. Models genuinely trained at 8,192 tokens exist and are a legitimate choice; treating the parameter as a slider on a 512-token model is not.

Fragmentation is also not a vocabulary-coverage failure, and this is the belief most likely to keep somebody from looking. Every one of those Turkish words tokenised into something — there were no unknown tokens anywhere in the trace — which is exactly why nothing errored. Coverage means the input maps to pieces; fertility means how many pieces, and how meaningful each one is. A byte-fallback tokeniser has perfect coverage of every language on earth and terrible fertility for most of them, which makes "no unknown tokens" a check that returns the same answer for English and for Estonian while the quantity that actually matters differs between them by more than a factor of two. A check that cannot vary with the thing it is supposed to detect is not evidence about that thing.

Nor is fragmentation the whole cause of a broken locale, and overclaiming here costs credibility in the next conversation. It was worth 0.22 of the 0.77-point Turkish deficit through normalisation and another large slice on the monolingual side through chunking, and it left a deficit of 0.55 on the cross-lingual cell that belongs to the model. What makes it the overlooked cause is that it sits before every component anyone thinks to inspect, it produces no error, and it is invisible in any log that stores the query as a display string — a log line rendering iptal etmek istiyorum faturalarimi looks like Turkish to a reader who does not speak Turkish, and looks like Turkish with the accents knocked off to one who does.

Normalisation and chunking are now correct and the tail locales are instrumented, and the Turkish cross-lingual cell still reads 0.31 against 0.86 for English. Two lines of case folding and one chunk-size unit were worth a third of the gap, which is a large return for a day of work and is not the whole of it. The rest is the alignment deficit, and there is no configuration change that removes it. The choice at that point is architectural: keep embedding Turkish and Spanish queries directly and accept what the multilingual space gives you, or translate the query into the corpus language and search with a stronger monolingual encoder.