Accepting a language is not aligning its representations

Cross-lingual retrieval — cross-language information retrieval, or CLIR, in the research literature — is searching a collection of documents written in one language using a query written in another: a German question, a set of English articles, and no German copy of the answer anywhere in the index. Multilingual retrieval is the wider case, where one system holds documents in several languages and accepts queries in several more, so a Spanish question may have to be matched against a Japanese document and an English one in the same ranking. The two phrases are often used interchangeably, and the distinction that decides how a system behaves is whether the query and the document that answers it are in the same language.

Neither arrangement exists because crossing languages is interesting in itself. They exist because the obvious alternative is expensive. Keeping a current, maintained translation of every article in every language you sell into means paying per word forever, re-translating on every edit, and living with translations that lag their source, which is why most companies keep one authoritative copy of the content and ask the retriever to reach it from whatever language the user typed in. You meet the result in ordinary places: a documentation search box on a site with a language picker in the corner, a support assistant answering in fourteen locales out of an English knowledge base, an internal policy assistant at a company whose staff work in six languages and whose handbook exists only in English.

The mechanism that makes any of it possible is that the retriever compares meanings rather than words. Text goes through an embedding model, which turns a string into a vector of a few hundred numbers, and searching means finding the document vectors nearest the query vector, where nearness is cosine similarity — a score close to 1.0 when two vectors point in the same direction. Nothing in that machinery requires the query and the document to be in the same language. Whether it works across languages depends entirely on where the model chose to put each language's vectors, and that is the property a model card — the page of specifications a vendor publishes alongside a model — is usually silent about.

Two properties that a card reports as one#

There are two separate things a text encoder can do with German, and the phrase "supports 104 languages" is only ever a claim about the first.

Acceptance means the model takes the input without falling over: the tokeniser has coverage for the script, no character maps to the unknown token, the forward pass produces a dense vector of the right dimension, and that vector is stable across runs.

Acceptance is close to free, and it is worth seeing why rather than taking it on trust, because the reason is what makes the guarantee so weak. A modern multilingual tokeniser is built with byte-level fallback, which spells an unfamiliar character out in its raw bytes, and every byte value is in the vocabulary by construction. That guarantees every input tokenises into something, in any script. Tokenisation succeeding therefore tells you that the fallback exists, and tells you nothing about whether the pieces it produced carry meaning the model has learned to use. A model can accept Amharic, produce a perfectly well-formed 768-dimensional vector for it, and have that vector carry almost none of the sentence's meaning.

Alignment means the geometry does the job you need: a query vector in German lands closer to the English passage that answers it than to English passages that do not. Alignment is a property of pairs of languages, in a specific direction, over a specific kind of text.

The distinction has a close analogue in a database, and it is an analogy rather than a mechanism, so it is worth saying where it stops holding. A Postgres column declared text in a UTF-8 database will accept Turkish, Greek and Japanese without complaint; storage acceptance is trivially true and tells you nothing. Whether WHERE name = 'İstanbul' matches a row inserted as 'istanbul' depends on the collation, which is a separate property that has to be chosen, and which nobody discovers is wrong until comparisons start silently missing. Acceptance is the column type. Alignment is the collation.

Where the comparison breaks is in what you can do about it. A collation is a declared, named, queryable property of a column: you can ask the database what it is and change it. Alignment has no such field. There is nothing to read, no setting to inspect, and no name the model will give you — which is why the rest of this chapter is about producing the evidence yourself rather than about finding where it is written down.

Twelve pairs and a cosine#

The cheapest instrument for telling the two apart is a handful of pairs and a dot product. Take a question you know the answer to, the English passage that answers it, and an English passage from the same product area that does not.

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("intfloat/multilingual-e5-base")

en_q = "query: Why was my direct debit returned by my bank?"
de_q = "query: Warum wurde meine Lastschrift von meiner Bank zurückgegeben?"

gold = ("passage: A returned direct debit means your bank rejected the "
        "collection. Common reasons are insufficient funds, a closed "
        "account, or a mandate that was cancelled by the account holder.")
near = ("passage: To update the credit card on your account, open Billing, "
        "choose Payment methods, and select Replace card.")

v = model.encode([en_q, de_q, gold, near], normalize_embeddings=True)
pairs = {"en_q->gold": (0, 2), "en_q->near": (0, 3),
         "de_q->gold": (1, 2), "de_q->near": (1, 3)}
for name, (a, b) in pairs.items():
    print(f"{name:12s} {util.cos_sim(v[a], v[b]).item():.3f}")

Run that over twelve questions and average, and the interesting output is not the raw similarities. It is the margin — the gap between the gold passage and the plausible distractor — because absolute cosine values are not comparable across models, across languages, or even across query lengths, and the gap between right and wrong is what a ranker consumes. A ranker never reads a cosine on its own; it sorts, so what decides the ordering is whether the correct document's score exceeds the wrong one's, and by how much relative to the noise.

The margin column is the subtraction of the second column from the first, row by row, and it closes exactly in all three rows — which is the only reason the table can be read as gaps at all. Read that way it says something the levels hide. Going from English to Turkish, the gold column falls by roughly seven hundredths while the distractor column falls by under three. Both fall, so there is a general penalty for the query not being in the corpus's language. The penalty is not uniform, and that is the whole finding: a uniform penalty would subtract the same amount from both columns and leave the margin untouched, and a retriever does not care about a shift that moves every candidate together. What actually happens is that the gold similarity erodes faster than the distractor's, so what the language change destroys is the relevance signal specifically.

The German margin is half the English one, which costs you ranking accuracy without destroying it. The Turkish margin is two thousandths, which is smaller than the difference between running the same sentence with and without a trailing full stop — so the ordering of the top ten Turkish results is being decided by whatever else those vectors happen to encode. That table took twenty minutes to produce. It is worth running against two or three candidate encoders at once, because the spread between them is larger than most people expect: the same twelve Turkish pairs on a model trained with a translation-ranking objective produced a margin of 0.031, which is not English-level and is an order of magnitude away from being noise.

Why acceptance and alignment come apart#

The gap is a direct consequence of how a multilingual encoder was trained, and this is the one part of a model card that genuinely predicts cross-lingual behaviour.

Take the commonest recipe first. A model pretrained with masked language modelling — an objective that hides words in a sentence and trains the model to guess them from what surrounds them — over a union of monolingual corpora — a hundred separate piles of text, no pairing between them — has no objective anywhere in its training that ever asked it to put a German sentence near its English translation.

Follow what that objective rewards, step by step, and the outcome stops being surprising. The model is scored on how well it guesses a hidden word from its neighbours. Knowing which language a sentence is in narrows the candidate words from the whole multilingual vocabulary to one language's slice of it, which is an enormous reduction in the guessing problem — so language identity is among the most predictive features available and the model is directly rewarded for representing it. Nothing anywhere in the loss ever penalises representing it, because no training example asks the model to treat two languages as interchangeable. Neither term pushes back, so the feature is free to grow as large as it is useful.

The result is that the model learns each language well and organises the space by language. If you project the embeddings of a mixed-language corpus down to two dimensions, you see islands, one per language, and the strongest single direction in the space is often language rather than topic. Cross-lingual transfer does emerge from shared subwords and shared entities — proper nouns, numbers, borrowed technical terms — which is why the German margin above is degraded rather than absent, but it emerges as a side effect and nobody tuned it.

Alignment gets built deliberately, by one of a few objectives, and each of them works by making language identity insufficient to win. Translation ranking trains the model on bitext, meaning pairs of sentences that are translations of one another, so that a sentence and its translation are positives and every other sentence in the batch is a negative. A batch drawn that way contains sentences in the same language as the query that are not its translation, so a model that scored by language identity would rank those above the true positive and lose — the only way to win is to make translation-equivalence a stronger direction in the space than language identity. That is how LaBSE was built and why it is unusually strong at pairing sentences across languages.

Multilingual knowledge distillation gets to the same place by a different route. Take a strong English teacher model, and train a multilingual student to produce the teacher's English vector for both the English sentence and its translation. The target is the same vector in both cases, so the student is being told, in the most direct terms available, that the two inputs must map to one point — which forces the student's languages into the teacher's single space. Nils Reimers' write-up Multilingual and cross lingual embeddings walks through that construction, and it is the origin of the paraphrase-multilingual-* family. Contrastive pretraining on mined multilingual pairs, the recipe behind the multilingual E5 models, sits between the two.

Any of these appearing in the training description is real evidence. A card that says "trained on Common Crawl in 100 languages" and lists nothing about pairs, bitext or distillation is describing acceptance.

Two further things on a card carry signal. Script coverage is not the same as the language list, because a language list counts languages while the tokeniser was fitted to a token budget: a 250,000-piece vocabulary spread over 100 languages is generous for Latin-script European languages that share subwords and thin for Telugu or Khmer, and the tail languages of that vocabulary are the ones whose margins collapse. Low-resource behaviour is usually visible only as pretraining volume per language, when it is published at all, and the drop-off is steep — the languages beyond roughly the top thirty by corpus size are typically training on two or three orders of magnitude less text than English.

Reading a card for evidence of alignment#

The first row is worth naming as the stumble it is, because it is the one a careful engineer walks into. "Supports 104 languages" reads exactly the same whether the model in question is LaBSE or a masked-language-model checkpoint with no bitext anywhere in its history. The sentence is a constant across models whose cross-lingual behaviour differs by an order of magnitude, and a claim that does not vary when the thing it is supposed to describe varies is not weak evidence — it is evidence about something else, in this case about the tokeniser.

The MIRACL row is the one that catches experienced people, because MIRACL is a genuinely good multilingual retrieval benchmark with human judgments in eighteen languages and it is the number most multilingual embedding cards quote. Every one of its query sets is judged against a corpus in the same language, so a strong MIRACL score says the model does Swahili-to-Swahili retrieval well and says nothing about Swahili-to-English. The BGE documentation's Evaluate on MIRACL page is worth reading precisely because it makes the per-language structure of that evaluation explicit; the same page also shows how little work it is to run per-language numbers yourself once the harness exists.

Where alignment is not what you are missing#

Alignment is not translation quality, and conflating them leads people to test the wrong thing. A model can align "Lastschrift" with "direct debit" in vector space while being unable to generate a usable German sentence, and a translation system can produce beautiful German while its encoder is irrelevant to your retrieval. Topic 4.12 covered what changes in a model's behaviour and instruction-following across languages; that is a different mechanism from the geometry discussed here, and a model that behaves well in German at generation time tells you nothing about where its German query vectors land.

Alignment also does not survive a domain gap on its own, and the reason is visible in the objectives above. Every one of them learns alignment from a supply of paired sentences, so the pairs it saw are the region of meaning where its languages are pinned together. The bitext used to align a model is overwhelmingly news, subtitles, parliamentary proceedings and web-mined pairs, so the alignment you inherit is strongest on general prose and weakest on the vocabulary that is specific to your corpus. A model aligned on Europarl will map "Vertrag" to "contract" reliably and has never seen your product's German term for a partial refund reversal, which is why the probe above should use questions from your own support queue rather than sentences you invented.

Finally, alignment is orthogonal to the asymmetry question from topic 8.8, where a model has to be told whether the text in front of it is a question or a passage. A model can be beautifully aligned across German and English and still be a symmetric-similarity model that treats a nine-word question and a four-hundred-word article as objects to be compared for sameness. Both properties have to hold, they are trained by different objectives, and a card that gives you evidence for one gives you none for the other.

The margin probe is a smell test with all the limitations of one. Twelve questions have no statistical weight, they were chosen by whoever was in the room, and the distractor was hand-picked rather than drawn from the 41,000 articles a real system ranks against. Cosine margin is not the metric anyone is accountable for either: what a German locale is measured on is whether the right article appears in the top ten of a real index, under fixed relevance judgments, at a fixed cutoff.

What the probe does is separate two claims that a card runs together, and it does it cheaply enough to run before a rollout rather than after one. Acceptance is guaranteed by a fallback rule in the tokeniser and costs nothing to obtain, so a language list is a statement about the tokeniser. Alignment is built by an objective that pairs sentences across languages, holds only where those pairs went, and shows up as a margin between a right answer and a wrong one. Turning that margin into a number somebody can be accountable for requires an evaluation you can defend — the same judgments, the same corpus, the same k, with only the query language changed, so that the difference between two runs is attributable to the language and nothing else. Building that comparison, and being honest about the bias baked into the cheapest way of building it, is the next chapter.