Pivot translation and native prompting

Carrying a request out of its own language into English, running an English-language pipeline over it, and bringing the answer back is called a pivot. Both halves of it run through a machine translation (MT) engine — a dedicated translation service, a cloud MT API or a self-hosted translation model, rather than the model that generates your answers. Teams reach for it because it lets a knowledge base, a system prompt and an eval set that all took a year to tune keep working in a language nobody on the team reads.

The decision is usually framed as pivot against native, and framing it that way hides the option most teams eventually land on. Translating inbound and translating outbound are independent choices, and the arrangement where you translate the question into English but instruct the model to write its answer directly in the user's language is a distinct third design with its own cost profile. This chapter is about what each of the three genuinely differs on: terminology, register, stage count, latency, money, and where a failure can originate.

Three arrangements, not two#

The middle arrangement is worth naming carefully because its advantage is structural rather than incidental. When the generation model produces the answer in Thai, it does so with the retrieved passages and the full conversation in front of it, which is a strictly better position than a standalone translator working on finished answer text alone.

Position is doing real work in that sentence, and it is worth spelling out what the translator is missing. A standalone MT engine receives one string: the finished English answer. It cannot see the passages that answer was built from, so where the English is ambiguous it has to guess, and it has no evidence at all about which tokens are product names rather than ordinary nouns. The generation model has the passages, the conversation and the system prompt in its context at the moment it chooses each Thai word. Both systems meet the same ambiguities. Only one of them holds the material that resolves them.

Laid against the dimensions that actually decide it:

The latency and cost figures come from a commercial MT API at $20 per million characters with a question averaging 180 characters and an answer averaging 900, and p50 there is the median turn rather than the slow tail. Two things in that row are easy to miss. The outbound direction dominates both cost and latency because answers are five times longer than questions, so arrangement B removes about 80% of the translation bill by removing the half that was doing the least work for you. And 2.2 cents per turn is the same order as the generation call itself on a mid-tier model, which makes the full pivot a materially more expensive product per conversation, not a rounding error.

Both figures come straight off those inputs. A full pivot translates the question and the answer, which is 180 characters plus 900, a little over a thousand; at $20 per million characters that is a little over two cents, which is the 2.2¢. Arrangement B translates only the question, so 180 characters at the same rate is under half a cent. The saving is the outbound share of the characters — 900 out of the thousand-odd total — which is where the 80% comes from, and the reason answers dominate is simply that they run five times longer than the questions that prompted them.

Latency does not follow the same proportions, and the discrepancy is informative. The outbound call carries five times the characters of the inbound one and adds under twice the milliseconds. That says most of what a translation call costs is a fixed network round trip rather than the translating, which is worth knowing before anybody proposes shortening the text or batching the calls: shrinking the characters barely shrinks the latency, and removing a call removes all of it.

What the pivot costs, term by term#

Terminology normalisation is the concrete loss and it is measurable. An MT engine that renders a Thai product name into fluent, generic English has not made an error any translation metric will report, and it has changed which document the retriever finds, because a knowledge base indexes the SKU — the catalogue identifier the company sells and bills under — as an exact string. A do-not-translate glossary handles most of it, and it is the first thing to build rather than the last:

{
  "source": "th", "target": "en",
  "glossary": [
    {"src": "แพ็กเกจธุรกิจ", "dst": "Business tier"},
    {"src": "แพ็กเกจโปร",    "dst": "Pro tier"},
    {"src": "สิทธิ์คืนเงิน",   "dst": "refund entitlement"}
  ],
  "do_not_translate": ["Business tier", "Pro tier", "Studio", "Relay", "SKU"]
}

What a glossary does not touch is the residue: terms coined by users, misspellings of product names, and cases where the source phrasing is genuinely ambiguous between two SKUs. That residue is the honest floor of arrangement A.

Why this class of failure is silent is worth following to the end, because it is what makes a glossary urgent rather than tidy. A retriever verifies nothing. It scores every document against the query and returns the best few, and it returns them whether or not any of them is relevant, because returning the best available is the whole of what it is defined to do. Change the query string and the scores change; change the scores and the documents that come back change. Nothing along that path holds a notion of a wrong answer, so no stage raises, no counter increments, and the model downstream receives a set of passages that looks exactly like a successful retrieval. The generation stage then does its job correctly on the input it was handed — which is why the stage everybody instruments is the stage that will exonerate itself.

Register is the second loss and it does not respond to a glossary. Thai marks politeness through sentence-final particles and lexical choice, and the level appropriate to a company writing to a customer is not the level a translation engine defaults to; MT output tends to land in a flat, neutral register that reads as brusque in a service context. Japanese has the same problem with more structure, since the choice between plain, polite and honorific forms is grammatically pervasive and a support answer in the wrong one is not a stylistic complaint, it is a customer noticing that you are not really speaking to them. Arrangement B fixes this cheaply, because the generation model can be told what register to use and generally holds it, and instructing an MT engine to produce keigo, the Japanese honorific register used when addressing a customer, is not a thing MT engines do.

The third cost is structural and it is the one that shows up in every incident afterwards. A five-stage pipeline has five places a failure can originate, and a per-language quality number covering the whole chain tells you which language broke and not which stage. Topic 3.14 drew the general distinction between component and end-to-end evaluation; the pivot is a case where the end-to-end number becomes actively misleading, since the generation stage is the one everyone instruments and it is frequently the stage behaving correctly on the input it was handed.

The pivot gets harder on turn two#

Every diagram of a pivot shows one question and one answer, and the arrangement that looks clean for a single turn has a decision waiting in it for multi-turn conversations. On turn four, the user writes Thai, the stored history contains three prior exchanges, and something has to decide what language that history is in when it goes back to the model.

Translating the whole history on every turn is the version that gets written first and it degrades in a way that is easy to miss. Text that has been through a translation engine and back accumulates drift, so by turn six a phrase the customer used on turn one may have been rewritten twice and no longer says what they said; worse, the translation is non-deterministic across engine versions, so the same conversation replayed next month produces different history. The fix is to store both forms at write time — the original text and the translation, tagged with the engine version that produced it — and to replay the stored English rather than re-translating. That makes the history stable, makes a stage-by-stage trace of a bad answer possible at all, and roughly doubles the storage for conversation text, which is not a meaningful cost next to the rest of the stack.

Storing rather than re-deriving buys a second property, and it matters more than the drift does. A pipeline that re-translates its history on every turn is a function of a remote service's current behaviour, so its output for a given conversation is not reproducible: the engine ships a new version, last week's conversation replays differently, and nothing in your system records that anything changed. Storing the translation alongside the engine version that produced it turns the history from a computation into data. That is the precondition for a stage-by-stage trace, because a trace of a stage you cannot re-run is a screenshot.

Arrangement B has a specific artefact on top of this. The model is generating Thai from English passages, and it will sometimes lift a phrase from a passage verbatim, so the answer arrives as fluent Thai with an untranslated English fragment embedded in it — a policy name, a UI label, a quoted sentence. Roughly half of those are correct behaviour, since UI labels that appear in English in the Thai product should stay in English. Separating the two requires knowing which strings are localised in the product and which are not, which is a list the localisation team already maintains and that nobody thinks to give the assistant. A detector that flags Latin-script runs longer than about twenty characters in a Thai answer, checked against that list, turns an ambiguous quality complaint into a countable metric.

Native prompting moves the problem rather than removing it#

The argument for arrangement C usually arrives as "models are multilingual now, and translation is always lossy." Both clauses are true and the conclusion does not follow, because the corpus is English. Removing the pivot removes the inbound translation, which means the query that hits the retriever is now Thai and the documents are still English. Matching a query written in one language against documents written in another is cross-lingual retrieval, and something has to bridge that gap: a multilingual embedding model, a translated index, or a query-translation step that is the inbound MT stage under a different name.

The reframe worth keeping is that the mismatch is conserved. A Thai question and an English corpus are in different languages, and no arrangement makes that untrue — every design has to bridge it somewhere. What the three differ on is where the bridge sits and how legible it is. In A and B it is an explicit stage with an input, an output, a version number and a bill, so it can be logged, diffed, glossaried and blamed. In C it is inside a similarity score, where there is nothing to inspect: a document either scored high enough or it did not, and the reason is a vector. Removing a translation stage does not remove translation. It converts a legible failure into an illegible one.

So arrangement C does not have three stages and no hard problems; it has three stages and one hard problem that arrangement A had already solved by accident. The other reason C takes six to ten weeks rather than three is that the system prompt does not port. A prompt carrying eighteen months of English fixes encodes specific phrasings, specific refusals and specific formatting rules, and translating it produces a prompt that has never been tested and whose failure modes are fluent output that quietly drops the fourth and fifth of five constraints. A native prompt is a new prompt, and it needs its own iteration loop against its own eval set.

The evidence that should flip the choice#

Moving to C is an evidence question, and the thresholds are worth writing down before launch so the argument later is about data:

• Cross-lingual retrieval recall@10 for Thai queries against the English corpus — the share of questions whose correct document appears anywhere in the top ten results — measured offline on a labelled set, comes within three points of the recall for the translated English queries. If it does not, C cannot beat B on answer correctness no matter how good the generation is.
• The terminology-alteration residue after the glossary stops falling with glossary work — it plateaus around 3% here — and those cases are concentrated in high-value questions such as billing and entitlements.
• A native Thai prompt, evaluated on the same fixed Thai case set, beats the pivot on a paired per-case comparison rather than on a rate comparison against a different sample.
• Thai reaches a traffic share where the 0.4¢ per turn and 210 ms are worth an engineering quarter, which for most products is somewhere north of 10% of volume.

None of this is a one-way door, and it does not have to be one decision for the whole product. Japanese is well covered in pretraining and has a large, well-behaved multilingual embedding story, so C is realistic there sooner. Thai is thinner on both counts, so B may remain correct for a long time. Running different arrangements per language is more operational surface, and it is honest about the fact that "multilingual support" is not one capability.

The choice is therefore a choice about which failure surface you would rather own: normalised terminology and a register you cannot steer, or a retrieval problem you have to solve from scratch against a corpus written in another language. Arrangement B exists because those two surfaces are separable, and separating them is nearly always cheaper than committing to either whole.

Two consequences of this choice have been deferred so far and they are the ones that get skipped in design reviews. Whichever arrangement you pick changes the model's refusal behaviour in ways that are not visible in a task-success number, and arrangements A and B both create a channel where a user's text arrives inside your pipeline having been rewritten into fluent English by a system that has no notion of instructions versus content.