What tokenization actually breaks

Read these first

• Tokenization

What you will know

• Demonstrate a task the model fails purely because of tokenization

  Character counting, reversing, and letter-position tasks are hard because the model never sees characters as units.

• Reproduce a reliability change by varying only whitespace and digit grouping in an arithmetic prompt, then record token boundaries and answer accuracy

  The semantic value is unchanged while token segmentation and positional patterns change, exposing brittle learned arithmetic routines.

• Predict which of two arithmetic prompts will be less reliable, and say why

  A token-count heuristic is useful for triage, but it is not a guarantee: familiar number patterns, position, and prompt wording can outweigh raw token count.

• Measure token inflation for equivalent sentences in English and two non-English languages, and calculate its effect on context and cost

  Unicode text may fragment into smaller pieces, reducing usable context and raising both input charges and the chance of truncation.

• Decide when to move a character-level task to code rather than fight the tokenizer

  If the task is mechanical and character-level, a three-line function beats any amount of prompt engineering.

Chapters

1. 1

   Character-level tasks

   Counting the letters in a word, reversing a string, naming its fourth character or holding a generated name to exactly eight letters are all functions of the character array, and the character array is not what a model receives. A tokenizer replaces the text with vocabulary indices before the first layer runs, which is why these failures survive stronger instructions. This chapter measures the gap between character tasks and token-level tasks on the same word list, and shows how many ordinary product requirements are character counts in disguise.

   Read chapter
2. 2

   Spelling and token familiarity

   A model that never sees characters can still spell a word out letter by letter, because training text is full of words being spelled, hyphenated, rhymed and corrected. That makes spelling a lookup rather than an observation, reliable on common English and steeply worse on rare surnames, invented brand names, booking codes, base64 and unfamiliar scripts. This chapter measures the gradient, shows that changing a word's capitalisation changes its spell-out accuracy, and draws the line for identifiers, which sit almost entirely in the weak zone.

   Read chapter
3. 3

   Chain-of-thought on character tasks

   Asking the model to list a word's characters one per line before answering raises the hit rate on counting and reversal, because the characters become tokens in the context that later tokens can attend to. That is a real mechanism with three limits: the enumeration is recalled rather than read, so its accuracy caps the whole procedure; it must come before the answer, not after; and it buys a probabilistic result for questions that often have an exact one. This chapter measures the gain, the decay with string length, the token cost, and the four failure classes the technique is applied to.

   Read chapter
4. 4

   Mechanical character work in code

   Redacting card numbers, truncating to a byte limit and validating an identifier against a pattern are exact operations over characters, and a prompt performs them approximately at latency and cost. This chapter sets out the code versions, compares them with the prompt versions on recall, determinism, latency, cost and testability, and gives the propose-and-check loop for the harder case where the model has to generate text under a mechanical constraint. It ends on the division of labour that holds up: the model returns spans and judgements, the code performs the transform.

   Read chapter
5. 5

   Digit grouping and number formats

   '1284.50' and '1 284,50' are the same quantity and different token sequences, and extraction accuracy moves with the sequence rather than with the value. This chapter encodes six real money formats, measures exact-match extraction on each over 200 documents, classifies the wrong answers to show they are re-groupings of the same digit tokens rather than misunderstood conventions, and repeats the measurement on dates. It also marks the boundary: token count is not the operative variable, and the numbers are a property of one vocabulary rather than a law.

   Read chapter
6. 6

   Normalisation and locale-aware parsing

   Once a formatting change has been shown to move extraction accuracy, two repairs are available: few-shot examples that demonstrate the new arrangement, and a system-side change that takes the arrangement out of the problem by canonicalising before the model or parsing locale-aware after it. This chapter measures what the examples buy on the targeted format and on the next one, gives the parser, the verbatim-span guard and the arithmetic guard that make the system-side version trustworthy, and sets out the ladder of inference for documents whose locale is unknown.

   Read chapter
7. 7

   The fragmentation heuristic

   Counting how many tokens each numeric literal in a prompt costs gives a ranking of which prompts are likelier to get arithmetic wrong, for the price of ten minutes and no model calls. This chapter gives the score, explains why it predicts anything, and tests it against five adversarial pairs it gets wrong and forty random pairs it gets right 78 percent of the time. It ends with the durable use — a lint that fires when a template's number formatting changes — and with the two boundaries: it does not carry across model families and it is not a confidence over any individual answer.

   Read chapter
8. 8

   Non-English text and the context window

   The same support conversation costs 1.32 times as many tokens in Spanish and 2.40 times in Japanese, which moves three things at once: the input bill, how much history fits inside a token budget, and how often a long session trips a truncation guard. This chapter measures the multiplier per market on real conversations, converts it into cost per conversation and turns per budget, and sets out what to propose instead of a flat per-seat price. It also separates the arithmetic from the capability question the two are routinely confused with.

   Read chapter