Spelling and token familiarity

A model that never sees characters can still spell a word out letter by letter, because training text is full of words being spelled, hyphenated, rhymed and corrected. That makes spelling a lookup rather than an observation, reliable on common English and steeply worse on rare surnames, invented brand names, booking codes, base64 and unfamiliar scripts. This chapter measures the gradient, shows that changing a word's capitalisation changes its spell-out accuracy, and draws the line for identifiers, which sit almost entirely in the weak zone.

What tokenization actually breaks — read the current version at https://daviddominguez.dev/learn/en/what-tokenization-actually-breaks/spelling-and-token-familiarity

Back to top