The fragmentation heuristic

Counting how many tokens each numeric literal in a prompt costs gives a ranking of which prompts are likelier to get arithmetic wrong, for the price of ten minutes and no model calls. This chapter gives the score, explains why it predicts anything, and tests it against five adversarial pairs it gets wrong and forty random pairs it gets right 78 percent of the time. It ends with the durable use — a lint that fires when a template's number formatting changes — and with the two boundaries: it does not carry across model families and it is not a confidence over any individual answer.

What tokenization actually breaks — read the current version at https://daviddominguez.dev/learn/en/what-tokenization-actually-breaks/the-fragmentation-heuristic

Back to top