Normalisation and locale-aware parsing

Once a formatting change has been shown to move extraction accuracy, two repairs are available: few-shot examples that demonstrate the new arrangement, and a system-side change that takes the arrangement out of the problem by canonicalising before the model or parsing locale-aware after it. This chapter measures what the examples buy on the targeted format and on the next one, gives the parser, the verbatim-span guard and the arithmetic guard that make the system-side version trustworthy, and sets out the ladder of inference for documents whose locale is unknown.

What tokenization actually breaks — read the current version at https://daviddominguez.dev/learn/en/what-tokenization-actually-breaks/normalisation-and-locale-aware-parsing

Back to top