From text to token IDs

Topic 1.0 established that a language model emits text one token at a time, each token selected as a continuation of the sequence in front of it. The unit that mechanism turns on is the token, and a token is not a word. Subword tokenization is the practice of cutting text into pieces smaller than a word and larger than a single character, so that a frequent word survives as one piece and everything rarer is assembled out of fragments of frequent ones.

Where that cutting falls is not a detail of the model's interior; it is visible in the bill and in the budget of any system that sends text to one. To see it happening, take an engineer building a support-ticket triage pipeline. It batches customer records — names, account notes, sometimes a raw JSON export dropped in by an integration, sometimes a ticket typed in Japanese by a customer in Osaka — assembles them into a prompt with a fixed instruction at the front, and sends the result to a hosted model that summarises each ticket and routes it to a team. Unless a second tokenizer is named alongside it, every count here was produced by running the text shown through cl100k_base, the tokenizer behind GPT-4.

The pieces that turn text into a request#

Every request that triage pipeline sends begins as a string an engineer wrote and arrives at the model as something else. Six things stand between the two, and they are easier to keep apart if they arrive named.

The first of them is byte pair encoding, the algorithm that builds a vocabulary once, offline, out of a training corpus, by repeatedly merging whichever adjacent pair of symbols occurs most often in that corpus. A vocabulary is the fixed list of pieces that algorithm produces when it stops. A tokenizer is that fitted vocabulary together with the procedure for applying it, used unchanged on every request afterwards. A token is one piece that comes out when the tokenizer runs. A token ID is that piece's integer index into the vocabulary. The tokenization pipeline is the three-step path — bytes, merges, integers — that connects the string the engineer wrote to what the model is handed.

Byte pair encoding runs once and is then finished with. The vocabulary, the tokenizer, the tokens, their IDs and the pipeline that produces them are live on every request the triage pipeline sends, which is why an engineer who will never run byte pair encoding still has to think about what it left behind.

A little bit of history#

This is why byte pair encoding is worth understanding: the alternatives were tried first, and the triage pipeline tried them in order. Its first version did what classical natural-language software did: it held a fixed dictionary of words, and it turned each word of a ticket into that word's entry. It worked on the tickets it was built against. Then a customer wrote in about a charge on an account registered to a Ms Vintrell, and misspelled her own product as a "mattres topper", and quoted a SKU the dictionary had never seen because the SKU was issued last Tuesday.

A fixed word-level vocabulary answers that ticket either by dropping what it cannot spell or by growing until it can spell everything, and neither answer is good. It can have no entry for Vintrell, for mattres, or for the SKU, and substitute a placeholder — the <UNK> marker that older pipelines are full of — in which case the surname, the typo and the product code all arrive at the model as the same anonymous symbol, and a ticket about a specific customer's specific order becomes a ticket about nobody's anything. Or the vocabulary can grow to cover every string anyone might type, which means every surname in every country, every typo of every word, every SKU any warehouse system will ever mint, and every one of those needs an entry that must exist before the ticket arrives. Neither branch is a matter of engineering effort. The first one throws away the part of the ticket the pipeline exists to read, and the second one asks for a finite list of an unbounded set.

The obvious escape is to go to the other extreme and split every ticket into individual characters, which removes the problem completely: there are only so many characters, so nothing is ever unseen. What it costs is length. A ticket that was three hundred words is about eighteen hundred characters, and every one of those characters is now a separate thing the model has to process, in a computation whose cost grows with the length of the sequence. A batch that fitted comfortably before now does not fit at all, and the part of the request that carries actual information has not grown by a syllable.

Andrej Karpathy's lecture Let's build the GPT Tokenizer and the accompanying karpathy/minbpe repository work through the escape from the dictionary and from the character split at once, in running code: they build a byte-level byte pair encoding tokenizer from scratch, deriving its vocabulary from a corpus rather than from a hand-maintained dictionary, and the resulting tokenizer has no unknown-word case at all — anything can be represented, in the worst case byte by byte — while common words still collapse into single pieces. The vocabulary is finite, the coverage is total, and neither property was bought with the other.

What that buys, it also costs, and the cost is visible in a failure anyone can reproduce. Chris Hay's walkthrough of tiktoken, the tokenizer library behind GPT-4, demonstrates it by asking a model to do something character-level — reverse a string, count the letters in a word — and watching it get the answer wrong. The reason is not that the model cannot count. It is that the string never arrived as characters.

How many letters are in the word 'refunded'?

['How', ' many', ' letters', ' are', ' in', ' the', ' word', " '", 'ref', 'unded', "'?"]

There are no letters in that sequence. refunded, the word the question is about, arrives as ref and unded, and a model asked to count something it was never given has to reconstruct it from what it learned about those pieces during training. Sometimes that works. It is not the same operation as counting.

So the fixed dictionary loses the ticket's content and the character split loses the budget, and what is wanted is a unit that keeps a frequent word whole while still being able to spell out a surname nobody has seen. Nothing about a language hands over that unit ready-made, so it has to be derived, and the deriving is a counting procedure run over text.

Byte pair encoding#

That counting procedure has to come from somewhere, and it is not going to come from a person. Nobody builds a vocabulary broad enough to cover a support pipeline's real traffic by sitting down and listing pieces. There has to be a mechanical rule for deciding which strings earn an entry of their own and which get assembled from smaller ones, and the rule has to be derivable from text rather than from judgement. Byte pair encoding is that rule: an algorithm that starts with the individual bytes of a training corpus as its symbols, counts every adjacent pair of symbols, merges the most frequent pair into a single new symbol, and repeats.

Here is the smallest version of the triage pipeline's world that shows it working — four short ticket fragments as the entire training corpus.

refund requested
refunded the annual plan
refunding the setup fee
please refund this

Starting from bytes, every character is its own symbol, and the algorithm counts the adjacent pairs. The pair r followed by e occurs five times: once in each of refund, refunded, refunding and the second refund, and once more in requested. Its closest rivals — un, nd and fu — occur four times each, once per member of the refund family. So r and e merge first, and the corpus is rewritten with a new symbol re wherever r and e used to sit adjacent. Each pass of counting and merging is a step, and the trace below numbers them.

step   pair          occurrences   new symbol
1      r  + e        5             re
2      re + f        4             ref
3      ref + u       4             refu
4      refu + n      4             refun
5      refun + d     4             refund

Step one won outright. Steps two through five did not: after re exists, the pair re followed by f occurs four times, and so do un, nd and fu, because every occurrence of the stem contributes exactly one of each. A tie is broken by whichever pair the count reached first, deterministically, and the effect is that the merge walks straight up the word — ref, then refu, then refun, then refund — until the whole stem is one symbol that the corpus contains four copies of. Nothing in the algorithm knew that refund was a word. It merged what was adjacent and frequent, and a word is what adjacent and frequent looks like in English.

Now change one thing about the corpus. Leave the four fragments exactly as they are and add a fifth. Because that fragment carries re in three places, the pair keeps its lead and still merges first.

reopened the refund request

step   pair          occurrences   new symbol
1      r  + e        8             re
2      re + f        5             ref
3      ref + u       5             refu
4      refu + n      5             refun
5      refun + d     5             refund
6      ' ' + t       4             ' t'
7      ' t' + h      4             ' th'
8      e  + ' '      4             'e '
9      e  + d        3             ed
10     ' th' + 'e '  3             ' the '

The margin is wider than before: the added fragment contributes an occurrence in reopened, another in refund and another in request, so the count rises from five to eight against five for its nearest rival. Then the sequence diverges. refund climbs to a single symbol exactly as before, because its interior pairs still occur five times each. The rest of reopened climbs nowhere: after the shared re, the pairs op, pe and en occur once apiece and never come near the top of the count, so no merge ever joins them. Ten merges in, refund and reopened share their first merge and nothing after it.

refund     ->  ['refund']
refunded   ->  ['refund', 'ed']
reopened   ->  ['re', 'o', 'p', 'e', 'n', 'ed']

That process — merge the most frequent adjacent pair, rewrite, repeat, stop after a fixed number of merges — is byte pair encoding, and the pairs it merged, in the order it merged them, are the whole of what it learned. Written down, a merge is a pair and a result: (r, e) -> re.

The merges freeze when training stops. If the triage pipeline signs a customer next quarter whose tickets are all about chargebacks, and chargeback was rare in the corpus the vocabulary was fitted on, chargeback gets fragmented on every one of those tickets forever. The vocabulary does not notice the shift in traffic, does not re-count, and does not adapt. What was frequent when the merges ran is what stays cheap.

Vocabulary#

Because byte pair encoding is a procedure that terminates, whatever it arrived at when it stopped has to be written down somewhere the tokenizer can consult on every future request, and the written-down form is a vocabulary: the fixed set of token strings a tokenizer can produce, each one assigned a unique integer index.

Every merged piece the trace produced became one entry. The first 256 entries are the individual bytes, present from the start so that nothing is ever unrepresentable, and each merge appends exactly one more.

index   entry        index   entry
0-255   one byte     261     ' t'
256     're'         262     ' th'
257     'ref'        263     'e '
258     'refu'       264     'ed'
259     'refun'      265     ' the '
260     'refund'

Read that table in both directions, because that is the whole of what a vocabulary does: entry 260 is the string refund and the string refund is entry 260, and there is nothing else in there — no part of speech, no meaning, no record that refunded is the past tense of anything. The leading space in entries 261, 262 and 265 sits inside the entry exactly as the letters do, because the merges that produced those entries counted a space as a symbol like any other.

Production vocabularies are built the same way and are merely longer: cl100k_base holds just over a hundred thousand entries, o200k_base roughly two hundred thousand. Size is the one real decision, and it is a trade rather than a correctness question. A larger vocabulary lets more strings collapse into a single token, which shortens every sequence the model ever processes. It also lengthens the embedding table that topic 1.0 owns, because every additional entry is one more row that table has to carry, in memory, for the life of the model. Whichever size is chosen is then lived with, because changing it means refitting the vocabulary and retraining everything downstream of it.

The tokenizer#

A vocabulary on its own is inert — it is just a list of strings and integers, and a list does nothing to a ticket by itself. Something has to take a string the triage pipeline has never sent before and walk it through the same merges, in the same order, that built the list. That something is the tokenizer, and the words worth pressing on in its definition are fitted and deterministically.

Deterministic means what it says. The same string, run through the same tokenizer twice, produces the same tokens both times. There is no sampling, no temperature, no dependence on what came before in the conversation, and no context in which the tokenizer decides refund should be handled differently today. It is a table lookup and a fixed merge order. The model that consumes its output is probabilistic; the step that feeds the model is not, and that asymmetry is why a token count is something an engineer can assert in a test rather than something to be sampled and averaged.

Fitted means the vocabulary is a property of the corpus it was built on, not a property of text in general. The consequence shows up the moment the same string meets two tokenizers that were fitted separately.

Customer Brzezicki asks why the refund was reversed.

13 tokens
['Customer', ' Br', 'ze', 'z', 'ick', 'i', ' asks', ' why', ' the', ' refund', ' was', ' reversed', '.']

11 tokens
['Customer', ' Br', 'zez', 'icki', ' asks', ' why', ' the', ' refund', ' was', ' reversed', '.']

One string, one sentence of ordinary support English, 13 tokens and 11. Nothing about the ticket changed between the runs; what changed is which corpus the merges were counted on. o200k_base happens to hold zez and icki as entries and cl100k_base does not, so the surname costs five pieces under cl100k_base and three under o200k_base, while every other word in the line — all of them common English — comes out identically under each.

A count is therefore a reading of a named tokenizer taken on an exact string, and belongs to that pairing the way a measurement belongs to the instrument that produced it. A different model family's tokenizer is a different instrument, not the same one with a different price attached to it.

Tokens and token IDs#

Running the ticket line above produced a list of pieces on one side and a count on the other, and the pieces themselves have two forms worth telling apart precisely.

What they share comes first. Every entry a tokenizer can produce has two faces at once: a string, and the integer position of that string inside the vocabulary. Tokenizing a piece of text produces a sequence in which each element is exactly one vocabulary entry, and the sequence can be read off in either face without changing which entries are in it. A token is that entry considered as text — the form a person reads when inspecting what a tokenizer did to a ticket. A token ID is the same entry considered as the integer a request actually carries.

text     refunded the refund

tokens   ['refund',  'ed',   ' the ',  'refund']
ids      [260,       264,    265,      260     ]

The same sequence, twice. Entry 260 appears at both ends, the noun no different from the verb, and that is the first thing that varies independently: two different pieces of text produce the identical token whenever the same string turns up in each with the same whitespace in front of it. In cl100k_base, the sentences The refund was approved. and Customer asked about the refund again. share nothing except a word, and both carry token ID 21639 at the point where that word sits.

The second thing that varies runs the other way: how many tokens a string costs is not a function of how long it is. A string is one token when the whole of it earned a vocabulary entry, and several when it did not, and length has very little to do with which.

' refund'         7 characters   1 token    [21639]
'Refund'          6 characters   2 tokens   [4032, 1263]
' Brzezicki'     10 characters   5 tokens   [3320, 3059, 89, 875, 72]
'ACC-77Q-4831'   12 characters   7 tokens   [30542, 12, 2813, 48, 12, 21884, 16]

The longer string is the cheaper one at the top of that list, because refund is a frequent enough string in web text to have earned an entry of its own and Refund with a capital is not. Further down, the mechanism from the merge trace shows up in production form: a rare identifier gets no entry, so it is reassembled from whatever fragments the vocabulary does hold. The surname the triage pipeline is handed comes back as Br, ze, z, ick, i, and an account code fragments harder still, because the digits and hyphens carry no frequent pairs to merge with. Put the surname into a ticket and the whole sentence costs more than its length suggests.

The refund for Brzezicki is pending.

36 characters, 11 tokens
['The', ' refund', ' for', ' Br', 'ze', 'z', 'ick', 'i', ' is', ' pending', '.']

Five of those eleven tokens are one customer's surname. A batch of a thousand customer records is a batch of a thousand surnames, and the difference between a name the vocabulary knows and a name it does not is a factor of five on the part of the payload that identifies who the ticket is about.

The tokenization pipeline#

That factor-of-five difference in payload comes from the path the text takes on its way in, so the path is worth walking once end to end — bytes go in at one end and integers come out at the other, with the fitted merges happening in between, and the path is literally what a request contains. The tokenization pipeline is that sequence of steps: encoding the text to bytes, merging those bytes into subword tokens using the tokenizer's fitted vocabulary, and mapping each resulting token to its integer index.

To watch those steps happen, take the shortest instruction the triage pipeline sends.

Refund policy

52 65 66 75 6e 64 20 70 6f 6c 69 63 79

['Ref', 'und', ' policy']

[4032, 1263, 4947]

Processing that instruction yields thirteen characters, thirteen bytes, three tokens, three integers. The space between the words did not vanish at the byte stage or at the merge stage — it is byte 20, and it ends up inside the third token rather than beside it.

Those three integers are where the pipeline's job ends. On the model's side they index the embedding table that topic 1.0 owns, and from that point on they are next-token prediction's input rather than tokenization's output. The boundary is worth locating precisely, because everything an engineer can control about this path sits on one side of it.

Walking the path is also what makes the letter-counting failure — the one where a tokenized sequence hands the model integers and no letters at all — actionable: since the pieces are whatever the merges happened to produce, the fix for that class of task is to do the character work in code rather than to ask the model more insistently.

Leading-space tokens and prompt seams#

The trace above ran one clean string through a tokenizer, and the character and word positions that had to be recovered in code were recovered from a string one author wrote end to end. The triage pipeline never sends one clean string. It sends an assembly: a fixed instruction the engineer wrote months ago, concatenated to a ticket body that is different on every request. The join between them is a detail the pipeline does not forgive, and the leading-space token and the prompt seam both live there.

What they share is the setting: a point in an assembled prompt where two independently authored strings meet, and the question of what the tokenizer does with the bytes sitting right at that boundary. The leading-space token is about which vocabulary entry gets chosen there; the prompt seam is about the boundary itself.

Start with the vocabulary entry, using the word already traced through the toy corpus.

refund

[69583]

 refund

[21639]

The same six letters, one token each way, two different integers. A leading-space token is a vocabulary entry that bundles a preceding whitespace character into the token itself, so that a word tokenizes differently depending on whether a space sits in front of it. The mechanism is the one the toy vocabulary already showed at entries 261, 262 and 265, and it scales: since most words in most text are preceded by a space, the space-carrying form of a common word is usually the more frequent string in the training corpus, and the vocabulary ends up holding the spaced form and the bare form as separate entries with unrelated integers.

Now the boundary. Here is the pipeline's prompt template, with the ticket body appended directly after the final newline.

Summarise the ticket below and route it to one team.

Ticket:
{body}

Summarise the ticket below and route it to one team.

Ticket:
 {body}

The difference is one space, sitting after a newline, where no reviewer will see it and no diff tool will draw attention to it. The bare template and the padded template are then filled with the identical ticket body, refund not received for the annual plan.

22 tokens
[..., 35158, 512, 69583, 539, 4036, 369, 279, 9974, 3197]

22 tokens
[..., 35158, 512, 21639, 539, 4036, 369, 279, 9974, 3197]

Same length, same instruction, same ticket, one integer different. A prompt seam is the point in an assembled prompt where two independently authored strings are concatenated, and where the tokenizer's boundaries may not fall where the author of either string intended. The seam here falls between Ticket:\n and the body, and which entry the body's first word gets depends entirely on whether the bare template or the padded one was deployed.

The engineering cost of that lands on prefix caching, on stop sequences and on reproducibility, and in each of them the count is no help at all, because the count did not change. Prefix caching keys on the token sequence, so a template edit that adds or removes a seam space invalidates the cache for every request that uses the template, and the invalidation is silent — throughput drops and nothing logs a reason. A stop sequence is matched against generated tokens, so a stop string authored against one seam can stop matching when the seam moves. And reproducibility goes with them: a request replayed from a stored prompt string reproduces the original run only if the seam is byte-identical, which is a stronger condition than looking identical. Mid-sentence, none of this is fragile, because mid-sentence nobody is concatenating anything. Fragility is a property of the joins.

Counting tokens#

Those engineering costs are worth paying because the question the triage pipeline actually has to answer is not about single strings at all: it is about a whole ticket, or a whole batch of them, and the question is how many. A token count is the number of tokens a tokenizer produces for a given piece of text — the length of the sequence, not a measure of the text's size or its information content.

Being a reading of a named tokenizer, a count carries that tokenizer's determinism — there is no error bar to put on it — and, as the surname sentence already showed, it carries the tokenizer's name with it wherever it is quoted.

Tokenizer compression ratio#

A raw count says little on its own. Fifty tokens is small or large depending entirely on how much text produced it, and the quantity that connects a count to its text is the tokenizer compression ratio: the average number of characters, or bytes, packed into one token for a given piece of text. It is the number a count divides by, and because a count belongs to its tokenizer-and-string pairing, so does this. It is not a constant: it moves with the language the text is written in and it moves with the format the text is serialised in, independently, so both directions are worth seeing with the other one held still.

Hold the format at plain prose and vary the language. Here is one ticket the triage pipeline received, and the same complaint as a customer in Osaka wrote it.

The customer says the refund for the annual plan never reached their card. They
cancelled on the third of the month and were charged again the next morning. They have
written to us twice already and would like the charge reversed and a confirmation sent
by email.

お客様は、年間プランの返金がカードに届いていないとおっしゃっています。月の三日に解約されましたが、翌朝にもう一度請求されました。すでに二度お問い合わせをいただいており、請求の取り消しとメールでの確認をご希望です。

English    263 characters   263 bytes   50 tokens    5.26 characters per token
Japanese   106 characters   318 bytes   109 tokens   0.97 characters per token

The same complaint costs the triage pipeline 50 tokens from one customer and 109 from the other. The mechanism is where the merges were counted. English prose is close to the distribution most hosted vocabularies were fitted on, so ordinary English words and their leading-space forms already hold dedicated entries and whole words collapse into single tokens — ' customer', ' cancelled', ' confirmation' are one token each. A script the corpus contained less of never accumulated those merges, so its text fragments into short pieces, and several characters in the Japanese ticket do not survive as a single token at all: their UTF-8 bytes are split across more than one entry, which is how a 106-character string reaches 109 tokens. Topic 1.2 treats the gap between the English ticket's 50 and the Japanese ticket's 109 as a cost and access problem, and the compression ratio measured here is the mechanism underneath what it treats.

Now hold the language at English and vary the format. The routing job batches customer records, and the same three records — ticket ids, account ids, channel, priority and status — can leave the pipeline in any of four shapes. Here is the CSV one, the cheapest of them.

ticket_id,account_id,channel,priority,status
4471,ACC-77Q-4831,email,high,open
4472,ACC-31B-9920,chat,low,open
4473,ACC-77Q-4831,phone,high,waiting

The other three carry those same three records with the same five fields, and differ only in what they wrap around each value: minified JSON names every field inside every record, pretty-printed JSON adds an indent and a newline per line, XML names each field twice per record. What that wrapping costs, at the three records above and at five hundred of them:

Identical records cost three times as much at the bottom of the table as at the top, and the ordering falls out of what each format repeats. CSV names each field once, in the header, for the whole payload; after that a record is its values, a comma between each pair of them and a newline at the end. Minified JSON names every field again in every record, and pays for {, ", : and } around each naming, so a record's overhead recurs once per record rather than once per payload. Pretty-printed JSON pays everything minified JSON pays and adds the indent, which is not free — ' ' and ' "' are tokens like any other. XML pays for a closing name as well as an opening one, which is why it sits at the bottom despite being the format with the fewest punctuation marks per name.

The gap compounds with the record count, because the repeated part is per-record and the CSV header is not. At three records the minified JSON payload costs 1.5 times the CSV one; at five hundred it costs 1.74 times, and the multiple keeps drifting upward for as long as records keep being added, because the fixed cost CSV pays once is being amortised over more of them while JSON's per-record cost is not being amortised over anything.

Both halves of the pairing have now moved a measurement. The 5.26 measured on English prose says nothing about what cl100k_base will do to Japanese prose or to a JSON batch, as its 0.97 on the Osaka ticket and its 3.32 on the minified batch show directly, and it carries no guarantee at all about a different tokenizer. Run the Japanese ticket through o200k_base and it comes back at 71 tokens instead of 109, on text that did not change by a byte.

The token-to-character heuristic#

Running a tokenizer over every piece of text just to sanity-check a rough budget is more machinery than the question usually deserves, so there is a shortcut, and it is worth having for the one case it was built from. The token-to-character ratio heuristic is the rule of thumb that one token is roughly four characters of English prose, applied as len(text) // 4 to estimate a count without running anything.

The four is the compression ratio of ordinary English prose, rounded down, and the rounding down matters. Across four ticket-length samples of ordinary English — a customer complaint, an agent's account note, a frustrated follow-up and the pipeline's own instruction text — the pooled ratio measured on cl100k_base is 4.97 characters per token, so the heuristic's divisor of four is conservative on the material it was derived from. For the English ticket above, len(text) // 4 returns 65 against a true count of 50: an overestimate of thirty per cent, in the direction that makes a budget check fire early rather than late.

Every other case measured on the same tokenizer is an underestimate instead. The Japanese ticket is 109 tokens and len(text) // 4 returns 26, which is a quarter of the truth. The minified JSON batch is 90 tokens and the estimate returns 74. In each case the heuristic is being asked about material it was never measured on: the script is not the script the ratio came from, and the JSON payload is not prose at all.

Between ordinary English prose at one end and the Japanese ticket and the JSON batch at the other sit the kinds of English text the heuristic is most often assumed to break on, and it is worth knowing which of them actually does. The triage pipeline sees plenty of each. Some tickets are transcribed from phone calls and are full of filler and false starts; others are dense with brand names, product codes and customer surnames.

So um, I called about, you know, the refund, like, last week I think, and the guy said
it would, um, it would come back in three to five days, but then it, you know, it just
never showed up, and so I'm calling again because, like, I still don't have it.

The customer bought a Vantiq Aeroloom mattress topper, SKU VTQ-AL-X2-QN-114, and a
Kelbrook Nimbus pillow pair, SKU KBK-NMB-PP-22, on the Halvorsen Home account, and wants
the topper returned under the ninety-day policy while keeping the pillows.

pooled English prose        4.97 characters per token
transcribed call            3.67 characters per token
brand-heavy ticket          3.57 characters per token
minified JSON               3.32 characters per token

The transcript sits below the prose baseline but stays close to the divisor, and len(text) // 4 estimates it within nine per cent. The filler itself is not what costs: ' um', ' like' and ' basically' are one token each, because they are among the most frequent strings in spoken English and the vocabulary holds every one of them whole. What pulls the ratio down is the comma after each of them — a comma is its own token and buys one character — and the effect is visible when the same transcript is written out with light punctuation, where the ratio recovers to 4.40. Disfluent text looks unstructured to a reader and is close to the safest possible input for the heuristic.

Proper nouns behave differently, and for the reason the Assembled sentence exhibit already gave. The brand-heavy ticket carries the same content shape as an ordinary one, and rewriting it with the brands and codes replaced by generic nouns brings its ratio back to 4.80, so the brands and codes are responsible for essentially the whole of the drop from 4.80 to 3.57. That is a real drag and it is still partial, because the fragmenting strings are a minority of the characters on a page of otherwise ordinary prose — a dozen expensive words carried by two hundred cheap ones.

Structured text degrades the ratio by a different route, and the difference is one of proportion rather than degree. In minified JSON the punctuation and the repeated key names are not a fraction of the payload competing with prose; they are the payload. Brand names move the heuristic's error from thirty per cent high to twelve per cent low. A format change moves it further, and a script change moves it off the scale.

When to trust the heuristic, and when to measure#

Both mechanisms — a script change and a format change — show up in production as concrete cases, and two of them are particularly hard to diagnose from the outside. A customer who writes in Japanese and a paste box for stack traces each break a len(text) // 4 budget check, and they break it in different shapes.

Take the Japanese account first, a support tenant in Osaka. The compression gap between the English ticket and the Osaka one is systematic rather than occasional: it applies to that customer's first ticket and their thousandth, and it applies by roughly the same multiple every time. So every request for that tenant is under-budgeted by about the same amount, and the symptom is an account that does not work — this one customer's tickets fail while everybody else's succeed. What the budget check is protecting is the handoff to the LLM provider, the service that actually runs the model on the far end of the request, and providers differ in what they do with a prompt that arrives over the limit. If the provider truncates a too-long prompt silently rather than returning an error, the symptom is worse than a failure: the requests all succeed, the summaries are built from a ticket with its ending cut off, and what reaches the team looks like a model that has become bad at Japanese.

The paste box fails on a different rhythm. It is a product change that lets a user drop a stack trace or a log excerpt into a ticket, and a traceback is structured text of exactly the kind the minified batch already measured at 3.32, so the underestimate is real. It arrives in a completely different distribution, though. Most tickets contain no paste at all and are fine. Occasionally one arrives with forty lines of traceback and blows the budget on its own. The failure is heavy-tailed rather than systematic, so it presents as rare, unreproducible incidents scattered across every tenant, which is the presentation that survives longest without being understood.

The rule the Japanese account and the pasted traceback yield is narrower than "always measure" and wider than "the estimate is fine". A system that might see either a script change or a format change — which is any system taking text from the outside world — has to gate on a real count from the tokenizer the request will actually use. The estimate still earns its keep in front of that gate, as a cheap pre-filter: text whose estimate is nowhere near the budget does not need tokenizing at all, and skipping the tokenizer on the obvious cases is what makes measuring the ambiguous ones affordable.

An accurate count is the input a context-window budget check consumes, and topic 1.4 owns what that budget is. A systematic undercount is what makes the check pass when it should have failed.

A token-budget check you can reuse#

Of the eight steps above, the fifth is the one most likely to be skipped, and the cross-provider case is where an estimate does the most damage, because a per-document price built by counting with today's tokenizer does not carry over to another provider's model by any fixed conversion factor. The gap between two tokenizers is itself content-dependent, and the triage pipeline's own traffic measures how far it swings. cl100k_base and o200k_base return exactly the same count on the English ticket, on the CSV batch and on the JSON batch — 50, 60 and 90 tokens apiece — and they differ by 38 tokens on the Japanese ticket, and by two tokens on a single sentence containing one unusual surname. A correction factor fitted on the English documents would be 1.00 and would be wrong by 35 per cent on the Japanese ones. Honouring the same price on a different model therefore means re-running the real corpus through the target tokenizer, not adjusting a ratio.

What that count then feeds is owned elsewhere and worth naming. Topic 1.3 covers the input-versus-output token distinction that turns a count into a per-request cost estimate, and topic 1.9 covers the inter-token latency that turns a count into a generation-latency estimate. Each takes an accurate count as its input, which is why the measurement is worth getting right before either estimate is built on top of it.

Both estimates inherit whatever the count is, and the count itself, once the tokenizer is chosen, leaves nothing to argue with. What that determinism does not reach is the choice of tokenizer itself, which is standardised nowhere — a new model family from the same provider can ship a vocabulary of its own, so two prices quoted per million tokens are two numbers whose denominators were never checked against each other.

Whether a rough estimate is good enough for a particular piece of text is not a question that can be settled in the abstract, because the answer depends on the script, on the format, and on which tokenizer is in front of you at the time. It can be settled for a specific case in about a second, by running the real text through the real tokenizer and looking at what comes back.