The encoder's 512-token ceiling indexes only a chunk's opening

An encoder is the part of an embedding model that reads a piece of text and produces the intermediate representations a pooling step then collapses into a single vector. It accepts a fixed maximum number of tokens — the word fragments that model's own tokenizer emits — and that maximum is a property of the trained weights and of the library serving them. It is not a property of your configuration file, and it does not move when you edit a chunk-size setting.

That fixed maximum is easy to ignore, because chunk size is the dial people reach for when retrieval feels too narrow. A corpus of operational runbooks whose procedures keep arriving half-finished in a model's context window invites one obvious repair, which is to make the chunks bigger. Above the encoder's limit that repair changes how much text gets delivered and changes nothing about how much text is described in the index. The measure that registers the difference is span containment@10 — the fraction of evaluation queries whose answer-bearing passage appears anywhere in the top ten retrieved chunks — and it moves the wrong way.

Where token 513 goes#

Take a single chunk of 1,000 tokens, handed to a model whose encoder accepts 512. What happens to token 513 is that it is dropped, before pooling, without a warning, an exception or a log line. The tokenizer emits 1,000 ids, the model's configured max_seq_length is 512, and the forward pass sees the first 512.

The reason the cut happens at all is worth walking rather than asserting. A transformer encoder does not read a sequence position by position and stop when it is tired; it consumes the whole sequence at once, and every position in that sequence needs a positional representation the network learned during training. The network learned a fixed number of those. A sequence longer than the number of positions the network has representations for therefore has token ids with nowhere to sit. The serving library resolves that by keeping a prefix and discarding the tail, and the tail is what gets discarded because that is the end the sequence runs off.

What happens next is the step that makes the failure invisible. Mean pooling averages over the token representations the forward pass produced and emits one 768-dimensional vector. An average has one term per position it was given, and it was given 512 of them. There is no term in that average for position 513, or for any position after it. The vector is therefore not a blurred summary of the whole chunk. It is an exact description of a prefix of the chunk, and it is exactly as confident as a vector over a chunk that genuinely ended at token 512.

That distinction is the whole failure mode in one sentence. A blurred description degrades gracefully — a little more text dilutes it, a little more dilutes it further, and recall sags in proportion. A prefix description does not degrade at all. It is a perfectly good description of the wrong quantity, and nothing downstream can tell the difference, because a vector carries no record of how much text went into it.

One thing here is handed to you and the rest is chosen. The number of positions the encoder accepts is a requirement: it follows from the weights you loaded and the max_seq_length your serving stack resolved, and no amount of configuration changes it. What happens to a chunk that exceeds it is a design decision, and it is made by whichever component sees the chunk first. Truncating, rejecting and splitting are all available, different vendors chose differently, and the default you inherit decides which of them your pipeline is running.

A cliff at the boundary, not a slope#

Bucketing evaluation queries by the length of the chunk that holds their answer makes the shape of the regression clear.

The short chunks got better, which is what you would expect from fewer arbitrary cuts. The long ones — the ones the change was made for — got dramatically worse.

Read the table by its gaps rather than by its levels, and the gaps say something the levels do not. Stay inside the 1,000-token column and step from the first row to the second: containment falls by thirty-three hundredths as the answer-bearing chunk crosses from under the encoder's limit to just over it. Step again, from the second row to the third, and it falls by a further twelve hundredths. The first gap is nearly three times the second, and the two gaps are describing different things. Crossing the boundary is what costs most of the recall; getting further past the boundary costs a slower, smaller amount on top. That is the fingerprint of a hard limit rather than of a general dilution effect, because dilution would spread its cost evenly across both steps.

The horizontal gap says the change did what it was supposed to do wherever it was allowed to. On chunks the encoder can read whole, containment rose by eight hundredths, which is the honest benefit of cutting a document at fewer arbitrary places. The change was not wrong about its own mechanism. It was applied past the point where the mechanism can operate.

The two dashes are not missing measurements, and reading them as gaps in the data would be reading the table wrong. The bucket is defined by the length of the answer-bearing chunk, and under a 300-token configuration no chunk is 513 tokens long, so those two cells have no queries to describe. This is also why the table cannot be read as a row-by-row comparison of two configurations: only its first row compares like with like, and the rows below it exist in one column only.

One residual in this table should be named rather than smoothed over. Weighting the three cells of the 1,000-token column by their query counts produces a figure a little below the aggregate containment the run reported for that configuration, and the gap is far larger than rounding the cells to two decimal places could account for. Either the aggregate was computed over a set of queries that is not exactly these, or one of the two numbers is wrong. The chapter's own artefacts do not settle which, so neither figure should be quoted as a decomposition of the other, and nothing below rests on the two of them agreeing.

Indexed and served are two different quantities#

The stored text is untouched. An ingest pipeline that wrote the whole 1,000-token chunk into the document store wrote all of it, and when that chunk is retrieved the whole thing goes into the context window. So the system ends up in a state that is easy to describe and hard to notice: the second half of every long chunk is stored, is served, is visible when you open the record, and is not indexed. Queries about steps seven through eleven of a procedure are scored against a vector built from steps one through six, and the procedure's tail is retrievable only by luck — when the opening happens to be similar enough to the query to win on its own.

Two quantities that a configuration file presents as one number have come apart here, and it is worth naming them separately because every later argument turns on the distinction. The served length is how much text a hit delivers, and it is whatever you wrote to the store. The indexed length is how much text influenced the vector that decides whether the hit happens, and it is the served length or the encoder's limit, whichever is smaller. Below the limit the two move together and a single chunk-size setting describes both. Above it they come apart, and only one of the two is still responding to the setting.

Chapter 8.7, on how API clients handle over-long input, established that clients truncate silently and that what gets cut is the end. The point that matters for sizing is narrower: the number you choose for chunk size is only real up to your encoder's limit. Above that, increasing chunk size increases how much text you serve and does not increase how much text you index. The two halves of the pipeline drift apart while the configuration file suggests one coherent number.

The tokenizer that counts is not the tokenizer that truncates#

There is a defence against all of this that feels complete and is not, and it is worth naming because it is the one a careful engineer reaches for first. The defence is counting. If you count a chunk's tokens before you index it, the reasoning goes, then you know its length, and knowing its length is knowing whether it fits.

What that reasoning omits is that "token" is not one unit. tiktoken, OpenAI's tokenizer library, implements the byte-pair encoding OpenAI's own models were trained with. A BERT-family model such as bge-base-en was trained with a WordPiece vocabulary built from different text by a different procedure. Both libraries return a count of real tokens for the same string, and the two counts are different numbers, in a direction that depends on the text. A chunk built to 1,000 tokens under one vocabulary is some other length under the other, and the encoder truncates according to the second.

The shape of the mistake is the shape most defences in this book fail in: the defence stays constant while the thing it has to discriminate varies. "We counted the tokens" is the same sentence whichever model the corpus is about to be embedded with, and the count it refers to only predicts truncation for one of them. That is also why the guard further down this chapter builds its counter with AutoTokenizer.from_pretrained on the same model id that is passed to SentenceTransformer, rather than reaching for a general-purpose counter.

The scope of the error deserves to be stated precisely, because overstating it would teach the wrong lesson. A thousand tokens under any vocabulary in common use clears a 512-token limit comfortably, so a mismatched counter is not what caused a regression at that size. Its cost arrives when a size is chosen near the boundary, where a count off by a modest fraction is the difference between a chunk that fits and a chunk that is half-indexed — and at that size nothing in the logs, the store or the evaluation will point at the tokenizer.

What your model actually does, measured rather than looked up#

Model cards are a poor source for this, because the number they quote is frequently the architecture's positional-embedding capacity — how many token positions the network has learned representations for, which is a fact about the weights — rather than what the serving stack applies, and the serving stack is what you are running. A model whose card says 8,192 will truncate at 512 if sentence-transformers loaded a max_seq_length of 512 from its config, and it will do so quietly.

Two probes settle it. The first asks whether the limit is enforced by truncation or by rejection, and the second finds the number.

import numpy as np
from sentence_transformers import SentenceTransformer
from transformers import AutoTokenizer

model = SentenceTransformer("BAAI/bge-base-en-v1.5")
tok = AutoTokenizer.from_pretrained("BAAI/bge-base-en-v1.5")
print("configured max_seq_length:", model.max_seq_length)   # 512

body = " ".join(f"step {i}: rotate the credential and confirm the checksum."
                for i in range(1, 200))                     # ~1,600 tokens
ids = tok(body, add_special_tokens=False)["input_ids"]

def head(n):  return tok.decode(ids[:n])

full, clipped = model.encode([body, head(510)], normalize_embeddings=True)
print("cos(full, first-510):", float(full @ clipped))       # 0.9997

The logic of that probe is a comparison between two encodings that ought to differ and do not. One input is a body of text long enough to be well past any limit under discussion; the other is its first 510 tokens, cut just short of 512 so that adding the model's own start and end markers lands the encoded sequence at the limit rather than over it. If the encoder read both inputs whole, the two vectors would describe different amounts of text and their cosine would sit somewhere in the 0.85–0.95 range, because the added text is repetitive but not identical. A cosine of 0.9997 says the two encodings saw very nearly the same tokens, which is only possible if everything past the limit was thrown away. On the estimate in that comment, the extra 1,100 tokens sitting beyond the prefix changed the vector by less than rounding. The remaining thousandths come from decoding ids back to a string and re-tokenizing it, which is not a perfectly lossless round trip.

Two cautions attach to that probe. The comment putting the body at roughly 1,600 tokens is an estimate rather than a measurement, and a count of words is not a count of tokens — a WordPiece vocabulary splits some words into several pieces and merges nothing, so the token count of a passage is at or above its word count and the ratio varies with the vocabulary. Nothing in the probe depends on that estimate being exact; it depends only on the body comfortably exceeding the limit, which it does under any plausible ratio. And the probe reports what the loaded stack does today, so it has to be re-run when anything under it moves.

Finding the exact cut-off is a search on the same principle — encode prefixes of increasing length and find where the vector stops moving.

lengths = [128, 256, 384, 448, 512, 640, 768, 1024, 1536]
vecs = model.encode([head(n) for n in lengths], normalize_embeddings=True)
for n, v in zip(lengths[1:], vecs[1:]):
    print(f"{n:5d}  cos to previous prefix: {float(v @ vecs[lengths.index(n)-1]):.4f}")

The similarities climb steadily and then pin at 1.0000 from 512 onward, which is the limit reported by the only authority that matters. A cosine of exactly one between two prefixes of different lengths means the encoder produced the same vector twice, and it can only do that if the extra tokens in the longer prefix never reached it. Run this against every embedding model you are considering, and re-run it whenever the model or the client library version changes, because a sentence-transformers upgrade that alters how max_seq_length is resolved will move it without any change to your code.

Hosted APIs behave differently from each other, and the differences change what your pipeline has to do.

Prices are list rates at the time of writing and move often enough that they are worth confirming rather than quoting. The column that changes your architecture is the third one. An API that returns 400 is the friendly case, because an over-long chunk becomes an ingest failure you have to handle, and handling it means deciding — split it, drop it, or summarise it — where a silent truncator makes that decision for you and does not tell you which chunks it applied to. The Cohere row is the trap in the middle: the default is server-side truncation, so a pipeline written against the defaults behaves like the self-hosted case, and setting truncate="NONE" converts it into the OpenAI case. That one parameter is the difference between an index with an unknown fraction of half-described vectors and a pipeline that cannot produce one.

The gap between the first column and the third is the one to read across rows. Two of these four models accept 512 tokens and the two of them behave differently at the boundary, while two models whose capacities differ by several times over behave identically at theirs. Input length and over-limit behaviour are independent facts about a vendor, and knowing one tells you nothing about the other.

Whichever behaviour you are running against, the guard is the same and belongs in the ingest path rather than in a runbook. Count the chunk's tokens with the model's own tokenizer, compare against the measured limit, and record the outcome on the chunk record so that the fraction of half-indexed vectors is a queryable number rather than an open question.

LIMIT = 512   # measured, not read off the model card

def prepare(chunk_text: str) -> dict:
    n = len(tok(chunk_text, add_special_tokens=False)["input_ids"])
    if n > LIMIT:
        raise OversizedChunk(f"{n} tokens exceeds encoder limit {LIMIT}")
    return {"text": chunk_text, "n_tokens": n, "encoder_limit": LIMIT}

Raising here rather than clipping forces the decision back to the splitter, where it can be made with the document's structure in view: subdivide the section at its next paragraph break, or at its next sentence break, and index two well-described chunks instead of one half-described one. A single n_tokens column also makes the whole class of problem auditable in one query — select count(*) from chunks where n_tokens > encoder_limit is the check that would have caught the regression on the morning it shipped.

Sizing as a budget, not a preference#

Once the ceiling is known, chunk size becomes an allocation across four quantities that move together. Take a 90,000-document knowledge base averaging 4,200 tokens per document, which is 378 million tokens of text.

Every cell in the second and third columns comes out of one line of arithmetic, and it is worth doing once so the table stops looking like a set of quoted results. The corpus is a fixed pile of tokens: ninety thousand documents times forty-two hundred tokens each is the 378 million in the header, and dividing that pile by the chunk size gives the number of chunks, because chunking with no overlap partitions the text and every token lands in exactly one chunk. Divide 378 million by 120 and you get the 3.15 million in the first row; divide it by 300 and you get 1.26 million; the last two rows are the same division again. The index column is then that vector count times the width of a vector, which is 1024 components at four bytes each. Three and a bit million vectors at four kilobytes apiece is the 12.9 GB in the first row, and running the same product down the column reproduces 5.2, 3.0 and 1.5 with nothing left over.

An identity that closes with no residual is worth more than the numbers it reproduces, because of what it rules out. There is no constant term in that model — no per-chunk overhead, no floor, no correction factor — which licenses treating chunk size as a pure allocation dial across the whole table. Halving the chunk size doubles the vector count and doubles the index, exactly, and any real index that behaves otherwise is telling you about its own storage format rather than about chunking.

Two figures in that table are describing different systems and should not be reconciled. The 1024-dim width in the index column is a sizing assumption for a generic model, while the 768-dimensional vector the probe above produces belongs to bge-base-en specifically. Both are correct about what they measure, and substituting one for the other would move every figure in the index column.

The embedding pass is the same size at every row, because you are embedding the same corpus either way, which is the first thing people get wrong when they reason about chunk size and cost. Smaller chunks do not cost more to embed; they cost more to store and to search, and they cost more at query time only through k. What does raise the embedding bill is overlap, which duplicates text: 512-token chunks with 128 tokens of overlap embed 504 million tokens instead of 378 million, a third more, and at the OpenAI rate above that is the difference between roughly $49 and roughly $66 for one pass over the corpus. On a self-hosted GPU it is the difference between about four hours and about five and a half.

The overlap figure is the one place in this table where a step is easy to skip, and it closes exactly. Chunks of 512 tokens overlapping by 128 advance through the document by the difference between those two numbers, so each new chunk contributes that many fresh tokens and repeats the rest. Dividing the corpus by that stride gives the chunk count, and multiplying it back by the full 512 tokens each chunk is embedded at gives 504 million with no remainder. The ratio between the two totals is the ratio between the chunk length and the stride, which is where "a third more" comes from — and because that ratio is the only thing driving the increase, it carries straight through to both costs. A third more tokens at a fixed price per million is a third more money, which is what takes $49 to $66, and a third more tokens on a fixed GPU throughput is a third more time.

The last column is the one that invalidates rows rather than pricing them. At 1,000 tokens against a 512 ceiling, essentially every chunk is half-indexed, and the row's apparently attractive index size of 1.5 GB is 1.5 GB of vectors that describe openings. The 189M in that row is the corpus counted as far as the encoder reads it, rounded to half the total, and the half that is not counted is still in the store waiting to be served. There is no sense in which that configuration is cheaper; it is a smaller index of a corpus half the size, with the other half stored alongside for delivery.

Two neighbouring failures with different fixes#

An encoder limit is a specific failure with a specific fix — keep chunks under the accepted length, and if a structural unit exceeds it, subdivide rather than clip. It is worth being disciplined about not reaching for it as an explanation of every disappointing recall number, because two neighbouring problems look similar from the outside.

The first is a genuine context problem rather than an indexing one. A chunk can be well under 512 tokens, fully indexed, retrieved at rank 1, and still fail, because the text it contains does not answer the question. No sizing choice touches that.

The second is the mirror image of the runbook case and is the reason this chapter should not be read as an argument for large chunks. Sizing every chunk at 480 tokens to stay comfortably under the ceiling produces vectors that average over a lot of unrelated content, and a mean-pooled vector over 480 tokens of mixed material is a weaker match for a specific query than a vector over the 90 tokens that actually answer it. The mechanism is the pooling step again, read in the other direction: an average over many positions is pulled towards whatever the passage mostly talks about, and a specific answer occupying a small share of the positions contributes a correspondingly small share of the result. The ceiling sets an upper bound; it says nothing about where within that bound the optimum sits, and on many corpora the optimum is far below it.

Which sets up the failure at the other end. Chunks small enough to be precise are small enough to separate a claim from the number that completes it, and when that happens the arithmetic of ranks stops behaving the way intuition expects.

The rule to carry out of this chapter is a separation rather than a size. A chunk has a served length and an indexed length, they are equal only below the encoder's limit, and every configuration file in a retrieval system presents them as one setting. Above the limit the served length is whatever you asked for and the indexed length is the limit, so the tail of every long chunk sits in the store, reachable by id, greppable, visible in a record view, and absent from the only artefact that decides whether it is ever retrieved.

Which limit applies is not a fact you can look up, because a model card frequently reports the architecture's positional capacity while your serving stack applies something smaller, and a library upgrade can move it without touching your code. Measure it with a prefix probe, count against it with the tokenizer that model was trained with rather than a general-purpose one, and raise at ingest instead of clipping, so the decision about an oversized section is made by a splitter that can see the document's structure. Then keep the count on the record, because a chunk store carrying its own token count answers the question "how much of this index describes openings" in one query, and a chunk store without it cannot answer that question at all.