Mid · 9 chapters
Chunking strategies
Fixed-size, overlap, recursive, sentence and paragraph, semantic, layout- and heading-aware, late chunking
Chunking choices often dominate retrieval quality, so tune them with controlled measurements rather than treating a popular split size as a default.
Read these first
What you will know
Chunk a corpus three different ways and measure retrieval for each
Hold the corpus, queries, index settings, and evaluator constant so the comparison isolates segmentation effects.
Measure how chunk size affects embedding truncation, rejection, and token cost for the selected model
Embedding endpoints impose maximum input lengths and billing or latency costs that can silently invalidate oversized chunks.
Show over-chunking destroying the context an answer needed
A fact split across adjacent results may require retrieval and synthesis behavior the system does not reliably perform.
Choose heading-aware splitting where the documents have structure
Preserving section boundaries keeps local context and metadata aligned, but only helps when headings are reliable.
Write a starting configuration and a measurement rule that would trigger changing it
A preset is useful only when metrics such as recall, answer attribution, latency, or token cost can disprove it.
Chapters
- 1
Every chunking rule is a bet about where a fact ends
Fixed windows, overlap, recursive splitting on paragraph then sentence, semantic similarity and heading-aware sections are five answers to a single question: where does a retrievable unit stop. Put one page through each and watch the boundaries land in different places, then see what overlap actually buys — a fact straddling a cut showing up whole in one of two neighbours. This vocabulary is what every later measurement is written in.
- 2
A nine-point win with three variables moved measures nothing
Heading-aware beat the others by nine points on recall@10, and that run also switched to top-k 20 and re-embedded with a newer model, so the nine points contains a k effect and a model effect of unknown size. Pin corpus, queries, embedding model, index parameters and k so boundaries are the only difference. You also learn which answers to reject: recall@10 cannot separate a chunk containing the answer from one merely near it.
- 3
The encoder's 512-token ceiling indexes only a chunk's opening
Going from 300 to 1,000 tokens made recall worse for precisely the long chunks, because the encoder accepts 512 and discards the remainder before pooling without raising anything. The vector then describes the opening alone, while the full text is still what gets served the moment that chunk is hit, so every runbook tail is stored and delivered but never indexed. Measure your own model's limit, rejection behaviour and token cost before picking a size.
- 4
Rank 14 is not a near miss, and no chunk size reaches it
The chunk saying claims must be filed within the window defined in section 4.2 lands at rank 1; the chunk holding the 60-day figure sits at rank 14 with top-k of 8. Rank 14 is where a chunk sharing nothing with the query falls, so raising k is a lottery a rephrasing wins back. No size short of the whole document spans the cross-reference, and the model answers from the fluent pointer rather than chasing it.
- 5
Heading-aware splitting
Heading-aware splitting cuts a document at its own section boundaries and labels each chunk with the headings above it. That is worth ten points of retrieval where the headings are real markup a person typed, and costs nine where a layout model inferred them from font size. Measure the structure signal per source format, route on where the structure came from, and evaluate the halves of a mixed corpus separately.
- 6
Semantic chunking cuts where consecutive similarity drops
Embed each sentence or sliding window, walk the document, and cut wherever the similarity between neighbours falls below a threshold or a chosen percentile — the threshold is the entire design, and it is a corpus-specific number, not a default. The ingest cost is structural rather than incidental: an embedding pass over every sentence has to finish before a single chunk exists, which is where a 40% rise in p95 ingest comes from. That tax is not tonight's batch. It is charged again on every embedding-model upgrade, every re-parse and every configuration you wanted to compare, which is how a team quietly stops measuring chunking at all. On well-paragraphed prose its boundaries often land where the paragraph breaks already were.
- 7
Late chunking changes the vectors and nothing else
Documents that name an entity once and then say 'the company' leave every later chunk scoring badly on the entity's name. Late chunking pushes the whole document through the embedding model first, so each token representation has attended to the rest, and only then applies boundaries, pooling each vector from context-carrying tokens. The served text is unchanged. It repays a full re-embed only where chunks are genuinely context-dependent and the document fits the window.
- 8
A heading prefix buys most of what late chunking promises
The same anaphora problem has a far cheaper answer: before embedding, prefix each chunk with its document title and section heading, or with a generated one-line statement of what the chunk is about, so the entity name and the section scope are inside the text being embedded. There is no new embedding model, no context-window ceiling and no ingest rework. It is also the experiment that falsifies a three-week migration before you fund it — if prefixing recovers most of the recall on the query class that motivated the proposal, the corpus's context-dependence was the shallow kind and late chunking has little left to buy. Measure both on the same held-out slice.
- 9
A starting configuration needs a rule that can disprove it
For API reference pages, splitting on headings with a token ceiling that recursively subdivides any overrunning section is defensible — but only when paired with the condition that would retire it. Commit in advance: recall@k on a held-out set dropping below the fixed-size baseline, or cited chunks ceasing to contain the answer span. A preset no metric can contradict is a preference, and it outlives every corpus it stopped fitting.
External resources
- Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Modelsarxiv.org
The embed-then-chunk inversion and the recall it buys.
- Chunking strategies for RAG tutorial using Granite | IBMibm.com
This is a substantive hands-on tutorial that explains fixed-size, recursive, semantic, document-based, and agentic chunking and provides implementation steps and code for experimenting with them in a RAG pipeline.
- The 5 Levels of Text Splitting (notebook)github.com
Implements fixed, recursive, semantic and agentic splitting side by side.
- Chunking Strategies for LLM Applicationspinecone.io
Decision guide for size, overlap and structure-aware splitting.
- The 5 Levels Of Text Splitting For RetrievalGreg Kamradt
The reference tour of chunking strategies, simplest to most adaptive.
- Understanding late chunking in RAG systems (for beginners!)Weaviate vector database
Makes late chunking concrete against ordinary fixed-size chunking.