Semantic chunking cuts where consecutive similarity drops

Semantic chunking cuts a document where the text stops being about what it was about. It does this without reading a single heading or paragraph mark. Each unit of text — a sentence, or a small sliding window of sentences — is turned into a vector, an ordered list of numbers produced by an embedding model, whose direction summarises what that text is about. Walking the document then gives a series of consecutive similarities: for each adjacent pair of units, one number saying how nearly the two point the same way. A high number means the text carried on with its subject. A dip means it did not, and a dip is where the splitter cuts.

Which dips count is a decision, and it is the only decision the strategy exposes. A scanned half of a mixed corpus has no structure signal worth routing on — a layout model's inferred headings being too unreliable to split on, which is where chapter 5 leaves such documents — so similarity-based cutting is the obvious candidate for it. The first run uses the 95th-percentile breakpoint from a tutorial and produces chunks averaging 640 tokens, a third of them over the encoder ceiling of chapter 3, where a chunk longer than the model's input limit is indexed by its opening alone. Changing one argument to the 80th percentile produces chunks averaging 138 tokens on the same documents with the same model. Both runs are semantic chunking; the mean chunk length differs by 4.6×.

That is the honest starting position. The mechanism from chapter 1 — embed the units, take consecutive cosine similarities, cut where they dip — has one free parameter, and that parameter determines everything the strategy does. There is no configuration of it that is neutral or default, and a number copied from a notebook that was demonstrating the idea on Paul Graham essays is a number about Paul Graham essays.

Where the dip comes from#

Before arguing about thresholds it is worth being precise about what quantity is being thresholded, because the argument is unreadable otherwise. Two adjacent units of text go into the same embedding model and come out as two vectors. The number placed between them is the cosine similarity: the cosine of the angle between those two vectors, which runs from one when they point the same way, through zero when they are at right angles, to minus one when they point opposite ways.

Computing it is one pass of multiply-and-add. Line the two vectors up component by component, multiply each pair, and sum the products — that sum is the dot product, and it equals the length of the first vector, times the length of the second, times the cosine of the angle between them. On its own that is not yet a usable measure of agreement, since two of its three ingredients are lengths and have nothing to do with direction. The way out is to make the lengths stop mattering, and the code below does it in one argument: normalize_embeddings=True rescales every vector to length one before it is returned. Both lengths in the product are then one, one times one is one, and the two length terms cancel exactly out of the identity. What remains is the cosine of the angle and nothing else.

That exact cancellation is what licenses everything the rest of the chapter does with these numbers. Because no length survives into the result, a similarity computed between two long sentences is measured on the same scale as one computed between two short ones, and the whole series for a document is comparable to itself end to end. Taking a percentile over that series, or a standard deviation, or a ranking of the biggest falls, is a meaningful operation only because every entry in it is an angle.

The next step is why a dip in that series is evidence about the document rather than about the model. An embedding model is trained so that two pieces of text a person would say are about the same thing come back as vectors pointing in similar directions. Adjacent sentences inside one passage are, by construction, about the same thing — that is what a passage is. So their vectors point similarly and the cosine between them is high. When the author finishes with one subject and starts another, the sentence before the join and the sentence after it are about different things, their vectors point in different directions, and the cosine between them falls. Running down the series, a local minimum therefore marks a place where the text stopped being about what it was about, which is exactly the property a chunk boundary is supposed to have.

Two honest limits sit on that chain, and both matter later. The dip is evidence of a subject change and not a measurement of one, because the cosine also falls for reasons that have nothing to do with subject — which is what the window argument below exists to suppress. And nothing in the series carries a notion of importance, so no rule applied to it can say whether a change is large enough to deserve a boundary.

The threshold is the design#

Four ways of turning the similarity series into cut points are in common use, and they fail in different directions.

Chapter 1 gave the concrete version of the first row: the tenth percentile of consecutive-sentence similarity on a policy page came out at 0.612, while on terse API reference lines the same statistic sits near 0.88. An absolute threshold of 0.612 makes reasonable cuts on the first corpus and zero cuts on the second, which produces one chunk per document and an index of 4,000-token vectors describing their openings.

There is a direction word in this material that trips almost everybody, and it is worth settling before the code arrives. The two runs at the top of this chapter are quoted as the 95th and the 80th percentile, and the higher of those two numbers produced the longer chunks. The code below takes pct=15 and cuts where similarity is below that percentile. Those look like contradictory conventions and they are the same convention seen from two ends. A tutorial setting names a percentile of the drop — cut only at falls in the top few percent of all falls — while the code names a percentile of the similarity, cutting where the similarity is in the lowest few percent. The highest drops and the lowest similarities are the same joins. Chapter 1's summary of this strategy states the correspondence outright, describing a ninetieth-percentile setting as cutting where neighbour similarity is in the lowest decile. So a 95th-percentile setting cuts at the smallest sliver of joins and produces long chunks; an 80th cuts at a fifth of them and produces short ones. Neither figure is wrong, and reading either one without asking a percentile of what is how a configuration gets copied with its meaning inverted.

That correspondence also makes the fourfold spread at the top of the chapter predictable rather than surprising. Model a document crudely as a run of joins of which a fraction p are eligible to be cut; the number of chunks is then roughly p times the number of joins, and the mean chunk length is roughly one over p. Moving from cutting the top twentieth of falls to cutting the top fifth multiplies p by four, so the model predicts mean chunk length falling by a factor of four. The measured figure is 4.6×. A one-parameter model that predicts fourfold where the measurement says a little over four and a half has got the direction and the order of magnitude from arithmetic alone, and it has a real residual that the chapter's figures cannot attribute — the model ignores the per-document scope of the percentile and the minimum-length floor, and neither effect is measured here. The useful part is what survives the residual: this parameter does not adjust chunk length at the margin, it sets it.

The percentile rule avoids the absolute rule's scale problem by being relative, and it buys the scale-invariance with a different flaw. It has no notion of "this document does not need cutting". A three-paragraph release note about one bug fix has 11 consecutive similarities, a lowest decile of one of them, and gets cut there, at whatever the weakest link in a perfectly coherent document happens to be. Percentile rules answer "where are the n weakest joins" and never "are any of these joins actually weak", which is a different question and the one you meant.

That flaw is worth naming as the standing mistake of this strategy, because the property that causes it is the property the rule is chosen for. Scale-invariance is genuinely attractive: a percentile means the same thing on a policy page sitting near 0.612 and on API reference lines sitting near 0.88, so it looks like the setting that transfers, and the table above says so in its third column. What the column does not say is that a rule which always cuts the same fraction of joins is a rule held constant against the one thing it is supposed to discriminate — how much cutting this particular document needs. A tightly coherent document and a document covering six unrelated subjects get the same proportion of cuts, and the rule cannot tell you which one it just did the wrong thing to. Transferring across corpora and discriminating within a corpus are different capabilities, and this rule was selected for the first.

The practical configuration combines a relative rule with hard bounds, because the bounds are what keep the output usable regardless of what the rule decides.

import numpy as np

def semantic_chunks(sents, model, window=3, pct=15,
                    min_tokens=120, max_tokens=480, count=len):
    # window smoothing: compare groups of `window` sentences, not single ones
    groups = [" ".join(sents[max(0, i - window // 2): i + window // 2 + 1])
              for i in range(len(sents))]
    v = model.encode(groups, normalize_embeddings=True, batch_size=256)
    sims = (v[:-1] * v[1:]).sum(axis=1)

    cut = np.percentile(sims, pct)
    chunks, buf, n = [], [], 0
    for i, s in enumerate(sents):
        buf.append(s); n += count(s)
        weak = i < len(sims) and sims[i] < cut
        if (weak and n >= min_tokens) or n >= max_tokens:
            chunks.append(" ".join(buf)); buf, n = [], 0
    if buf:
        chunks.append(" ".join(buf))
    return chunks

The two lines doing the arithmetic repay being read slowly. v is a stack of unit-length vectors, one per position in the document. v[:-1] is that stack with its last row dropped and v[1:] is the same stack with its first row dropped, so lining the two up puts every unit opposite its successor. Multiplying them elementwise and summing along the component axis with .sum(axis=1) performs one dot product per adjacent pair, which — the lengths having already been normalised away — is one cosine per pair. sims is therefore the similarity series itself, one entry shorter than the document, because the last unit has no successor to be compared against. np.percentile(sims, pct) then reads a threshold off that series rather than off a constant, which is the percentile rule written out.

Everything after that line is bounds rather than rule. min_tokens refuses a cut the rule wanted, on the grounds that the chunk so far is too short to stand alone; max_tokens forces a cut the rule did not want, on the grounds that the chunk has grown past what the encoder will index. Those are two design decisions sitting on top of one measurement, and they are worth distinguishing from it: the similarity series is what the documents say, while the floor and the ceiling are what the retrieval system needs, and only the second pair is negotiable by argument.

The window argument matters more than its obscurity suggests. Comparing single sentences is noisy, because a one-clause sentence like "This is important." has a vector dominated by its syntax rather than its subject, and it produces a similarity dip that has nothing to do with a change of topic. On a scanned corpus, the standard deviation of consecutive similarity was 0.11 with single sentences and 0.061 with a three-sentence window, and the number of breakpoints that a human annotator agreed with rose from 54 percent to 71 percent. The smoothing costs nothing except that boundaries become fuzzy at the sentence level, which does not matter, because min_tokens was going to move them anyway.

Those two standard deviations are the mechanism of the smoothing, stated as a number. Averaging a sentence together with its neighbours before embedding means each vector is built from more text, so the idiosyncrasy of any one sentence is diluted by the sentences either side of it. Dilution shrinks the spread of the series, and the measured spread duly falls to a little over half of what it was. The agreement figure is what says the shrinkage removed the right variation: had smoothing suppressed real subject changes along with the noise, agreement would have fallen with the spread rather than risen by seventeen points.

The cost is in the shape of the pipeline#

Fixed-size, recursive and heading-aware splitting all run on text and finish before any model is loaded, so ingest is a single embedding pass over the chunks. Similarity-based cutting inserts an embedding pass over every sentence before the first chunk exists, and the second pass cannot start until the first has finished for that document.

That ordering is a requirement rather than an engineering choice, and the reason is a data dependency. The input to the chunk-embedding pass is the list of chunks. The list of chunks is an output of the cutting step. The cutting step consumes the similarity series, which is an output of the boundary pass. Each stage needs the previous stage's result for the same document, so no amount of concurrency removes the serial chain — it can only run different documents' chains at the same time. Every other splitting strategy computes its boundaries from the text alone, which is why for those the chain has one model pass in it instead of two.

Per document that is 40 percent. That figure is not an extra measurement: divide the semantic column's bold row by the fixed column's bold row and the answer lands a shade under one and two fifths, which is the stated 40 percent once rounded to the nearest whole one. Knowing it is a ratio of exactly two numbers is what makes it diagnosable — every argument about whether the tax is affordable is an argument about one of those two figures, and any change to either moves the headline directly.

The stage rows above the bold one do not add up to it, and that is expected rather than a slip in the table. A p95 is the 95th percentile of total per-document time, and percentiles do not add: the document sitting at the 95th percentile of the total is not the document sitting at the 95th percentile of every stage. Sum the fixed column's stages and the total falls three hundredths of a second short of its bold row; sum the semantic column's and it falls seven hundredths short. Both gaps point the same way, which is what a slow tail looks like when it is slow for more than one reason at once. Reading the stage figures as a decomposition of the bold row is the way to talk yourself into a correction the table does not need.

Over a 90,000-document knowledge base that 40 percent is the difference between roughly four hours of GPU time and roughly five and three-quarter hours, and the extra is not proportional to the extra text, because there is no extra text — the same 378 million tokens go through the encoder twice. The boundary pass is worse per token than the chunk pass despite being the same volume, since sentences are short and padding to the batch's longest member — every input in a batch is padded out to the length of the longest one so the tensor stays rectangular — wastes a substantial fraction of every forward pass. Grouping sentences into the sliding window helps here as a side effect; batching by length helps more.

The 40 percent and the pair of hour figures in that paragraph do not describe the same proportional rise, and the mismatch is informative rather than a contradiction. The percentage is an increase measured on a 95th-percentile document; the batch figures are an increase measured on an average one, since a total over ninety thousand documents is set by the mean and not by the tail. A second encoder pass adds a cost that depends on the document's length rather than on how slow that document already was, so if the addition is roughly the same for a typical document as for a slow one, the same absolute addition is a larger fraction of the smaller base. The average document is faster than the 95th-percentile document by definition. The batch total should therefore rise by somewhat more than 40 percent, and in these figures it does.

There is a cheaper variant worth knowing about, which is to run the boundary pass with a much smaller model. Detecting a topic shift is an easier task than retrieval, and bge-small-en-v1.5 at 384 dimensions runs the boundary pass in about 0.24 s rather than 0.71 s while agreeing with the large model's breakpoints 92 percent of the time. That drops the p95 penalty from 40 percent to about 12 percent, and it is the first thing to try before arguing about whether 40 percent is affordable.

Why a smaller model is allowed to be enough is the part worth holding on to. The boundary pass is not asked to rank documents against a query. It is asked whether two adjacent stretches of text are about the same subject, and it only has to get the ordering of the similarity series roughly right, because the percentile rule reads a rank rather than a value. A model too coarse to separate two plausible answers to a question can still be sharp enough to notice that the text has changed subject, which is why the agreement figure between the two models is high while their retrieval quality is not comparable at all.

Why 40 percent is not a one-off#

The reason to be careful with an ingest regression is that ingest is not a thing you do once. Every event in the following list re-runs the whole pipeline over the whole corpus:

Upgrading the embedding model, which happens when a better one ships or when the current one is deprecated by a provider. Re-parsing after an extractor fix, which happens whenever someone finds a document class the parser was mangling — and after chapter 5, whenever the layout model improves. Changing any chunking parameter, which is the whole activity this topic is about. Rebuilding after a schema change, a vector store migration, or a dimension change.

A team that does four of those a year at four hours each is spending 16 GPU-hours; at 5.75 hours each it is spending 23. Those two totals are the same two per-run figures multiplied by the same count, with no residual anywhere: four times four is sixteen, and four times five and three-quarters is twenty-three, and the gap between them is four times the one-and-three-quarter-hour gap of a single run. That alone is not the problem. The problem is the second-order effect on the experiment from chapter 2, where comparing three configurations meant three re-embeds. Under semantic chunking each of those three carries the boundary pass as well, so the comparison that cost twelve GPU-hours and three engineer-days now costs seventeen and change, and the engineer-day per configuration does not shrink. The same two per-run figures again: three times four is twelve, and three times five and three-quarters is seventeen and a quarter. Comparisons that cost more get run less often, and the practice of measuring chunking at all is the thing that quietly stops. That is the real bill: not the 1.75 extra hours tonight, but the sweep nobody proposes next quarter because the last one took a week.

Every figure in that paragraph is a per-run cost multiplied by a count of runs, and that multiplication is what decides whether an ingest regression is a one-off or a rate. A cost paid once is weighed against the benefit once. A cost paid on every re-embed is weighed against the benefit divided by however many rebuilds the system sees, which is a number nobody has in advance. The list above is what makes that count large.

Where the similarity signal was already free#

The uncomfortable result, and the one that decides most of these arguments, is what happens when you compare semantic boundaries against paragraph boundaries on prose that has paragraphs. On the Confluence half of the mixed corpus chapter 5 routed by provenance, 71 percent of semantic breakpoints fell within one sentence of an existing paragraph break, and recursive splitting on paragraph-then-sentence reached 0.63 span containment against semantic chunking's 0.65 — two points, for double the ingest cost and an extra parameter to maintain. Well-edited prose changes subject where its author pressed Enter twice, and a splitter that reads \n\n gets that signal for free.

That agreement figure and that two-point gap are two independent readings of the same fact, and their agreeing is what makes the conclusion safe to act on. One is a boundary-position measurement made by comparing two lists of offsets and never running a query; the other is a retrieval measurement made by running queries and never looking at an offset. The two share no step. When a positional measurement says most of the boundaries were already there and a retrieval measurement says the strategies score within a couple of points of each other, the second is what the first predicts, and neither had to be taken on trust.

Semantic chunking earns its cost where the typographic signal is absent or lying. Meeting transcripts and call recordings have no paragraphs at all, only speaker turns that change every few seconds while the subject holds for minutes. OCR output has paragraph breaks that are artefacts of line-spacing heuristics rather than authorial intent, so \n\n is noise. Chat exports, subtitle files, and anything reflowed from a two-column layout are in the same position. On the scanned half of this corpus, where chapter 5's routing sent everything to recursive splitting as a fallback, semantic chunking reached 0.58 against recursive's 0.51 — seven points, which is worth 40 percent of an ingest run in a way that two points is not.

Two points on well-paragraphed prose and seven points on OCR output are not two noisy estimates of one effect. They are one mechanism producing opposite recommendations on documents that differ in a single respect, so a corpus-wide verdict averaged over both would have been worthless in either direction.

That split is also the shape of the general rule. Similarity-based cutting is a way of recovering a segmentation signal that the document failed to encode, so its value is roughly the value of the signal you are missing, and on documents that encoded it properly there is nothing to recover. The cost is fixed by the shape of the pipeline and charged on every rebuild; the benefit is set by how much segmentation information the source documents threw away. Buy the strategy where that second quantity is large, and the only way to find out where it is large is to measure the halves apart.

What none of the strategies so far has touched is what each chunk's vector knows. A perfect boundary around section 3.1 still produces a vector built from section 3.1's text and nothing else, so if that section says "the company" where the first page said the company's name, the boundary was correct and the vector still cannot match the name.