Rewriting can invent a constraint; history embedding cannot

There are two obvious ways to get conversational context into a retrieval query, and both are about fifteen lines of code. History embedding concatenates the recent turns into one string and searches with that string's vector. Rewriting calls a small model to produce a standalone query from the transcript, and searches with the vector of that query alone. Teams argue about which is better on the strength of how each one reads on a whiteboard, and the argument does not settle, because the two differ along several axes at once and each side is looking at a different axis.

With the 300-item follow-up set from chapter 3 in place, the parts of that argument that are measurable become a measurement. Team A takes the first approach, Team B the second. The parts that are not measurable — what each leaves behind when it fails, what each sends to a third party, which failures each one is capable of at all — are the parts that decide the deployment, and they are the reason the recall column is a poor summary of the table it sits in.

# A: history embedding
def query_vector_a(history: list[Message], latest: str) -> list[float]:
    turns = history[-4:] + [Message("user", latest)]
    joined = "\n".join(f"{m.role}: {m.content}" for m in turns)
    return embed(joined)          # 1 embedding call, same as a normal query

# B: rewrite, then embed
def query_vector_b(history: list[Message], latest: str) -> list[float]:
    rw = contextualize(history, latest)   # small model, JSON out (chapter 2)
    log.info("rewrite", extra=rw.model_dump())
    return embed(rw.query)                # 1 model call + 1 embedding call

Run on the same 300 items, against the same index, with top_k=10:

The third column is the option people forget: embed the follow-up alone and the last turn alone, run both searches, and fuse the two ranked lists with reciprocal rank fusion, which scores each document by the reciprocal of its rank in each list and adds those scores, so a document that either search ranked well survives into the merged list. It costs one extra vector search and needs no model. Of the eight recall points separating the first two columns it recovers three, which makes it a serious answer for a latency-bound product and a poor one for anything you will later have to debug.

What a rewrite costs per turn, and where the cost sits#

The cost line is worth deriving rather than quoting. A four-turn transcript plus the system prompt runs about 450 input tokens and the JSON rewrite about 60 output tokens; at small-model prices in the region of $0.80 per million input and $4 per million output tokens, that is roughly $0.0006 a turn, or $240 a month at 400,000 conversational turns.

Take that in two pieces, because the pieces are more useful than the total. Divide the input tokens by a million and multiply by the input price to get the input side of the turn; do the same with the output tokens and the output price. Those two products add to exactly the quoted per-turn figure with nothing left over, which is what licenses reading $0.0006 as the whole model cost of a turn rather than a rounded input cost with an output cost still to come. Multiplying that figure by the monthly turn count reproduces the $240 exactly as well, so the per-turn price and the monthly bill are one estimate stated twice rather than two that could disagree. The only ways to move either of them are the two token counts and the two prices, all four of which you can look up or measure.

The split inside the total is the part worth carrying. Output is about a seventh of input by token count and priced five times higher, so the input side comes to three fifths of the turn and the output side to two fifths. A lopsided-looking prompt is therefore closer to balanced in money than it looks, and the two sides move under different levers. Lengthening the system prompt or widening the history window moves the cheap side. Letting the model write more than a query moves the expensive side, which is why a cap on output tokens is a cost control and not only a latency control.

The latency rows are the ones that actually decide deployments, and they have to be read across rather than down. The rewrite adds 320 ms at the median and 1,150 ms at the ninety-ninth percentile, so its tail runs more than three times its median. The fusion column adds 9 ms and 14 ms, a spread of a few milliseconds. Comparing the two on the p50 row alone compares them at the point where they are most alike, and it hides the property that generates complaints: a user experiences the slow request, not the average one. The 320 ms is also the more damaging kind of latency, because it lands in front of everything else in the request and is visible as the gap before streaming starts.

The artifact you can read#

The strongest reason to run the rewrite is that it produces a string. When a follow-up returns the wrong passages under strategy B, the log contains What is the uptime SLA for the Frankfurt region? next to a transcript in which the user clearly meant Dublin, and the failure has been localised in the time it takes to read one field. Under strategy A the same failure leaves behind a 3,072-dimensional vector and a list of chunk ids, and the only honest thing you can say is that retrieval returned Frankfurt pages for a query about a conversation involving Frankfurt.

The step under the word "inspectable" is worth making explicit. A query string is in the same representation as the thing it is supposed to be about: the conversation is text, the query is text, and anyone who can read the first can check the second against it by eye. A query vector is in a representation with no such correspondence — its components do not name anything a person can point at, and no operation on it answers "does this still mention Dublin" without running it back through the model that produced it. So a failure under strategy A can be localised only by re-running the system with instrumentation added, while the same failure under strategy B is localised by reading one logged field.

That difference compounds into everything built on top of it. A rewrite can be diffed across deployments, so a prompt change that starts dropping plan qualifiers shows up as a diff on 40 items rather than as a two-point dip. It can be asserted on in unit tests, so the ordinal-off-by-one case gets a regression test that costs one model call to run. It can be shown to a support engineer who does not know what an embedding is, and they will immediately say "that's the wrong region". None of those are available for a vector.

The measurement in chapter 3 that distinguished a retriever regression from a contextualization regression depended on exactly this. Without a readable query, the follow-up set gives you one number that moves for two reasons, and every investigation begins by trying to reconstruct what the system was searching for.

The error class the rewrite has and history embedding does not#

The case against calling rewriting strictly better is that it introduces a failure mode with no counterpart on the other side. A rewriter can drop a qualifier the user still means, and it can add one they never said, and in both cases the artifact it leaves behind is a well-formed, confident, entirely plausible query.

{"turn": 6,
 "user": "and what's the rate limit on the export API?",
 "query": "What is the rate limit for the billing export API on the free tier?",
 "resolved": [{"surface": "the export API", "antecedent": "billing export API", "turn": 4}],
 "dropped": []}

The user is on the enterprise plan, said so in turn one, and never mentioned a tier in turn six. free tier came from the model, which has read a great deal of API documentation in which rate limits are discussed per tier and for which the free tier is the most frequently written-about case. The mechanism behind that is the same one that makes rewriting work at all. The model is asked to produce a query that stands on its own, completeness is the property it is being scored on by its own training and by the prompt, and a rate-limit question without a tier reads incomplete. Recovering a qualifier the transcript contains and supplying one it does not are the same move from inside the model: both make the query look finished. Nothing in the instruction distinguishes them, because the distinction lives in the transcript rather than in the shape of the output.

The tempting belief here is that structured output closes this hole. The rewrite is logged, the resolved array names every substitution, the dropped array names every omission, and it feels as though a failure that is written down in full cannot hide. Reading the record is what closes the hole, and nothing prompts anyone to read it, because the number that would send them looking does not move.

Work out why it does not. Recall@10 asks one question of an item: is a gold passage anywhere in the top ten. A fabricated tier changes which passages come back and does not reliably push the gold passage out of ten slots, since the free-tier rate limit table and the enterprise one are near neighbours in the index and a query naming either tends to retrieve both. An item whose query names the wrong tier can therefore score a hit while the answer built from it is wrong. The metric takes the same value in the world where the constraint survived and in the world where the model invented one, which is the invariance chapter 3 derived on the standalone benchmark showing up again from a different direction.

Recall is not the only quantity that fails to move, and the fuller version of that is the belief this section is aimed at: that a chain of individually correct stages is a correct chain. Check the quantities a downstream measurement could look at. The well-formedness of the query is identical whether the tier was said or invented, since the model writes a clean sentence either way. The relevance of the retrieved passages to that query is identical, since the passages genuinely answer the query that was issued. The faithfulness of the answer to those passages is identical, since the generator quotes them correctly. Every one is a real, measurable property, and not one of them distinguishes the two worlds, because each is evaluated against the artifact handed to that stage rather than against the conversation. The two worlds differ only between the transcript and the query, and the only instrument that sits there is the rewrite itself.

Dropping is the same failure with the sign flipped and is more common. On the 300-item set, hand-auditing every rewrite found 11 cases (3.7%) where a constraint present in the conversation and still live was omitted, and 4 cases (1.3%) where a constraint appeared that no turn contained. Both rates are low enough to be invisible in the recall number and high enough that a system doing 400,000 turns a month produces around 20,000 silently mis-constrained queries in that month.

Those figures close on each other, and checking that they do resolves an ambiguity in the sentence above. Fifteen bad rewrites out of three hundred is exactly one turn in twenty; the two rounded percentages add to exactly five in a hundred as well; and one turn in twenty of the monthly volume is exactly the quoted monthly count. So the monthly figure is the sum of both error classes rather than the fabrication count alone, and the two classes are disjoint — every audited item is clean, dropped or fabricated, with no item counted twice. Because nothing is left over, the monthly figure has exactly three inputs, all of them measurable: the drop rate, the fabrication rate and the turn volume. It moves when one of those moves and for no other reason.

Invisibility in the recall number is itself derivable, using the standard-error recipe from chapter 3. Multiply 0.86 by its complement, divide by the same 300 items, take the square root, and the noise on a single run comes to about two points. The fabrication rate is smaller than that, so even a fabrication that cost a hit every single time would sit inside the run-to-run noise, and the previous paragraph says it does not cost a hit every time. Two independent reasons, then, for the same conclusion: the metric is too coarse to see the effect, and it is measuring the wrong object even when it is fine enough.

History embedding cannot do this. It is lossy, and the loss has a mechanism worth stating: embedding a concatenated string produces a single point meant to summarise the whole string, so every token added competes for room in the same fixed set of components. The follow-up's six tokens are a small fraction of a 400-token concatenation, and everything gets averaged toward whatever dominates the string. What that produces is a dilution of signal rather than an invention of one. A vector built from the actual conversation cannot carry a constraint the conversation never held, because there was no step at which anything was generated. That property is worth naming precisely because it is the only thing strategy A has that strategy B does not, and it is what makes the choice a trade rather than a ranking.

The claim needs its condition, and the condition is the difference between presence and salience. What strategy A guarantees is that no constraint absent from the conversation is present in the query. It guarantees nothing about a constraint that is present in the conversation and has been diluted until it no longer influences which documents come back: a live enterprise buried in turn one of a long transcript can contribute so little to the concatenated vector that the free-tier table is retrieved anyway. That is a loss rather than an invention, and it produces the same wrong document and the same confident answer. The distinction still matters, since a diluted constraint is recoverable from the input and a fabricated one is not, and the step to make somebody demonstrate is that recovery: show the constraint in the transcript the vector was built from, and show what its weight in that string was.

Dilution has a shape you can measure. On items whose prefix turns totalled under 100 tokens, strategy A scored 0.84, close to the rewrite; on items whose prefix exceeded 200 tokens, it fell to 0.61. Long assistant turns are the usual cause, and they are exactly what a good assistant produces, so the approach degrades as the rest of the system improves.

Those two figures are conditional scores on two subsets that do not partition the set, and the gap between the subsets is where the unreported items live. Anything whose prefix landed between the two thresholds is in neither subset and no score is given for it. Two consequences follow, and the second is the one that gets skipped. The overall 0.78 cannot be reconstructed from 0.84 and 0.61, so the pair says nothing about how much of the set — still less how much of your traffic — sits in the long-prefix regime. What the pair does support is a claim about direction and size: prefix length is doing real work, and the swing across it is wider than the entire gap between the two strategies. Turning that into an exposure estimate needs the length distribution of your own conversations, which is a measurement on your logs rather than a reading off this table.

There is a partial defence against fabrication, and it is worth knowing its limits. Because the rewriter returns structured output, you can check the emitted query against the transcript mechanically: extract the content words the rewrite added, and flag any that appear neither in the user's turns, nor in the assistant's, nor in a small allow-list of corpus aliases. On the audited set that check caught 3 of the 4 fabrications and fired on 9 items where the added word was a legitimate synonym the rewriter had introduced deliberately. A 3-in-4 catch rate at a 3% false-positive rate is a reasonable monitoring signal and a bad gate.

Those are counts over one set rather than independent estimates, which is worth confirming before leaning on them: the flagged legitimate items divided by the audited item count give the quoted false-positive rate exactly, and the two audit counts over the same denominator reproduce the two earlier rates to the precision printed. The catch rate and the false-positive rate can therefore be combined.

The reason it is a bad gate is arithmetic on those two numbers. Across the whole set the check raises twelve alerts, three of which are fabrications, so blocking on it would reject three correct queries for every wrong one it stopped. As a queue for a human to look at, twelve items per run is short enough to read. The residual is in the check's own definition: a fabrication escapes whenever the invented word already appears somewhere in the transcript in a different sense, or already sits in the alias allow-list, and one of the four did escape that way. No amount of tuning the word lists removes that case, because the check has no notion of what the word was doing where it found it.

When the conversation contains other people's data#

The privacy comparison is usually stated as "rewriting sends less data", and at the point where the model is called that is false, because the rewriter receives the same transcript the concatenation would have embedded. The difference is downstream of that call, and it is structural rather than a matter of volume. Volume is the wrong frame throughout: the question that decides this is how many boundaries the text crosses and whether any of them has somewhere to stand.

Consider a chat assistant over customer support tickets, where the assistant's turns quote ticket bodies containing names, email addresses, order numbers and occasionally a customer's account complaint verbatim. Under history embedding, that text becomes the query. It is sent to the embedding endpoint on every turn, it appears in the vector database's query logs, it flows into whatever analytics pipeline records queries for relevance tuning, and it lands in application performance monitoring (APM) traces, the per-request timing records your observability stack keeps, where query strings are captured by default. There is no point in that path where a person could reasonably be asked to redact it, because the query is the conversation.

Follow that last step rather than accepting it. The query string is the payload every one of those systems is designed to record, since a query is the natural unit of a search request. Under strategy A the query string is byte-for-byte the transcript. A redaction step would therefore have to alter the query, and altering the query alters what is retrieved, so redaction and retrieval are in direct conflict and no ordering of the two satisfies both. That makes it a property of the design rather than an omission somebody can go and fix.

Under rewriting there is a chokepoint. The rewrite is a short string, produced by a step you control, and you can run a redaction pass over it before it goes anywhere; you can assert that it contains no email addresses; you can log it in full without logging the transcript. The rewriter itself can be a 3B-parameter model running inside your own network, which means the raw transcript never leaves, while the embedding call — which usually is a third-party API — receives only the sanitised rewrite. That architecture is unavailable to strategy A, where the payload sent to the embedding provider is by construction the raw conversation.

The bottom row carries the argument, and it is a statement about architecture rather than about degree. Strategy A has no step between the conversation and the embedding provider at which any code of yours holds a version of the query differing from the transcript; strategy B has exactly one, and everything above is bought with it. So personal data strengthens the case for rewriting, for reasons that have nothing to do with the recall points and everything to do with having a place to stand between the conversation and the rest of the infrastructure.

Trimming the tail without losing the follow-ups#

Two reductions to the latency are routine. Capping max_tokens at 150 removes the case where the model decides to explain itself, which accounts for most of the tail. Skipping the call entirely when the latest message contains no referring expression and no elided predicate — a cheap regex-plus-heuristic gate that fires on roughly 55% of turns — halves the average added latency and removes the over-rewriting damage measured in chapter 2 on those turns at the same time.

The averaging works out as follows. A skipped turn adds nothing, so the mean added latency across all turns is the share of turns that still call the model multiplied by 320 ms. With the gate firing on more than half of them, the remaining share carries the full cost and the mean lands a little under half of where it started. What the gate does not move is the experience of a follow-up turn, and the shape of the distribution makes that precise. Skipping turns removes them from the bottom of the ordering and leaves the slow ones exactly where they were, so the ninety-ninth percentile of all turns now falls at roughly the ninety-eighth percentile of the turns that still fire, which is still deep in the tail. A user asking a genuine follow-up pays the same 320 ms and the same 1,150 ms as before. The gate improves a dashboard number and leaves the requests it was supposed to be about untouched.

That asymmetry sets the direction the gate has to be conservative in. Sending a standalone query to the rewriter costs latency on a turn that did not need it. Skipping a follow-up costs a wrong answer, and it costs it silently, since the un-rewritten follow-up retrieves something and the generator answers from it. A gate tuned to fire more often trades an error that shows up in a percentile for an error that shows up in an escalation.

Where each one is the wrong tool#

Rewriting is poor value on conversations that are short and clean. On the subset of items whose prefix was a single turn with a pronoun antecedent, strategy A scored 0.89 and strategy B scored 0.91, which does not justify 320 ms and an extra dependency in the request path. Systems whose traffic is mostly two-turn exchanges can reasonably run the fusion approach and spend the latency budget elsewhere.

That subset also says something about the headline gap, and it is worth doing the algebra rather than eyeballing it. The whole-set gap is a weighted average of the gaps on the subsets that make up the set. One of those subsets carries a two-point gap, and the average over everything is eight points, so the gap on the remainder has to be larger than eight points — larger by more, the bigger the easy subset is. The eight points are therefore a property of the item mix in that set rather than a property of the two strategies, and a product whose traffic looks like the easy subset will measure a much smaller gap and be right to.

History embedding is the wrong tool anywhere the answer has to be auditable or the conversation has to be kept out of a vendor's logs, and it is also the wrong tool in any product where users work through lists across many turns, because dilution grows with every turn the assistant contributes. The requirement in the first case is auditability and in the second it is stable recall over long prefixes; which strategy you pick is a design decision, and the two are worth keeping apart when somebody proposes tuning their way out of one of them.

Neither one addresses what happens when the user changes the subject. Strategy A drags the old topic in by construction, since it is embedding turns about it. Strategy B drags it in for a more interesting reason: the rewriter is looking at a history in which nothing distinguishes a live entity from a dead one, and the rewrites it produces in that situation read perfectly well right up until you check them against what the user asked.

The recall column is the smallest difference between these two approaches and the one every comparison leads with. What separates them is that one of them leaves a readable object behind and the other does not, that one of them has a place to stand between the conversation and a third party and the other does not, and that one of them can state a constraint nobody ever said. The first two favour rewriting and the third does not, which is why a system that runs rewriting has to run something that reads the rewrites — a sampled review queue, a diff across deployments, a regression test on the cases already known to be hard. Choosing the approach with an artifact and then never reading the artifact buys the failure mode without the defence.