'What about the second one?' carries no retrievable signal

Multi-turn query contextualization is the step that sits between a conversation and a retriever and turns whatever the user just typed into a query that stands on its own, meaning a query that still identifies what is being asked about when it is read with no transcript attached. A turn is one message in the exchange, so a multi-turn conversation is the ordinary case of several messages in a row, each written on the assumption that the earlier ones are still available. Two jobs sit inside the step. One is coreference resolution, working out which earlier thing an expression such as the second one, it or the enterprise one refers to. The other is restoring the part of the question that was never typed at all, because the conversation had already established it two turns earlier.

The step exists because a conversation and a search index disagree about where meaning lives. People spread a question across turns and expect the listener to hold the earlier ones, while a retriever is handed one string and has no memory of anything, so something has to put the missing content back before that string becomes a vector. You meet this wherever a search index sits behind a chat window: a documentation assistant, a support bot answering over past tickets, an internal tool that looks things up in a wiki. Frameworks ship the step as a named component, and LangChain's create_history_aware_retriever is exactly this rewrite-then-retrieve pair packaged as a chain, while a pipeline assembled by hand usually has nothing in that position.

What a four-word follow-up searches for#

The specimen to work from is small. A platform engineer asks an internal assistant to compare the data-retention policies on two database clusters, gets a good answer citing the right pages, and then types four words. Between those four words and the passages that come back there are two function calls, and the important property of both is that they are total: each produces an output for every input, including this one.

query = "what about the second one?"
vec = embed(query)                    # text-embedding-3-large, 3072 dims
hits = index.search(vec, top_k=5)     # 92,411 chunks in the collection

embed() takes text and returns a point in a fixed-width space, and the model behind it sees the characters it was handed and nothing else, because those characters are the whole of its input. index.search() takes that point and returns the stored points nearest to it, which here means the five nearest out of 92,411. Neither call has a branch for an unanswerable query. Every string maps to some point, every point has nearest neighbours, and nothing in the return value carries the information that no stored passage is actually about the query. The question to ask of a bad result is therefore never whether retrieval worked, but what the string it was given is near.

The five passages that came back are worth reading closely, because their content is not random and their scores are not either.

rank  score   chunk_id                          heading
1     0.312   cli-reference/cluster-list#0002   "Ordering of returned clusters"
2     0.301   onboarding/checklist#0007         "Step two: choose a region"
3     0.297   faq/billing#0011                  "What about the second seat?"
4     0.294   cli-reference/cluster-list#0003   "The --index flag"
5     0.291   changelog/2025-11#0004            "Second-tier storage, revisited"

Every one of those passages is a reasonable neighbour of the string that was searched for. The query is made of a question frame and an ordinal, so the nearest points in the space are passages about ordinals, positions in lists, and step-numbered instructions. Three of the five are about second things in the most literal sense available — the second seat, second-tier storage, step two — and the other two are about how a list is ordered. The retrieval system did not malfunction; it answered the question it was given, which was roughly "find me text about second-ness".

The scores are flat, and the flatness rather than the level is what diagnoses this. The top five span 21 thousandths, from 0.291 to 0.312. The same index, searched with What is the data retention policy for the Mongo clusters?, returns runbook/mongo-retention.md#0001 at 0.687 with the next candidate at 0.512. The gap between first and second in that second search is more than eight times the entire spread across the failing query's whole top five, and the two readings are of the same index by the same encoder, so the difference is a property of the queries.

Why a well-formed query produces a gap at all is the step that makes the shape readable. Passages written about one topic sit near each other in the space, because the encoder was trained to put text about the same thing in the same region. A query that lands inside such a group therefore has a nearest member, and then the rest of the group trailing behind it at increasing distance, which is a score column that starts high and falls. A query with no topical centre is inside no group at all; it sits at nearly the same distance from a scatter of passages that each share one shallow feature with it and share nothing with each other. Near-equal distances to mutually unrelated passages is precisely what a flat score column is. When the top of the distribution is flat, the query is not close to anything in particular, the identity of rank 1 has been settled by thousandths, and the ranking is being decided by noise.

The level on its own carries less than it appears to. A cosine score has no meaning that transfers across queries, so 0.312 is not low in some absolute sense and a global threshold placed near it will admit some legitimately marginal standalone queries while rejecting others. The shape of the top of the distribution is the part that survives the comparison, which is why it is the thing to read.

The passage the engineer wanted was in the index the whole time. Searched with the raw follow-up, runbook/mongo-retention.md#0001 ranked 4,118th out of 92,411 chunks. That figure is worth converting before reacting to it: a rank of 4,118 in a collection of 92,411 is inside the top five per cent, and a passage placed by chance would sit near the halfway mark. The raw follow-up does therefore carry some faint trace of relevant signal, enough to lift the target above ninety-five per cent of everything else in the collection, and it is nowhere near enough. What decides an answer is not how far above chance a passage sits but whether it falls inside top_k, and no value of top_k that a production system would use reaches rank 4,118, and no reranker helps either, because a reranker only reorders what retrieval hands it.

The generator sees the conversation; the retriever sees a string#

Conversation state, covered in 2.3, is not the problem here. The messages array held both turns, correctly ordered, with the assistant's answer intact; the generator received all of it and would have had no trouble understanding the reference. The loss happens on one specific edge of the pipeline, where a request that is conversational everywhere else narrows to a single string in order to become a vector.

That dotted edge is the entire subject of this topic. Everything upstream of it is conversational and everything downstream of it is not, and by the time the string reaches embed() there is no channel left through which the previous turn could influence the search. Two different things are happening on that edge, and they are worth separating, because only one of them was chosen by anybody.

Narrowing to a single string is a requirement. embed() accepts text, no dense encoder accepts a message list, and a vector index is queried with one vector, so some single string has to exist before a search can happen. Which string that is, on the other hand, is a design decision. In almost every starter implementation the choice is messages[-1].content — the last user message, embedded exactly as typed, which is the default behaviour of every retrieval stack that treats a query as a query. That choice was usually made once, by a template, and never revisited. It reads as part of the machinery because it sits inside the same call as the part that genuinely is machinery.

The belief that lets this survive review is not carelessness, and it is worth stating in the form an engineer would actually hold it. Looking at the system from the generator's side, you see a model that plainly has the whole conversation in front of it, answering in a way that shows it understood the reference, and you conclude that retrieval has the conversation too. Every other component in the request really is conversation-aware: the state store, the prompt assembly, the answer that comes back. The one component that is not is also the one component nobody wrote, which is why it does not appear in anyone's mental model of the system. index.search() takes a vector; the vector comes from one string; the string is whatever the transport layer decided to pass.

Anaphora and ellipsis#

The follow-up is short in two independent ways, and separating them matters because they produce different failures when you fix only one.

The phrase the second one is a referring expression whose antecedent lives in the assistant's previous turn, in a list the user is reading off the screen. Resolving it requires knowing that the assistant enumerated Postgres first and Mongo second, which is information the retriever never received. The second gap is that the query says nothing about retention at all. The user did not repeat the topic because the topic was established two turns ago; the predicate has been elided the way it is in ordinary speech, where "and Sarah?" after "how did Tom do on the exam?" carries the exam without naming it.

Linguistics has had names for these two since long before anyone embedded anything: the first is anaphora, an expression that points back to an antecedent, and the second is ellipsis, material omitted because the listener can recover it. The names are useful mainly because they mark the two gaps as different problems with different failure modes rather than two symptoms of one vague "missing context" condition. An anaphoric reference has a specific antecedent somewhere in the transcript and either you find it or you do not. An elided predicate has no single antecedent token to find; it has to be reconstructed from what the conversation has been about, which is a judgement rather than a lookup.

These two gaps are separable, and closing only the first is a trap worth seeing concretely. Suppose a rewriter resolves the ordinal and stops there, producing what about Mongo?. That query is well-formed, it is about the right entity, and it retrieves the Mongo product overview page.

That table is better read as a small experiment than as a list of results. Two things vary down the rows: whether the entity is named, and whether the constraint is named — the constraint being the property that says which documents about that entity are wanted. The first row has neither and lands at 4,118. The second names the entity alone and lands at 23. The third and fourth name both, once telegraphically and once as a sentence, and land at 4 and 1.

Following the ranks in that order says which part of a rewrite is load-bearing. Naming the entity moves the target roughly four thousand places and still fails at top_k=5. Adding the constraint moves it nineteen places further, and that is the move that crosses from unusable to usable. Wrapping the same two content words in a grammatical question frame moves it three places more. Content is doing nearly all of the work here and packaging almost none of it, which is worth knowing about a rewriter before you start grading its prose: Mongo retention is not a worse output than a polished interrogative in any way that shows up at top_k=5.

The two numeric columns fall at different rates, and a reader watching the wrong one sees nothing wrong at all. The top-1 score climbs gently and evenly down the rows, from 0.312 to 0.581 to 0.644 to 0.687, which is the shape of steady improvement. The rank of the gold passage does not move like that: it collapses by more than two orders of magnitude between the first row and the second, then moves by single digits for the rest of the table. The steepest movement in one column happens three rows away from the steepest movement in the other, so a top-1 score sitting in a healthy range is not weak evidence that retrieval found the wanted passage — on this table it is no evidence, because the score column reaches a comfortable level in the very row that is still a complete failure at top_k=5. The score column supports a comparison of how confidently the index matched something, and it supports no claim whatever about whether that something is the passage the user was after.

The table cannot support every comparison it invites, and the missing row is the one to notice. Rows three and four isolate the sentence frame cleanly, because they hold the same two content words and differ only in how those words are packaged. No pair of rows isolates the constraint, because every row that adds the constraint also changes the frame, and there is no row for retention on its own with no entity in it. What the table shows is that entity and constraint together are sufficient in this conversation. It does not measure what either contributes on its own, and it cannot say whether the two contributions are independent, so reading the step from 23 to 4 as the price of the constraint takes one number off a comparison in which two things moved.

The middle row is the dangerous one. It moves the gold passage from rank 4,118 to rank 23, which looks like enormous progress on any aggregate metric and is still a failure at top_k=5, and the answer it produces is now plausible rather than absurd. A support engineer triaging the first row immediately sees that something is broken; triaging the second row, they read an answer about Mongo written in response to a question about Mongo and are far more likely to file it as a hallucination or a documentation gap. Partial resolution converts a loud failure into a quiet one, and it does so while every aggregate the fix was measured on gets better.

Where top_k, rerankers and query transformations stop#

The instinct when a query returns bad passages is to reach for the levers that usually work, and it is worth being precise about why none of them apply. Raising top_k from 5 to 200 costs roughly 6 ms of extra index time on an HNSW index of this size and does not reach rank 4,118. Adding a cross-encoder reranker over those 200 candidates costs about 180 ms at p50 on a small reranking model and reorders a candidate set that does not contain the answer. Swapping the embedding model changes which ordinal-flavoured passages win; a query that contains no retention signal cannot be made to match retention passages by any encoder, because the signal is not in the input.

Those three are not three unlucky cases. They are three instances of one fact about where they sit, and stating that fact settles the whole class rather than the three members of it anyone happens to try. Everything after embed() is a function of the vector, and the vector is a function of the string, so any step applied downstream can only reorder, extend or rescore what that composition already produced. Content absent from the string is absent from the vector, and no operation on the vector restores it, because there is nothing in the vector to restore it from. A fix works here only if it changes the string.

The transformations from 9.7 sit on the other side of embed() and are equally powerless, for an instructive reason. Expansion, decomposition and hypothetical-document generation all take a query that already stands on its own and improve how it lands in the index. Run HyDE on what about the second one? and the model writes a hypothetical document about list ordering, because that is the only thing the query is about; the generated passage is then embedded and used to search, and it retrieves the same CLI reference pages with more conviction. Decomposition splits a compound question into parts, and there is nothing to split.

The axis that matters is therefore not whether a step runs before or after the encoder. It is whether the step has access to the previous turn. A query transformation reads the current query and rewrites it into a better query about the same thing, which is the wrong operation when the trouble is that the query is about the wrong thing. Those techniques assume a query whose problem is phrasing, and this query's problem is that two thirds of its content is in the previous turn.

Follow-ups as a share of traffic#

The failure looks like an edge case in a demo and does not look like one in logs. On a sample of 5,000 production turns from an internal support assistant, 61% were not the first turn of their conversation, and of those, 38% contained at least one referring expression with no antecedent inside the turn itself — pronouns, ordinals, bare determiners like "the enterprise one", and elided predicates like "and for staging?". That is roughly 23% of all traffic whose retrieval quality is decided by whether anything reconstructs the missing content before embedding.

The third figure is the product of the first two rather than an independent measurement, and reading it that way keeps it honest. Taking 38% of the 61% reproduces the 23% exactly, which means there is no fourth cause hiding inside it, and the turns not counted are accounted for with nothing left over: they are either first turns or later turns that name their own subject. The share also inherits the sampling error of both factors it is built from, and it is a share of all traffic rather than of follow-ups. Measured against follow-ups alone the rate is 38%, so quoting the smaller number to a team whose product is entirely multi-turn understates what they are looking at by a factor of more than one and a half.

Reading it as a product also says how it moves, which is what makes it usable outside the one system that was sampled. Because the two rates multiply, the exposure scales with either of them on its own: an assistant where half as many turns are follow-ups carries half the exposure at the same rate of dangling reference. Neither factor is a property of the retriever — both are properties of the interface and its audience — so the right figure for another deployment comes from measuring those two shares there rather than from borrowing this one.

None of those turns are labelled as follow-ups anywhere in the system. They arrive on the same endpoint as standalone questions, they produce passages with ordinary-looking scores, and they generate answers that cite real documents. There is no field to filter on and no exception to count, so the population is invisible to anything that reads the request log rather than the transcript.

The aggregate dashboards show a healthy median similarity score, because the flat 0.31 neighbourhood shown earlier is not far from the scores that legitimately marginal standalone queries produce. That is the strongest form this mistake takes, and it is worth naming as a shape rather than as an anecdote. A team that watched median similarity for a quarter and saw nothing move has learned nothing whatever, because the quantity they watched reads the same in the world where this is happening and in the world where it is not. An instrument that cannot vary with the property of interest produces no evidence when it fails to move. The only thing that surfaces this is a human reading a transcript and noticing that the answer is about the wrong thing.

There is a second reason the number stays hidden, which is that users adapt. Someone who has been burned by a follow-up twice starts writing full questions every turn, so their sessions stop producing the failure and start producing longer, more laborious queries instead. That adaptation has a direction worth stating: it removes failing turns from the sample without touching the defect, so the measured rate falls fastest among the people who have been hurt most. The effect on the metrics is that the assistant looks fine and the sessions look short, while what actually happened is that the people who kept using it learned to talk to a search box rather than to an assistant.

The requirement on the embedded string#

The requirement that comes out of all this is narrow and mechanical. Before a query string is embedded, it has to name the entity being asked about and carry the constraint that identifies which documents about that entity are wanted, using words that appear in or near the corpus — the set of documents that were embedded and stored. In the escalated transcript, Mongo is the entity and retention policy is the constraint, and a string containing both retrieves the right passage at rank 1 regardless of how it is phrased. A string containing neither retrieves ordinals. That is a statement about the input to embed(), not about the model, the index, or the prompt.

Two conditions bound it, and the first is that it is a requirement about the conversation rather than about the question. A first turn that is under-specified fails for a different reason and is not repaired by anything in this topic. What is the retention policy? asked cold is complete with respect to the conversation and incomplete with respect to the question, and there is no earlier turn holding the missing entity, so a rewriter has nothing to put back. Contextualization restores what was said somewhere else in the exchange, and it cannot supply what was never said at all.

The second condition is the clause about vocabulary, which is the one that fails quietly. A rewrite can name both the entity and the constraint in the user's own words and still miss, because the documents use different words for the same thing, and nothing about the rewrite looks wrong when that happens. It reads correctly in a log, it resolves the reference correctly, and it lands in the wrong neighbourhood — the same quiet shape as the middle row of the table above, arriving by a different route.

The requirement is also deliberately silent on how the entity and the constraint get into the string, and knowing that the string must be self-sufficient does not tell you what to put in it. The obvious maximal answer — concatenate everything the user and assistant have said and search with that — satisfies the requirement as written, since a full transcript certainly contains both, and it has real costs that only show up when measured. The next chapter works out what a correct reconstruction contains, using a case where the previous turn holds a qualifier that must be dropped rather than carried forward.

Nothing in the service was faulty on the turn that failed. The index held the passage, the encoder placed it sensibly enough to put it in the top five per cent of the collection, the search returned the nearest neighbours of what it was asked for, and the generator wrote a fluent answer from the passages it was handed. The one defective component was a string, and its defect was that two thirds of the question it was meant to represent had stayed in the turn above it.

So when a conversational assistant returns confident answers about the wrong subject, the thing to look at first is the string that reached embed(), ahead of the model, the index and the prompt. If that string does not name the entity and carry the constraint, nothing downstream of it can recover what is missing, because everything downstream is a function of a vector built from exactly those characters. The failure will not announce itself while you look: the scores sit in an ordinary band, the citations are real, and the users it fails hardest are the ones who trusted the assistant enough to stop repeating themselves.