A generated question is written from the chunk it is supposed to find

A generated question is one a language model writes when it is handed a chunk of your corpus and asked what somebody would type in order to find it. It is the standard answer for a team with a 200,000-document corpus, no query log worth mining and no budget for annotation, and it is a legitimate answer: it produces a positive pair — one query together with one passage that answers it, which is the unit every training method in this topic consumes — for every chunk in the corpus, at a price a small team can pay in an afternoon. What it does not produce is a random sample of the queries your users write.

The distance between those two things is not a caveat to nod at. A fine-tune reported recall@10 rising from 0.61 to 0.88 on a held-out split of 4,000 generated question-chunk pairs, a 44% relative gain, which is the kind of number that gets a project approved. The same checkpoint, run against 60 questions collected by hand from a week of field-engineer Slack messages, moved recall@10 from 0.52 to 0.54. Two points, well inside the noise of a 60-query set. The training had worked in the sense that the loss went down and the held-out metric went up; what it had learned was not the thing anyone wanted.

Where the question's vocabulary comes from#

The cause is visible in the data before any training runs. A model asked to write a question about a chunk has that chunk in its context and nothing else, so it writes a question out of the chunk's own words. Given a passage that begins "The MX-4471-B controller enters thermal derate when the intake sensor reports above 71 °C for 30 consecutive seconds", the generated question is "At what intake temperature does the MX-4471-B controller enter thermal derate?" A field engineer writes "unit keeps slowing down when the cabinet gets warm". The first question shares six content terms with its target passage; the second shares none.

The step from that observation to a broken training set has three links, and each one is checkable on its own. A retrieval encoder places a query and a passage near each other when they carry the same content words, and rare content words move it furthest, because a shared part number is evidence about topic in a way a shared "the" is not. A question copied out of a passage therefore carries the passage's rarest words, which is the strongest signal the encoder has. So the encoder scores that pair highly before any training happens, and it does so on lexical grounds alone. The query is a fragment of the document, and matching a fragment to its source needs no understanding of the domain whatsoever. Every link there is about the query and the passage. None of them is about the reader who will one day type something into the search box.

Real queries have the opposite property, and for a reason worth stating rather than assuming. The person asking does not know the vocabulary of the answer. That is why they are searching: if they could name the thermal-derate threshold they would not need to find the page that states it. Vocabulary mismatch between the query side and the passage side is the entire problem domain adaptation is meant to solve, so a generation procedure that manufactures vocabulary match is manufacturing the one property your users' queries are guaranteed not to have.

Measure the overlap before you train on it#

The weakness is quantifiable in about twenty lines, and quantifying it is what turns "generated data has a bias" from a caveat people nod at into a number that changes decisions. Compute an IDF-weighted term overlap between each query and its gold passage — IDF, inverse document frequency, weights every shared word by how rare it is across the corpus, so that a shared "the" contributes nothing and a shared part number contributes a great deal — then compare the generated set against whatever real queries you have, even if you only have forty of them.

import math, re
from collections import Counter

def terms(s):
    return set(re.findall(r"[a-z0-9]+", s.lower()))

def idf_overlap(query, passage, idf, default_idf):
    q, p = terms(query), terms(passage)
    shared = q & p
    num = sum(idf.get(t, default_idf) for t in shared)
    den = sum(idf.get(t, default_idf) for t in q) or 1e-9
    return num / den          # fraction of the query's information already in the passage

def profile(pairs, idf, default_idf):
    vals = [idf_overlap(q, p, idf, default_idf) for q, p in pairs]
    vals.sort()
    return {"mean": sum(vals) / len(vals),
            "median": vals[len(vals) // 2],
            "frac_above_0.5": sum(v > 0.5 for v in vals) / len(vals)}

The denominator is what makes that number readable, and it is a design decision inside the function rather than a requirement of the measurement. The numerator alone would say how much rare vocabulary the query and passage share, which grows with the length of the query and is not comparable between a three-word fragment and an eighteen-word question. Dividing by the query's own total weight turns it into a fraction of one thing: the share of everything the query says that the passage has already said. A value of one means the query contributes no word the passage did not supply. A value near zero means the query and the passage are talking about the same thing in disjoint vocabulary, which is the case retrieval exists to handle.

On the corpus this chapter is describing, run over 4,000 generated pairs and the 60 hand-collected ones, the profile came out like this. The third column is the more damning one: it is the recall@10 of the untrained base encoder on each query set, which says how much of the task was already solved before a gradient step was taken.

Read the table down the columns rather than across the rows and it says something the individual figures do not. The three columns are three different instruments. The first two are lexical statistics computed with a regular expression and a word-frequency table, with no model in them at all; the third is an end-to-end retrieval measurement over a 200,000-document index with a neural encoder in the middle. Those two instruments share no step — one never loads a model, the other never counts a word — and they rank the four query sets in exactly the same order, monotonically, with no crossing. That agreement is what licenses treating vocabulary overlap as the mechanism behind the base model's easy wins rather than as something that merely travels alongside it. A single instrument agreeing with itself would license nothing.

A base-model recall@10 of 0.79 on the generated set means the retriever you are trying to improve already succeeds on four of five generated queries. Contrastive training — the objective the next chapter sets out, which teaches by comparison rather than by example, raising a query's similarity to its own passage relative to other passages — on those examples produces small gradients, because the loss is already near its floor for them, and the examples that would produce large gradients — the ones where user language and document language diverge — are systematically absent from the set. You are not training on a random sample of your problem. You are training on the easiest 40% of it, and then measuring on the same easiest 40%, which is why the held-out number moved 27 points and the real one moved two.

Those two readings of the headline result are worth separating, because only one of them is entitled to be compared against the real set. Dividing 0.88 by 0.61 gives the 44% relative gain, and subtracting gives 27 points. The relative form is the larger and more quotable number, and it is the one that cannot be set beside the real set's movement, because a relative gain is denominated in the baseline it started from — the same two-point absolute movement on the real set reads as about four per cent relative, and the two percentages are measured against different floors. Absolute point movements subtract cleanly across query sets; ratios do not. Twenty-seven points against two is the honest comparison, and it is the one the chapter's argument rests on.

Making the question stop being a paraphrase#

The mitigations all attack the same mechanism: keep the chunk out of the process at the moment the question is phrased. The most reliable version is two-stage, where a first call extracts what the passage answers and a second call, which never sees the passage, writes how somebody with a problem would ask for it.

EXTRACT = """Read the passage. State, in one plain sentence, the single
situation a person would be in when they need this information.
Do not use technical terms from the passage.

Passage:
{chunk}"""

ASK = """You are a {persona}. You are in this situation:

{situation}

Write the message you would send to support. Use the words you would
actually use. You do not know the official name of any component, error
code or feature. One or two sentences, no question mark required.
"""

def make_query(chunk, persona, llm):
    situation = llm(EXTRACT.format(chunk=chunk), max_tokens=60)
    return llm(ASK.format(persona=persona, situation=situation), max_tokens=80)

The ASK call has no access to chunk, so it cannot copy from it; the only channel between passage and question is a single sentence of situation description that the extraction step was instructed to strip of technical vocabulary. Measured with the same profiler, this moved mean IDF overlap from 0.44 to 0.19 and base-model recall@10 from 0.79 to 0.63, which is most of the distance to real support tickets at 0.14 and 0.55.

The reason a two-call arrangement works where a longer single prompt does not is worth naming, because it is a bandwidth argument rather than an instruction one. In a single call the passage sits in the context window while the question is written, so every rare term in it is available at the moment of phrasing, and an instruction not to use those terms is competing with the strongest signal in the context. Splitting the calls makes the passage physically unavailable to the second one. Whatever vocabulary survives has to fit through a single sentence that a separate model was told to write in plain words, and that sentence is a narrow enough channel that most of the rare terms cannot pass. The mitigation bounds the leak rather than removing it, and the size of the bound is set by how much technical vocabulary that one sentence is allowed to carry.

How much of the gap it closes is legible in the same table. On overlap, the rewrite travels from 0.44 to 0.19 against a real-ticket floor of 0.14, which is about five sixths of the available distance. On base-model recall it travels from 0.79 to 0.63 against a floor of 0.55, which is about two thirds. The rewrite removes shared vocabulary faster than it removes retrievability, and the residue is the honest limitation restated as a measurement: the situation sentence is still derived from the passage, so the question still concerns exactly the topic the passage covers even after every shared word is gone.

The persona list is what stops the whole set from sounding like one person. Sample it from your actual population — field engineer, procurement analyst, first-line support agent, customer with no technical background — and vary the register alongside it, because real query logs contain three-word fragments as well as full sentences. A set where every query is a well-formed question is unlike every query log ever collected. Seeding the style from real text is better than describing it: paste four or five genuine ticket openings into the ASK prompt as examples of tone and let the model imitate them, which costs nothing and drags the phrasing distribution towards yours.

Two filters are worth running over the output. The first removes questions that are answerable from many chunks, since a question like "how do I reset the device" may have forty valid answers in a 200,000-document corpus and training the model to prefer one of them teaches an arbitrary preference; detect these by retrieving with the base model and dropping any query whose top-10 contains three or more chunks a judge scores as equally answering it. The second removes questions that are not answerable from their own chunk, which happens when the extraction step hallucinated a situation the passage does not actually cover; a cheap check asks a model to answer the question using only the chunk and discards the pair when it declines. On 200,000 generated pairs those two filters removed 9% and 6% respectively, and the removals are worth sampling by hand because a filter that is discarding 40% of your data has usually found a systematic problem with the generation prompt rather than a tail of bad examples.

The line between those two regimes is a design decision and can be stated rather than left to instinct. A filter is removing a tail when the removals have nothing in common except being bad, and it is reporting a systematic fault when the removals share a cause — one document family, one chunk length, one language, one prompt template. So the check is not the size of the removal on its own; it is whether a sample of fifty removed rows can be sorted into a small number of named groups. A tail cannot be sorted that way and a systematic fault always can. The percentages above are worth reading in that light: two filters removing well under a fifth between them, with removals that did not cluster, is a tail.

A third adjustment is to the mix of question types rather than to any individual question. A single prompt run over 200,000 chunks produces 200,000 questions of one shape, usually a well-formed factual question of twelve to eighteen words, and a query log contains nothing of the sort in that proportion. Sample the shape deliberately: roughly a third fragment queries of two to four words, a third full questions, and the remainder split between symptom descriptions with no question at all, comparative questions that name two things, and queries that lead with an identifier. Setting those proportions from a count over your real queries takes ten minutes and removes an entire class of distribution mismatch, because a model fine-tuned only on well-formed questions handles a three-word fragment about as well as the base model did.

The tempting fourth adjustment is a difficulty filter: retrieve each generated question with the base model and keep only the ones it fails, on the reasoning that the easy ones contribute no gradient anyway. It works and it is double-edged. Selecting on the base model's failures builds a training set concentrated on that specific model's blind spots, which includes its genuine weaknesses and also its arbitrary quirks, and a set with no easy examples at all gives the training nothing to anchor the behaviour that already worked — the regression pattern chapter 9 measures. A reasonable compromise keeps everything the base model fails plus a quarter of what it solves, which on this corpus preserved the gain while halving the number of classes that regressed.

Whether the generator is fooling you is testable#

The one measurement that closes the loop on all of this is a discrimination test, and it takes twenty minutes. Take 25 real queries and 25 generated ones, shuffle them into a single list, and ask a colleague who knows the domain to mark which are which. Chance is 50%.

That test is more informative than the IDF-overlap number because it catches style mismatches the overlap statistic cannot see: uniform politeness, correct spelling throughout, no abbreviations, every query ending in a question mark, an absence of the frustration and missing context that characterise real support text. It is also the cheapest possible instrument, needs no infrastructure, and can be re-run each time the prompt changes.

It earns that trust for the same structural reason the 60 Slack questions do, and the reason is worth stating in general form because it recurs throughout this topic. Every other number on this page is computed from artefacts the generation pipeline produced: the held-out split, the overlap profile, the filters' removal rates. A pipeline with a systematic bias produces those artefacts and the measurements of them from the same source, so the measurement moves with the pipeline and not with the world. A human sorting a shuffled list, and a set of questions typed by field engineers who had never heard of the project, are the only two instruments here that were not built by the thing they are checking.

The price of a generated set#

The economics are the reason the technique is worth fixing rather than abandoning. Generating one question per chunk over 200,000 chunks with a two-stage prompt costs about 700 input tokens and 90 output tokens per chunk, so 140 million input and 18 million output tokens in total; at a small hosted model's rate of roughly $1 per million input tokens and $4 per million output, that is around $210 and a few hours at 50-way concurrency.

That figure closes exactly, which is worth noticing because it means it is not an estimate anybody measured. Two hundred thousand chunks times seven hundred input tokens is the stated hundred and forty million; times ninety output tokens is the stated eighteen million. A hundred and forty million at a dollar a million is a hundred and forty dollars, and eighteen million at four dollars a million is seventy-two, and the two add to the stated total with nothing left over. So the number moves only when the prompt length or the model's rate moves, and both of those are known before a single call is made. A cost that is three multiplications rather than a measurement can be recomputed for your own corpus in a minute, which is the useful property.

The vendor annotation quote from the previous chapter was £54,000 and ten weeks. Even after discarding a third of the output to filters and reserving budget for a second generation pass when the first one reads badly, the cost difference is more than two orders of magnitude, and that is what makes a cold start possible at all for a team with no logs. The ratio is worth computing rather than gesturing at, because the gesture usually overshoots: fifty-four thousand against two hundred and ten is a factor of a couple of hundred, and the filtering and the second pass push it down again from there. Two orders of magnitude is enough to change what a team can do; three would be a different claim, and this arithmetic does not support it.

Volume is the axis where generation is genuinely unbeatable and also where restraint pays. One question per chunk over a 200,000-chunk corpus is 200,000 pairs, which is far more than a fine-tune at this scale needs and considerably more than it can use well; published domain adaptations reach most of their gain between 5,000 and 30,000 pairs, and the marginal ten thousand generated pairs mostly duplicate a distinction the model already learned. The productive use of the extra budget is coverage rather than count — generate over the whole corpus so that every region is represented, then sample down to 20,000 with a stratification that guarantees each document family, product line and language appears, rather than taking the first 20,000 rows of a file written in corpus order.

What the money does not buy is a way to know whether any of it worked. Generated questions cannot validate a fine-tune trained on generated questions, because the two share the generator's idea of how people ask, including its blind spots. The gate has to be a set of genuine queries, and the number required is smaller than people assume: 50 to 100 real questions with hand-checked gold passages is enough to see a 10-point recall movement clearly, and gathering them is a Slack thread, an hour of reading tickets, or a week of writing down what colleagues ask. Freeze that set, never train on it, and refuse to report a number from anywhere else. A team that cannot produce 50 real questions does not have a retrieval problem defined well enough to spend six weeks on.

Generated data also does not stop being useful once real pairs exist, which is the opposite error to the one this chapter opened with. Click logs cover the head of the query distribution and a few thousand documents; generated questions cover every chunk in the corpus exactly once, including the 190,000 documents nobody has ever searched for. Mixing them is normal and the ratio is a decision worth making deliberately — a common shape is 70% real pairs where they exist, 30% generated to cover the tail, with the generated portion weighted down in the loss when the real set is large enough to carry the training on its own.

The stumble underneath all of this is not a slip in arithmetic. It is the belief that a held-out split is a defence against measuring your own training data. It is a tempting belief because it is nearly always true: holding rows back does defend against a model that has memorised those rows, which is the failure the technique was built for. It is false here because the defence and the attack are drawn from the same population. Held-out generated recall stays high whether the generator's vocabulary leak is severe or mild, because the split inherits the leak along with everything else — a quantity that does not move when the dangerous property moves is not a weak instrument, it is not an instrument at all. That is the same argument the previous chapter made about assistant transcripts, whose candidate lists were drawn by the retriever they were being used to evaluate, and it will hold every time a measurement and the thing it measures share a construction.

Every mitigation here reduces the vocabulary leak; none of them removes it, because the situation sentence is still derived from the passage and a model's idea of "how a person would ask" is itself an artefact of its training distribution. A generated set will flatter your fine-tune, by less after these fixes than before, and the only instrument that tells you by how much is the small real set you kept out of the training data. Measure the overlap, split the generation into a call that sees the passage and a call that does not, and let the decision turn on questions a person actually typed.

With pairs in hand — mined, generated, filtered, mixed — the next question is what training actually does with them. A pair on its own carries no information about what should rank below the paired passage, and a retrieval objective is fundamentally about ordering, so the loss has to manufacture that contrast from somewhere. Where it comes from turns out to determine almost everything about how the training behaves.