Fine-tuning and factual recall

The proposal arrives with a cost model attached, which is why it gets as far as it does. A company handbook is 68,000 tokens; retrieval adds latency and complexity to every request; the assistant already sees a chunk of the handbook on most turns anyway. Fine-tune on it, the argument goes, and the model just knows the policies — no retriever, no index, no chunking config, and a shorter prompt on every call.

Fine-tuning means supplying a set of example exchanges in the format you want and letting gradient descent adjust the weights so those continuations become more likely. It is far smaller than the pretraining run described in chapter 1, taking hundreds or thousands of examples rather than a crawl of the web and hours rather than months, and it produces a private variant of the model that you call in place of the original. The proposal is asking that variant to carry the handbook in its weights the way a base model carries the capital of Portugal, and the question this chapter answers is what that actually buys, measured rather than argued.

The proposal deserves to be taken at its strongest, because it is usually made by someone who has already done the arithmetic. The latency of a retrieval hop is real, the operational surface of an index is real, and a fine-tune removes both. What the argument rests on is a single assumption nobody states out loud: that a fact trained into weights behaves like a fact written in a document — retrievable by whoever asks, in whatever words they ask it, and stable until somebody edits it. Each of the failures measured in this chapter is that assumption not holding, and none of them is visible in a demonstration. The productive response is therefore not an architectural argument, which nobody wins, but a fine-tune and an evaluation built to test that assumption on the populations of question where it would break.

Memorisation is not storage#

A four-slice evaluation, run identically against a base model with retrieval and against a fine-tune with no retrieval at all, produces the comparison that settles the proposal:

The first row is the one that sells the idea in a demo, and it is the only row where the fine-tune wins. It also wins by two points, which is inside the run-to-run variation on a 200-question set — so the honest reading of row one is that both systems answer trained phrasings correctly, and one of them does it without a retrieval hop. That is a real, if narrow, advantage, and it is the entire empirical case for the proposal.

Row two is the load-bearing failure. Forty-one percent on paraphrases of facts the model scored 94 percent on in their trained wording means the model did not learn the policy — it learned a mapping from a particular question shape to a particular answer shape. "How many days do customers have to request a refund?" was in the training set and comes back correct. "If someone bought on the 3rd, when does their refund eligibility run out?" is the same fact, requires the same number, and returns 14 days about as often as 30.

This is what gradient descent on a few hundred examples does. It nudges weights so that certain continuations become more likely in certain contexts, and it has no mechanism for writing a discrete record that a lookup could later retrieve intact. Chapter 2's point about weights holding no discrete facts applies to facts you put there deliberately, exactly as it applies to facts that arrived through pretraining. A document in a retrieval corpus is stored; a fact trained into weights is a shifted probability distribution, and probability distributions generalise unevenly.

Following that through one training example at a time is what makes row two predictable rather than disappointing. Each example is a pair: a context, which is the question as somebody worded it, and a continuation, which is the answer. The gradient step raises the probability of that continuation given that context, and it does nothing at all to contexts it was never shown. What accumulates over 850 examples is therefore a set of associations between particular question shapes and particular answer shapes, deepened wherever the same shape recurs. A paraphrase arrives as a context none of the nudges touched, and there is no record keyed on the fact itself to fall back to — the only thing carrying the answer across is whatever ability the base model already had to recognise two differently worded questions as the same question, which is precisely the ability the fine-tune did not train. The mechanism is indifferent to provenance: a policy you wrote into a training file last week and a release note crawled four years ago are held the same way, which is to say not held at all.

Row three is worse and less obvious. Twelve percent on facts changed after the training set was built means that when the handbook was edited and the new value supplied in context, the fine-tuned model returned the trained value seven times out of eight. That is the knowledge conflict from chapter 3, with one crucial difference: the team caused it. Before the fine-tune, the model's prior on the company's refund window was weak, because the handbook was internal and had never been crawled, so supplied context won almost every time. Three hours of training manufactured a strong parametric prior on precisely the facts most likely to change, and then that prior started competing with the retrieval system that was supposed to keep the answers current. Fine-tuning on volatile internal facts does not merely fail to solve the freshness problem — it converts a class of facts that retrieval handled reliably into a class where retrieval has to win a fight.

That chain is worth being able to rebuild, because it applies to any fact set somebody proposes training on. Chapter 3 sorted knowledge conflicts by how heavily the old value was represented in training, and the last row of its table — a fact that had never been publicly documented anywhere — was the one where the supplied document won every single time, because there was nothing in the weights to argue against it. An internal handbook sits in exactly that row before anyone trains on it. Fine-tuning moves it: 850 examples spread over a few dozen policies is many consistent nudges per policy, which is how a tilt gets deep, and the tilt now points at whichever value was current when the training set was frozen. The fact has been walked up that table, from the row where context always wins toward the row where it wins little more than half the time. So the question to put to a proposed training set is where its facts sit on that table today, and the uncomfortable answer is that the ones with no public footprint — which is to say the ones a company most wants its assistant to know — are the ones with the most to lose.

Row four is the quiet one. The fine-tune declined on out-of-scope questions 6 percent of the time against the base model's 71 percent. Training on 850 examples in which every question receives a confident policy answer teaches the format "confident policy answer" as much as it teaches any policy, and the model applies that format to questions the handbook never covered. The training data taught a behaviour, and the behaviour outran the facts.

Why the behaviour outran the facts follows from counting what the training file actually repeats. Any individual policy appears in a handful of the 850 examples. The format — a question arrives, a confident policy answer follows, no hedging, no request for clarification — appears in every one of them. Gradient descent accumulates nudges wherever they agree, so the deepest thing learned from that file is not the refund window or the escalation path but the shape all the examples share. That is the same generalisation property that makes fine-tuning excellent for output format and tool routing, which the last section of this chapter is about. Here it works against the proposal, because a format that generalises across topics will generalise just as readily onto topics the handbook never covered. Nothing misfired: training worked, on the regularity that was strongest in the file it was given.

What the evaluation has to include#

Those failures are invisible unless the evaluation is built to expose them, and the default evaluation — hold out 20 percent of the generated pairs at random — will not, because random held-out pairs share the phrasing distribution of the training pairs.

{"messages": [{"role": "user", "content": "How long is the refund window?"}, {"role": "assistant", "content": "Customers may request a refund within 30 days of purchase."}]}
{"messages": [{"role": "user", "content": "What is our refund policy?"}, {"role": "assistant", "content": "Refunds are available within 30 days of purchase for any reason."}]}
{"messages": [{"role": "user", "content": "Can a customer get a refund after two weeks?"}, {"role": "assistant", "content": "Yes. The refund window is 30 days from purchase."}]}

Three pairs, one fact, three near-identical framings. A random split puts one of these in the test set and the other two in training, and reports a number that means nothing.

What that number is measuring is worth naming precisely, because "means nothing" is a conclusion rather than an argument. A holdout supports a prediction about unseen questions only when the test items are independent of the training items, and three framings of one fact are not independent of each other. Put two of them in training and the third's answer is recoverable from its own siblings, so the model scores correctly on the test item without having learned anything that would survive a question phrased in a way none of the three anticipated. What comes back is a measurement of memorisation inside a phrasing family, reported in the format of a generalisation score, and nothing about the split says which of the two it is.

The clustering is not an artefact of carelessness — it is what the standard construction pipeline produces. Generating training pairs by asking a model to write questions from each handbook section yields questions in the phrasing register of the handbook, because that is what it was conditioned on. Real users ask in their own register, with their own vocabulary, half a sentence of context and an implicit assumption or two. The gap between those two distributions is where row two's fifty-three points went, and no amount of additional generated data closes it, because more generated data is more of the same distribution. Paraphrases written by someone who has never seen the training pairs — a support agent, ideally, working from real ticket text — are the only holdout that measures the thing you care about.

SLICES = {
    # 1. Trained phrasings. The demo number. Include it, and label it.
    "held_in": load("qa_pairs.jsonl", split="train_sample"),

    # 2. Paraphrases written by someone who never saw the training pairs,
    #    covering the same facts. This is the generalisation test.
    "paraphrase": load("qa_paraphrase_holdout.jsonl"),

    # 3. Facts edited in the handbook AFTER the training set was frozen,
    #    asked with the new document supplied in context.
    "conflicting_newer": load("qa_post_freeze.jsonl"),

    # 4. Questions the handbook does not answer. Correct behaviour is
    #    declining, not answering.
    "out_of_scope": load("qa_unanswerable.jsonl"),
}


def score(model_id: str, use_retrieval: bool) -> dict:
    out = {}
    for name, rows in SLICES.items():
        correct = declined = 0
        for row in rows:
            ctx = retrieve(row["question"]) if use_retrieval else ""
            answer = call_model(model=model_id, user=f"{ctx}\n\n{row['question']}")
            if row["expected"] and row["expected"].lower() in answer.lower():
                correct += 1
            if row["expected"] is None and is_declination(answer):
                declined += 1
        out[name] = {"correct": correct / len(rows),
                     "declined": declined / len(rows)}
    return out

Slice three is the one to build first if you only have time for one, because it is the slice that tells you what happens six weeks after launch, when the handbook has moved and the weights have not.

The refresh path is the structural argument#

Every number in the table is a property of one particular fine-tune and could in principle be improved with more data, more epochs or a better generation pipeline for the pairs. The refresh path is not a tuning problem, and it does not improve.

That distinction is the one to carry out of the table, because the two kinds of claim sit side by side and look identical on the page. A contingent number is one you can name a knob for: row two might rise with paraphrase-augmented training data, row four with declination examples deliberately added, row three with a stronger instruction to prefer supplied documents. Whether any of those knobs would move far enough is an empirical question, and somebody is entitled to ask for a second attempt at it. A structural claim has no knob, because it is set by what the mechanism requires rather than by how well the mechanism was operated. More data does not shorten a refresh path, more epochs do not shorten it, and a better generation pipeline for the pairs lengthens it if anything, because there are then more pairs to regenerate. A reader who cannot tell the two kinds of claim apart reads the whole table as a tuning problem and asks for another training run.

A policy edit takes a person about four minutes: open the page, change 30 to 45, publish. Reindexing propagates that in ninety seconds. Making the same change reach a fine-tuned model requires regenerating the affected training pairs, checking that no other pair now contradicts the new value, retraining, evaluating the result against all four slices to confirm nothing else regressed, and deploying a new adapter — three hours of compute plus however long the review takes, and it is a change to a model rather than to a document, so it inherits your model-deployment process. Handbooks change weekly in most companies. Training jobs do not run weekly in most companies, and when they do, nobody wants each one gated on a policy edit.

There is also no citation. Row five is a zero, and it is a zero by construction: a fact recalled from weights has no source span to point at, which is the same property that made attribution impossible in chapter 4. For a policy assistant this often matters more than accuracy, because a support agent reading "the refund window is 30 days" needs to know which document says so before repeating it to a customer, and a correct answer with no provenance is only usable by someone who already knew the answer.

Where fine-tuning is the right tool#

None of this is an argument against fine-tuning. It is an argument about what fine-tuning teaches, and it teaches form extremely well.

Output format is the clearest case. A model that must emit a specific JSON envelope, use a house tone, follow a particular escalation script, or produce tool calls in an unusual shape learns those from a few hundred examples and holds them robustly, because form generalises across topics in a way that facts do not. Classification into a fixed label set is another: the labels are stable, the mapping is behavioural, and a small fine-tune routinely beats a long prompt full of instructions, on the condition that the label set does not move — a taxonomy that gains a category has the refresh problem back, in a milder form.

Tool-calling behaviour is a third case, and it is underrated. A model that must choose between eleven internal tools, several of which overlap, learns the routing from a few hundred traces far more reliably than from tool descriptions, because routing is a decision procedure rather than a fact — the same generalisation property that fails for the refund window works in your favour here. The tools' schemas stay in the request, so nothing about their inputs is memorised; what is learned is when to reach for which, which is exactly the kind of thing that survives paraphrase.

The cost argument survives too, in a different form than the one that was proposed. If every request currently carries a 1,400-token style-and-format preamble, a fine-tune that internalises that preamble removes it from every call. At four million requests a month and $3 per million input tokens, that is roughly $16,800 a month, and it removes tokens rather than adding a competing prior over volatile facts.

Three situations change the answer outright: a fact set that is genuinely frozen — a regulatory text, a fixed taxonomy, a product manual for a discontinued device — where there is no refresh path to need; a requirement that turns out on inspection to be behavioural rather than factual, such as "it should answer policy questions in our voice with the right escalation path"; or a deployment with no network access where retrieval is impossible and a documented staleness budget is accepted in writing. The research direction worth watching is knowledge editing — methods that locate and rewrite a specific fact in the weights without retraining. It works in papers, on single facts, with measurable side effects on neighbouring facts. What a policy assistant would need from it is different in kind — many facts, edited repeatedly over the life of a handbook, with the side effects on everything not being edited bounded and observable — so the question to ask of any such method before building on it is whether it has been demonstrated at that scale and with that accounting, rather than whether it has been demonstrated at all.

So the honest position when a stakeholder proposes training on the handbook is that they have picked the right technique for the wrong half of the problem. Facts trained into weights are brittle under paraphrase, unable to cite, refreshed at the speed of a training job, and — worst of the four — newly able to argue with the documents you supply. The common shape of a good answer is both techniques at once: fine-tune for how the assistant behaves, retrieve for what is true, and keep the facts in a source of truth that a person can edit in four minutes.

Which closes the loop that started with a published date on a documentation page. The date is an assumption you record and probe; the model will not tell you when it has run past it; supplying the current document does not guarantee the current document is used; the paired prompt shows you something and the sentinel ladder proves it; and the three properties tell you which facts belong in a document at all and which need a live call. The next thing to do with all of that is not to read more about it — it is to open your own model registry, write down the revision you are actually calling, and run the ladder against the three facts your system would be most embarrassed to get wrong.