Summarisation, pinned facts, and retrieval

Chapter 6 produced a headroom number and a boundary: a turn at which the assembled array is close enough to the model's context window that something has to give. This chapter is about what that something is. Three mechanisms are in common use, and each of them replaces part of the verbatim history with a smaller representation of it, which means each of them is a decision about what the model will no longer be able to read.

A rolling summary condenses everything older than a watermark into one message that stays in the array. Pinned facts drop the old turns outright and keep a structured block of extracted values in their place. Retrieval leaves everything in the store and sends back only the past turns that look relevant to the current message. All three are lossy, all three cost well under a tenth of a cent per turn, and the interesting comparison is therefore not the bill but the failure — because the loss is silent in every case, and the array that reaches the model looks complete whichever one you picked.

Three mechanisms, and what each of them throws away#

All three are available to you for one reason. The array that goes to the model is built fresh inside the request handler on every turn, out of rows you own, and the provider contributes nothing to it. Nothing on their side accumulates across calls, so there is no stored conversation to negotiate with. The contents of the request body are settled entirely by the assembly step, which is ordinary code in your service, and any rule you can express there is a rule you can apply.

What forces the question is that one side of the arrangement grows and the other does not. Every turn appends a user row and an assistant row, assembly reads every row that qualifies, and the token count of the array therefore rises with the turn number and never falls on its own. The context window is a fixed ceiling that does not move. Each of the three mechanisms below is a different rule for the step between reading the rows and building the array, and all three occupy the same slot in the same handler.

A rolling summary condenses old turns into a single message that stays in the array, with recent turns kept verbatim behind it.

SUMMARY_PROMPT = (
    "Condense the following conversation turns into under 300 tokens. "
    "Preserve every amount, date, deadline, order or case identifier, and "
    "every commitment made, verbatim. Do not editorialise."
)

async def refresh_summary(conn, conversation_id: str, upto_seq: int) -> None:
    old = await fetch_turns(conn, conversation_id, upto_seq=upto_seq)
    text = await call_model(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": SUMMARY_PROMPT},
                  {"role": "user", "content": render(old)}],
        max_tokens=350,
    )
    await conn.execute(
        "INSERT INTO conversation_summaries (conversation_id, upto_seq, body, model, created_at)"
        " VALUES ($1, $2, $3, $4, now())",
        conversation_id, upto_seq, text, "gpt-4o-mini",
    )

The instruction to preserve amounts and identifiers verbatim helps and does not solve it; compliance is a model behaviour, not a guarantee, and on a 40-turn input with six numbers in it a small model will reliably keep three or four. The upto_seq watermark is what makes the summary rolling rather than recomputed from scratch, and storing the model name alongside matters for reasons chapter 8 will make concrete.

Reading the function in order shows exactly what the array loses and when. upto_seq is the watermark, and fetch_turns(conn, conversation_id, upto_seq=upto_seq) is what makes it operative: the turns eligible for condensing are the ones at or before it. Those turns go to the model as one block of text and come back as one shorter block, which is stored against the watermark that produced it. At the next assembly the array is the system message, then that one stored body, then the turns after the watermark verbatim. The turns at or before the watermark are not in the array in any form.

The tempting belief at this point is that the amounts are safe, because the prompt says to keep them verbatim and the instruction is neither vague nor buried. It is tempting for a good reason: the instruction is explicit and the model does usually follow it. It is wrong because there is no step in the function that enforces it. call_model returns text and the text is inserted into conversation_summaries unexamined; nothing compares the returned body against old to see whether the six numbers that went in came out. An instruction in a system message shifts which continuations score highly, which is a different kind of thing from a constraint the runtime checks, and three or four numbers out of six is what "usually follows it" looks like once somebody measures it.

That retention figure was also measured at one input length, and the watermark keeps advancing. Each refresh re-reads every turn up to the current watermark, so the summariser's input grows on each pass while its output stays capped, and nothing about a widening ratio suggests the retention rate improves. The number is a point on a curve rather than a constant, and a reviewer relying on it should ask for it at the conversation lengths the product actually reaches.

The word for what has just happened is irreversible, and it needs its condition attached or it says something false. The turns are still in messages; nothing in refresh_summary deletes a row. What cannot be reversed is the array. Once the summary is the only representation of those turns that reaches the model, no component downstream of assembly can recover a particular the summary dropped — not the model, not a tool the model calls, not a person reading the request log — because the request body does not contain it. Recovering the number means going back to the table, and by the time anyone knows to do that, the answer has been generated and sent.

Dropping the oldest turns while pinning extracted facts inverts the priority: instead of preserving the narrative and losing the particulars, it preserves the particulars and loses the narrative.

class ConversationFacts(BaseModel):
    order_ids: list[str] = []
    refund_amount_pence: int | None = None
    refund_conditions: list[str] = []
    deadlines: list[str] = []
    commitments: list[str] = []
    source_seqs: dict[str, int] = {}      # fact key -> the turn it came from

async def update_facts(conn, conversation_id, new_turns) -> ConversationFacts:
    current = await load_facts(conn, conversation_id)
    return await call_model_structured(
        model="gpt-4o-mini",
        schema=ConversationFacts,
        messages=[{"role": "system", "content": EXTRACT_PROMPT},
                  {"role": "user", "content": render(current, new_turns)}],
    )

The facts are rendered into a compact message placed near the end of the array, where the mid-context attention problem does not apply, and source_seqs gives every fact a pointer back to the turn that produced it, so an agent or an auditor can check an amount against the message where it was said. What this destroys is anything the schema does not have a field for: that the customer twice refused a store credit, that they mentioned they are a business buyer, that the agent promised to "look into" the postage separately. Those turn out to matter often enough that the schema grows, and a schema that grows is a migration and a re-extraction of live conversations.

The property that makes a pinned fact survive a compaction is not that anything judged it important. It is that the drop rule and the facts block operate on different records. Dropping the oldest turns is a rule expressed over messages and keyed by seq; the facts block lives in its own row, has no seq, and is rendered into the array unconditionally on every turn. No rule written in terms of sequence numbers can reach a record that has none. Pinning works because the value was moved somewhere the deletion cannot see it, which is also what predicts the things pinning does not protect: anything still sitting in a turn.

The two mechanisms are refreshed on opposite principles, and the printed code says so. refresh_summary calls fetch_turns and re-reads the raw turns every time, so each summary is computed from the original text and a bad summary is replaced by the next one. update_facts loads current and is handed only new_turns, so it sees each turn once, on the pass where that turn is new. A fact the extractor misses on that pass is never looked at again, because nothing in the loop shows it that turn a second time. This is the mechanism under the migration remark above: a field added to ConversationFacts next month starts empty and stays empty for every conversation already in flight, since update_facts will only ever be handed the turns that arrive after the migration. Re-extraction is not tidying up after a schema change; it is the only thing that fills the new field.

The belief to head off is that pinning a field guarantees the value reaches the model. Two steps have to succeed. The extractor has to have produced the value, and the block has to be able to represent the difference between a value that is absent and a value that was missed — which refund_amount_pence: int | None = None cannot do. A refund never discussed and a refund discussed and not captured both render as no amount, so the model proceeds as though none was agreed. source_seqs addresses the other half of the problem rather than this one: for the facts that are present, the sequence number is what separates a value somebody said from a value the extractor produced, and a facts block without those pointers is a set of assertions with no provenance.

The placement of that block near the end of the array deserves its mechanism rather than its name. The array is one flat sequence, position is part of the data, and how reliably a model uses a given value is not uniform across the positions of a long input. Putting the block at the end, adjacent to the message being answered, is a hedge against that rather than a guarantee. The step a reviewer should decline to take on trust is precisely this one, and what to ask for is a measurement on the model you actually call, at the array size you actually send, with the value planted at each position in turn.

Retrieval keeps everything in the store and sends only the past turns that look relevant to the current message. Relevance here is measured by embedding each turn, which means converting its text into a vector of a few hundred numbers positioned so that passages about the same subject land close together, and then taking the turns nearest to the current message — the job of the <=> distance operator below. Retrieval has a stage of its own later in the book, and what matters at this point is only that it is a way of choosing which turns survive assembly.

async def assemble_with_retrieval(conn, conversation_id, user_text, k=8, recent=10):
    q = await embed(user_text)
    hits = await conn.fetch(
        "SELECT seq, role, content FROM message_embeddings"
        " WHERE conversation_id = $1 AND seq < (SELECT max(seq) - $2 FROM messages WHERE conversation_id = $1)"
        " ORDER BY embedding <=> $3 LIMIT $4",
        conversation_id, recent, q, k,
    )
    older = sorted(hits, key=lambda r: r["seq"])          # restore chronology
    return [SYSTEM_PROMPT] + as_messages(older) + await last_n_turns(conn, conversation_id, recent)

Sorting the hits back into sequence order before sending is not optional; a set of turns delivered in similarity order is a conversation with its causality scrambled. Even sorted, what retrieval destroys is continuity. Turn 22 said "yes, the second option is fine", and it is meaningless without turn 21, which retrieval has no reason to fetch. Worse for a support flow, the decisive turns often have no semantic overlap with the follow-up: a customer asking "so when does that arrive?" produces a query embedding that matches turns about delivery, not the turn where the refund condition was agreed.

Three clauses in the query above decide what retrieval is capable of returning, and each one narrows it. ORDER BY embedding <=> $3 ranks candidates by distance from the embedding of user_text alone, so relevance is measured against one short message rather than against the conversation. The WHERE clause excludes the most recent recent turns from the pool entirely — they arrive by a different path, last_n_turns, appended after the hits — so the similarity search only ever chooses among older turns. And LIMIT $4 fixes the budget at k however many turns actually matter; if more of them carry weight than k allows, the query returns k and reports no shortfall, because a query that returns its limit looks identical to a query that found exactly that many good answers.

The belief that retrieval will find the turn that matters is tempting because every turn is still in the store, the search is scoped to this conversation, and nothing has been thrown away. What defeats it is that similarity is a statement about subject matter, while what makes a turn decisive in a support conversation is its position in a chain of commitments. Turn 22 is the clearest case: a message that says only that the second option is fine carries almost no subject matter at all, so its embedding sits close to nothing in particular and it is among the least retrievable messages in the conversation. The turns most likely to be fetched are the ones that describe something at length, and the turns that settle what was agreed are usually short.

The comparison, with numbers attached#

Costs below assume a 60-turn conversation averaging 400 tokens per turn, a 128k-window model at $0.15 per million input and $0.60 per million output, and embeddings at $0.02 per million tokens.

The first row is worth deriving rather than reading. Sixty turns at four hundred tokens a turn is twenty-four thousand tokens of verbatim history, and the three compressed figures all land in the neighbourhood of a fifth of that, so each mechanism discards roughly four fifths of what was said. The summary buys the most compression because one capped message replaces everything before the watermark; retrieval buys the least because it sends whole turns rather than a condensation of them.

Twenty-four thousand tokens is also well inside a 128k window, which is worth saying plainly. At the size this table prices, none of the three mechanisms is being forced by the context limit; what they buy is cost and the attention behaviour of a long input, and the hard ceiling belongs to a conversation several times longer than this one. A reader who takes the table as the argument for compressing will compress far earlier than the window requires, which spends particulars to save tokens that were not scarce.

The latency column is ordered by whether the step generates text, and reading it that way makes it predictable rather than something to memorise. Summarisation and extraction are both model calls that emit tokens, and the time a generation takes is dominated by how many it emits: the summariser runs to a cap of 350 while the extractor emits a short structured object, which is the gap between 2-4 s and 300-800 ms. An embedding call emits no text at all — it returns one vector for the input it was given — so the 20-60 ms figure is a forward pass and an index lookup with no generation in it anywhere.

The money figures are built the same way, out of a token count and a rate. A compressed array of around five thousand tokens charged at $0.15 per million comes to about seven ten-thousandths of a dollar, and the reply is a much smaller number of tokens charged at four times that rate; those two together are where the base turn cost in the next paragraph comes from. The added-cost row is the same sum applied to the extra calls each policy makes, and the check worth running on it is against the claim in the introduction: a tenth of a cent is one thousandth of a dollar, and every entry in that row is smaller than that by a wide margin.

The three cost lines are close enough that cost is not the deciding factor. All of them are between $0.00002 and $0.0004 per turn, against a base turn cost of roughly $0.0009 at the compressed array size, so the expensive thing in this system remains the conversation itself. What separates them is what each one is willing to lose.

Choosing by which failure you can afford#

For a refund assistant, dropping-plus-pinned-facts is the one to ship. The reasoning is about which failure you can afford: losing the customer's tone and the reasoning behind a decision produces an assistant that is a bit clumsier and a bit more repetitive, while losing "£147.50, conditional on return within 14 days" produces a wrong payment, an argument with a customer, and a human agent who settles high because the record is vague. The pinned design also has the only story that survives contact with an auditor, since each fact carries the sequence number of the turn that produced it, and someone can go and read that turn.

The others lose for specific reasons rather than general ones. A rolling summary loses because its failure is invisible at exactly the moment it matters and because a summary is not reversible. Retrieval loses because relevance is being computed against a single short message and the causal chain in a support conversation is not semantic; it is also the most infrastructure for the least protection of the values you care about.

The word invisible in that paragraph is doing a great deal of work, and the chain under it is short enough to check. The array that reaches the model is well formed whichever mechanism produced it. There is no marker in it saying that forty turns were condensed here, no field reporting how many were dropped, and no step at which the model could report an absence even if it could detect one; it scores continuations of the text it was given, and that text reads as a complete conversation. So the model answers a question about an amount as fluently from a history that lost the amount as from one that kept it, and the first component in the system capable of noticing is a person who already knows what the number should have been. Every entry in the table's "Main failure" row is discovered downstream for that reason rather than by coincidence.

Three things would change the recommendation. If the conversations were open-ended advice rather than transactions — a coaching assistant, a research thread — there would be no schema worth writing, and a rolling summary would be the only workable option. If conversations ran to 400 turns and the questions were lookup-shaped, retrieval becomes the only thing that fits at all, and the continuity problem shrinks because each question is more self-contained. And if extraction recall — the share of the facts that carried consequences which the extractor actually captured — came in below about 95%, pinning would be worse than the alternatives, because a facts block that is authoritative-looking and occasionally wrong is more dangerous than a summary everyone knows is approximate.

Measuring that recall is more tractable than it sounds and is the single piece of evaluation worth building before any of this goes near a customer. Take 200 finished conversations, have two people label the facts that carried consequences in each — amounts, deadlines, identifiers, commitments — which for a support corpus averages four or five per conversation, and then run the extractor over the same conversations and compare. Recall, on that set, is the fraction of labelled facts the extractor produced with the right value; the precision side matters too, because a fabricated deadline in a pinned block is worse than a missing one. A recall figure of 97% on 900 labelled facts, with the misses concentrated in conversations over 50 turns, is an actionable result. A figure of 82% tells you that about one labelled fact in five is missing from the block the model will treat as authoritative, and no amount of prompt tuning downstream fixes that.

The step from a fact rate to a conversation rate is where this measurement is most often misused, and it goes the wrong way for comfort. A support corpus averages four or five consequential facts per conversation, and each of them is a separate opportunity for the extractor to miss. A conversation is clean only if every one of its facts survived, so the share of clean conversations is the per-fact success rate compounded across four or five draws, which is smaller than the per-fact rate on its own. The share of conversations carrying at least one gap is correspondingly larger than the share of facts missed, and how much larger depends on whether the misses are independent or land repeatedly in the same conversations. That is what the qualifier on the good figure is telling you: 97% recall with the misses concentrated in conversations over 50 turns is a better result than 97% with the misses scattered evenly, because the damage is confined to a population you can identify and treat separately.

Where these policies fire matters as much as which one you choose. The summarisation and extraction calls are model calls with their own latency, and running them inline on the turn that crosses the budget means a user waits for two generations instead of one. Both can run after the response has been streamed, against the turn that just completed, so the compressed history is ready before the next turn needs it and the user never waits. The refresh should also land on a fixed boundary — every tenth turn, at the same watermark rule — rather than opportunistically, because a prefix that changes on an unpredictable turn is a prompt cache miss on an unpredictable turn, and the saving from compression is partly eaten by the saving it destroys.

The caching mechanism underneath that is worth spelling out, because it makes the placement decisions above pay off somewhere unexpected. A cache hit requires a prefix identical to one already processed, and the moment a token inside that prefix changes, everything from that point to the end of the array has to be processed again. The summary sits near the front, ahead of the verbatim turns, so replacing it invalidates every token behind it. The facts block sits near the end, so re-rendering it on each turn invalidates only the tail. Two placements chosen for how a model attends to a long input turn out to have opposite consequences for what a refresh costs, and the pinned design is the one that is cheap to refresh often.

In practice the shipped system converges on a combination — pinned facts as the spine, the last ten to twenty turns verbatim, and a rolling summary carrying the narrative in between — and the reason to reason about them separately first is that the combination's cost and failure modes are the union of theirs. Pinecone's Conversational Memory for LLMs walks the buffer, window and summary variants with runnable code, and RedisVL's Manage LLM Message History shows the storage and trimming side against a real store, if you want a second treatment of the mechanics.

None of this is the model remembering anything. Every strategy here is a decision your application makes about what to put in an array, and the model's apparent memory is entirely a function of that decision — which is worth keeping in view when a discussion starts to be about "giving the assistant long-term memory". The broader field around agent memory, which splits it into working, episodic, semantic and procedural kinds, is a taxonomy for organising these same decisions rather than a way out of them; IBM Technology's The Four Types of Memory Every AI Agent Needs is a compact tour if you want the vocabulary.

Summarisation, pinned facts and retrieval are three ways of spending the same small amount of money to buy different losses, and the choice is made properly only when the loss has been named against the product. A coaching assistant can afford to forget a number and cannot afford to forget a tone; a refund assistant is the reverse, and a lookup-shaped assistant over a 400-turn history has no choice at all. What none of them can do is fail loudly, which is why the extraction-recall measurement above belongs before the launch rather than after the first escalation.

There is a consequence of adopting any of them that lands squarely on the earlier chapters' guarantees. Once assembly involves a summary produced by a model call, or a facts block produced by an extraction call, or a top-k retrieval against an index that has since been rebuilt, the outbound array has stopped being a pure function of the rows in your messages table. Chapter 1's restart test still passes for the raw path, and yet the array you sent on 14 March is no longer something you can regenerate on demand. The next chapter is about being asked to produce it anyway.