Memory — evaluation

Read these first

• Memory — as an attack surface
• Multi-turn and conversational evaluation

What you will know

• Measure whether memory improves task success at all

  Compare equivalent tasks with memory enabled and disabled on a held-out set rather than treating retrieval or write rates as success.

• Build an eval that exercises write-then-read across sessions

  Persist the write, start a separate session, and verify that the later task depends on retrieving and using the stored information.

• Detect a memory system that is net-negative

  Look for lower task success, increased correction rates, or unnecessary token and latency costs caused by irrelevant, stale, or conflicting context.

• Instrument memory evals to attribute failures to write, retrieval, or use

  Record what was stored, which candidates were returned, and whether the model incorporated them so a failed answer is not misdiagnosed.

• Decide to remove a memory feature based on data

  A rollback is justified when measured gains do not offset quality regressions, operational cost, latency, privacy exposure, or maintenance burden.

Chapters

1. 1

   Retrieval hit rate is activity; task success is the measurement

   Both numbers a memory team likes to show, how often retrieval fires and how many facts are stored, are produced by the feature merely running. The question memory has to answer is whether the same held-out tasks succeed more often with it enabled than disabled, which means building matched task pairs before anything else. This chapter sets up that comparison and the eval noise band that will decide later arguments.

   Read chapter
2. 2

   A held-out set too small cannot resolve two points

   The previous chapter promised a noise band; this is where you compute one. The same tasks run twice do not produce the same score, because sampling varies, retrieval order varies, and a model-graded success criterion adds judge variance on top, so a 74-to-76 difference may be nothing at all. Size the held-out set against the smallest difference you would act on, run the with-and-without arms paired on identical tasks and analyse them as paired, and repeat runs to separate the memory effect from the variance underneath it. What comes out is the band every later argument in this topic is measured against, and the reason a flat result can be conclusive rather than merely unresolved.

   Read chapter
3. 3

   An eval inside one session tests context, not memory

   State a fact and ask about it two turns later and the answer is in the transcript; the store could be empty and the test still passes. A real write-then-read eval persists the write, ends the session, and starts a fresh one whose task cannot be completed without retrieving the stored fact. Watch for the softer version of the same trap: evaluating on the very conversations that produced the writes.

   Read chapter
4. 4

   Write, retrieval and use are three failures with three fixes

   The agent ignored a preference set last week, and the cheapest check is the one people skip: look in the store and see whether the record exists at all. Logging what was written, which candidates retrieval returned and at what rank, and whether the generated answer used them turns one indistinguishable failure into three diagnosable ones, each with a different repair in the gate, the ranker, or the prompt.

   Read chapter
5. 5

   A net-negative memory system shows flat success and higher cost

   The failure mode is not a crash, it is a shape in the numbers: task success unchanged inside the noise band, correction turns per conversation rising, and every retrieving task carrying extra input tokens and latency. This chapter reads that pattern back to its causes, namely irrelevant retrievals, stale records, and context that contradicts the current request, and shows how per-task deltas expose it where averages do not.

   Read chapter
6. 6

   A memory store costs more than tokens and milliseconds

   The rollback argument trades measured gains against several things, and two of them appear on no dashboard the earlier chapters built. A store of user facts is retained personal data: it carries deletion and retention obligations, a cross-session and cross-user leakage surface, and an incident whose blast radius is the store itself. It also carries a standing maintenance bill — schema and embedding migrations, backfills, the write-then-read eval suite you now have to keep green, and on-call for a stateful component. Put both on the table with real figures here, because the next chapter asks you to weigh them against two points of task success.

   Read chapter
7. 7

   Two points of task success against 400ms and 1,500 tokens

   74% to 76% is either a real gain or eval noise, and until you know which, the added latency and input cost cannot be weighed against anything. This chapter frames the decision in order: significance first, then cost, then the maintenance, privacy exposure and incident surface the earlier chapters produced. It ends with the harder move, retiring a feature the team is proud of, and choosing what survives the rollback.

   Read chapter