Recency is not relevance, and top-k lets it pass for it

A memory store can be correct in every way this topic has described so far and still produce a wrong answer. Every fact keyed to a subject and an attribute, supersession links on every change, per-class retention, nothing in it that is known to be false — and the agent books a restaurant that one of the party cannot eat at.

The reason is that everything up to here has been about which facts the store holds. What reaches the model on a given turn is decided by something else: a ranking function that scores each stored memory against the query and keeps the best few. That function is usually a blend of embedding similarity and a recency bonus, it is usually written once and never revisited, and it is systematically wrong about the class of fact that matters most.

What the ranking function did#

An assistant that books team lunches was asked to find somewhere for six people on Thursday. It booked a Thai restaurant, and Priya, who has a severe peanut allergy, spent the meal eating plain rice.

The allergy was in the store, correct, unexpired, at confidence 0.97. Here is what retrieval returned with k = 6, where k is the number of memories the read path is allowed to put in front of the model and everything below it is discarded.

The scoring function was 0.7 · cosine + 0.3 · exp(−age_days / 90), which is close to what most memory layers ship by default. Every column of that table is reconstructible from it, which is worth checking once so the rest of the argument can lean on the arithmetic. The recency column is the exponential applied to the age: a day old gives almost one, ninety days gives about a third, and four hundred and eighteen days gives a hundredth. The score column is then seven tenths of the cosine plus three tenths of the recency, so the top row is seven tenths of 0.71 plus three tenths of 0.99, and the allergy is seven tenths of 0.42 plus three tenths of 0.01.

The collapsed row in the middle of that table is worth expanding mentally rather than skipping. Ranks seven to ten are four notes about past bookings and venues, all scoring between about three tenths and four tenths, and they are collapsed because individually they are uninteresting. Collectively they are the mechanism: four memories that are about restaurants, and therefore topically close to the query, sit between the cut-off and the fact that decides the answer. They are not noise to be filtered — every one of them is a real, current, correctly stored memory of exactly the kind a lunch bot should have. The allergy is buried under the store working properly.

Under that function the allergy loses twice over. Its cosine against "book somewhere for the team on Thursday" is 0.42, because the query is about scheduling and venue while the memory is about a medical condition, and there is very little lexical or topical overlap between those. Its recency term is effectively zero, because it was recorded fourteen months ago and has had no reason to be repeated since — a peanut allergy is exactly the kind of fact that gets stated once. Every mechanism in the ranking function penalises the memory that decides the answer.

It is worth separating how much of the loss each mechanism owns, because the answer decides which fix can possibly work. The allergy trails the top row by about half a point of score. Split that: the cosine gap is 0.71 against 0.42, and seven tenths of that difference is about two tenths of a point. The recency gap is 0.99 against 0.01, and three tenths of that difference is just under three tenths of a point. So recency owns more of the deficit than similarity does — but the recency term is bounded by its own weight, and the most it can ever contribute to any memory is three tenths.

Give the allergy the best recency score in the table and its total comes to a little under six tenths, which on this query would place it fourth and inside k = 6. That looks like the fix, and this is the trap the rest of the chapter is built to escape: the arithmetic that solves the example does not solve the problem, because the cosine gap is a property of this particular wording and moves with it. The suite further down scores class-aware recency at 44 of 60 cases, which is a large improvement and a long way from enough.

Two different mistakes wearing one name#

Similarity is a measure of what a memory is about. Relevance is a measure of whether a memory changes the answer. Those two coincide often enough that embedding search works at all, and they come apart most sharply for constraints, which are the memories that matter most. A constraint is relevant precisely because it rules something out, and ruling something out does not require sharing vocabulary with the question — the answer "not Thai" is produced by a fact about peanuts, and nothing about peanuts has to appear in a request for a Thursday booking.

The recency term makes it worse in a specific way. Chapter 5 established that different classes of fact go stale at different rates, and a recency boost is a crude approximation of that: it assumes newer means truer. For active_project or working_context that assumption is roughly sound, since a note from yesterday about the current sprint really does supersede one from March. For an allergy, an accessibility requirement, or a legal restriction, age carries no information about truth at all, and applying an exponential decay to the ranking of such a fact is asserting something false about it.

The uncomfortable part is that the store already knows the class of every fact. Chapter 5 assigned classes precisely so that retention could differ by class, and the ranking function was written without consulting the same column — so one part of the system knows that an allergy does not decay and another part is discounting it at a third every ninety days.

The fix has three parts and they are not interchangeable. Class-aware recency applies the decay term only to classes whose truth is time-sensitive, and sets it to zero for the rest. A reranker — a cross-encoder scoring the query against each candidate jointly rather than through independent embeddings — reads "book somewhere for six people" against "Priya has a severe peanut allergy" and can score it as decisive, which a bi-encoder of the kind ordinary vector search uses, embedding query and memory separately and then comparing two vectors that never met, cannot, at a cost of roughly 90 ms for 50 candidates. And a constraint channel bypasses ranking entirely: certain classes are fetched by rule on subject id and injected unconditionally, because there are rarely more than a handful of them and the cost of omitting one is unbounded.

CONSTRAINT_CLASSES = {"allergy", "accessibility", "legal_restriction",
                      "do_not_contact", "hard_preference"}

def build_context(subject_id, query, k=6):
    constraints = store.fetch_by_class(subject_id, CONSTRAINT_CLASSES)  # ~4 rows
    candidates  = store.vector_search(subject_id, query, limit=50)
    ranked      = reranker.score(query, candidates)[:k]
    return dedupe(constraints + ranked)

The constraint fetch is not a ranking with a very high weight; it is a different query shape, and that is the whole point. Ranking asks which memories are most relevant and returns a fixed number of them, so every memory competes with every other and the count is capped. Fetching by class asks which memories belong to these classes and returns all of them, so a constraint competes with nothing and the count is whatever the person has. The first shape can always starve a constraint given enough distractors; the second cannot, by construction.

The dedupe at the end of build_context is not tidiness either. A constraint can be reached twice, once by the class fetch and once by the vector search, whenever it happens to be topically close to the query — which is precisely the easy case, where the user asked about allergies directly. Sending it twice wastes tokens and, worse, gives a repeated fact more apparent weight than the ones stated once, which is a bias in favour of constraints that are on topic and against the ones that are not, reintroducing the chapter's problem at a smaller scale.

That only works if the count stays small, and it does. Across 140,000 users in that deployment the median was 2 and the 99th percentile was 11, which at roughly forty tokens a memory is at most a few hundred tokens injected on every turn.

Raising k instead is the tempting alternative and it does not work. Reaching rank 11 requires k = 12, and the allergy was at rank 11 for this query; for a different phrasing it lands at 19, so any fixed k is a bet on the wording. Setting k = 50 costs about 2,000 tokens a turn at roughly 40 tokens per memory — the table below reports about 1,760, which is the same figure counted as the marginal cost over the six memories that were already being sent. It also dilutes the context with a dozen irrelevant facts, and still leaves the decisive one buried in the middle of a long list where models attend to it least.

The test that catches it, and the one that does not#

The obvious test is the one almost everybody writes:

def test_allergy_is_retrieved():
    store.write(subject, "Priya has a severe peanut allergy")
    results = store.search(subject, "does anyone on the team have allergies?")
    assert "peanut" in results[0].text

That passes on the system that booked the Thai restaurant, and it passes for two independent reasons, either of which alone would be enough. The query is a near-paraphrase of the memory, so its cosine is around 0.93 and it is rank 1 under any weighting of any scoring function. And the store the fixture built contains a single memory, so any ordering of one item is correct, including one that ignores similarity entirely. What the test exercises is that the embedding model maps related sentences close together, which was never in doubt. It exercises nothing about ranking, because nothing had to be ranked.

A test that can fail on a broken system needs four properties, and each one closes off one of the ways the test above succeeds without measuring anything. The query must not paraphrase the decisive memory, so that similarity alone cannot surface it. The store must contain distractors that are both more recent and more similar than the decisive memory, so that the scoring function is actively pulled the wrong way. k must be small enough to force an exclusion, since a k larger than the store tests nothing. And the assertion must be on the agent's action or answer rather than only on the retrieved set, because retrieval returning the fact and the model ignoring it in position nine is the same failure to the user.

CASES = [
    Case(
        query="book somewhere for the team on Thursday, six people",
        decisive=Memory("Priya has a severe peanut allergy",
                        klass="allergy", age_days=418),
        distractors=[
            Memory("Team loved the Thai place on Greek St", age_days=1),
            Memory("Thursday standup moved to 11:00",       age_days=6),
            Memory("Team lunch budget is £25 a head",       age_days=34),
            Memory("Last offsite dinner was at a tapas bar", age_days=45),
        ],
        forbidden_actions=[Books(cuisine="thai"), Books(cuisine="satay")],
    ),
    # ... 59 more
]

@pytest.mark.parametrize("case", CASES, ids=lambda c: c.query[:40])
def test_decisive_memory_survives_recent_distractors(case, agent):
    store = seed(case.decisive, *case.distractors)
    ctx   = build_context(SUBJECT, case.query, k=6)

    assert case.decisive.id in {m.id for m in ctx}, "decisive memory not retrieved"

    result = agent.run(case.query, context=ctx)
    for forbidden in case.forbidden_actions:
        assert not result.matches(forbidden), f"violated constraint: {case.decisive}"

Sixty cases is enough to produce a rate rather than a pass or a fail, which is the more useful form for this. Building them takes a day: the decisive memories come from the constraint classes already in the schema, the distractors come from real memories of the same users sampled from the last week, and the queries are written by hand to avoid paraphrasing.

Scored against the deployment above, the linear blend retrieved the decisive memory in 33 of 60 cases at k = 6 and produced an action that violated the constraint in 21. Those two numbers do not complement each other, and the gap between them is instructive: sixty cases minus twenty-one violations leaves thirty-nine that behaved acceptably, while only thirty-three had the decisive memory in context at all. So at least half a dozen cases avoided the forbidden action without ever seeing the constraint — the agent picked something safe for unrelated reasons. A suite that asserted only on behaviour would have counted those as successes and reported a system that was right by luck.

Adding the constraint channel took retrieval to 60 of 60 and violations to 4, all of which were the model seeing the constraint and disregarding it, which is a prompt problem rather than a retrieval one. Adding the cross-encoder rerank on top of similarity search alone, without the constraint channel, reached 53 of 60 at a cost of about 90 ms per turn. Those two results are worth holding side by side: the expensive fix that reasons about relevance gets most of the way there, and the cheap fix that refuses to reason about it at all gets the rest, because the remaining cases are ones where no amount of reasoning about a query would rank a constraint above the notes that are actually on topic.

Compare the last row against each of the three above it, column by column, and it wins every comparison. It retrieves more than raising k and costs a twentieth of the tokens. It retrieves more than the cross-encoder and costs a thirtieth of the latency. It retrieves more than class-aware recency at a latency cost that rounds to nothing. That is a dominance rather than a trade, and it is what you would expect from the shape argument above: the other three are all attempts to make a constraint win a competition, and the constraint channel removes it from the competition.

Two things the table does not settle are worth reading out of it anyway. Retrieval is necessary and not sufficient — the last row reaches every decisive memory and still produces four violations, so the remaining failures are downstream of retrieval entirely. And every row's violation count is higher than its retrieval failure count implies, which is the same arithmetic as the paragraph above applied four more times: retrieval and behaviour are separate measurements and the suite needs both.

Where the constraint lands inside the assembled context turns out to matter as much as whether it is there, which is why the assertion is on the action. In the four residual violations, the constraint had been retrieved and placed after eleven other memories in a single undifferentiated block, and the model treated it as one more piece of background. Moving the constraint channel's output into its own labelled section at the end of the system message, under a heading that says these are requirements rather than notes, removed three of the four. That change costs nothing and is invisible to any test that asserts only on the retrieved set, which is the second reason the assertion belongs on the answer.

Where the constraint channel stops#

This is not an argument that the embedding model is inadequate. Swapping to a stronger model raises the allergy's cosine from 0.42 to perhaps 0.50, which changes its rank from 11 to 9 and changes nothing about the outcome. The gap between 0.42 and 0.71 is not noise to be squeezed out; it is a correct measurement of topical distance between a question about restaurants and a fact about a medical condition, and no better encoder will report otherwise. An encoder that scored those two as close would be worse at everything else it does.

Nor is it an argument against recency. Chapter 5's whole point was that some facts do go stale, and for those classes a recency term is the ranking-side expression of the same policy — it is right for working_context and wrong for allergy, and the error is applying one coefficient across classes that the store has already told you are different.

The constraint channel also has a boundary worth naming, and it is the same property that makes it work. It works because constraints are few and enumerable per subject, so fetching all of them is affordable and complete. It fails on any class where the count is unbounded: you cannot unconditionally inject every project a user has ever touched, and a class that grows with tenure will eventually exceed any budget you set. For those, ranking is the only available mechanism and the reranker is the tool, with the residual failure rate that the table above reports. The design decision, then, is not "channel or ranking" but which classes are small enough to be exempted from competition, and that is a question about the schema rather than about retrieval.

Every fix above assumes the retrieved fact is true. The failure it prevents is a true fact losing a ranking contest, and the test suite is built to detect exactly that. When a fact that is no longer true wins the same contest — high similarity, recent, high confidence, correctly ranked, and wrong — every check here passes, retrieval reports success, and the answer that comes out is fluent, specific, and built entirely on something that stopped being the case.