A stale memory produces a plausible answer, not a visible error

Every failure in this topic so far has left a trace. A conflict leaves two rows. A bad match leaves a supersession that can be undone. A ranking failure leaves a retrieved set you can inspect. Each of them is visible to somebody who goes looking, which is why each of them can be measured and fixed.

Decay leaves nothing. A fact that quietly stopped being true is still one row, still correctly keyed, still within its retention policy, still the most relevant memory for the question it is about. Retrieval returns it, the model uses it, and the answer that comes out is well-formed and wrong. Every mechanism of the previous six chapters reports success on that case, and so does every automated check normally run over an answer, which is why detection for it has to be built on purpose rather than inherited.

Every mechanism reports success#

On 14 January a customer told a support agent they were on the Team plan, and the extractor wrote plan_tier = Team with a first-person source, confidence 0.94, class slow_attribute. On 3 March they downgraded to Pro in the billing portal, which is a web form that has never once talked to the memory system. On 9 April they asked the agent whether they could turn on single sign-on (SSO), the feature that lets a company's staff reach the product with their existing corporate login, and got this:

Yes — SSO is included on the Team plan. You'll find it under Settings → Security → Single sign-on, and you'll need to be an organisation owner to complete the setup.

Everything in that reply is true of the Team plan and none of it is available to them. They spent twenty minutes looking for a menu item that is not rendered for their account, then filed a ticket, which a human answered eleven days later.

Run the failure against the machinery of the previous six chapters and every mechanism reports success, each for its own reason. Chapter 2's matcher never fired, because no candidate write arrived — the downgrade happened in a form and generated no utterance to extract. Chapter 3's supersession never triggered for the same reason, since there was nothing to supersede with. Chapter 4's evidence counters never moved, because no observation contradicted the stored value.

Chapter 5's retention left the fact alone, correctly by its own policy, and it is worth doing that arithmetic because it is the one place where a number might have saved this. From 14 January to 9 April is 85 days, and a slow_attribute at 85 days of age decays to 0.94 × 0.92 = 0.86, comfortably in the assert-plainly band. The factor of about nine tenths is what a half-life of two years does over a quarter of a year: the fact has aged by roughly a ninth of a half-life, so it has lost about a twelfth of its confidence. No plausible half-life for a plan tier would have pulled it below that band in under three months, which is the honest limit of what retention can do here.

Chapter 6's ranking put it at the top of the retrieved set, which is right, because it is genuinely the most relevant memory to a question about plan features. That is the uncomfortable part of the trace: the ranking function was not fooled and was not badly configured. It correctly identified the memory that decides the answer, and the memory that decides the answer was wrong.

Why the usual detectors miss this#

Nothing in the request path has an opinion about whether a memory is still true. The store returned rows, the ranking function ordered them, the model consumed them and produced tokens. There is no exception to catch, no null to check, no timeout, and no place in the code where the question "is this still the case?" would naturally be asked — every component's contract is satisfied by returning what it was asked for.

The evaluation layer misses it for a more interesting reason, and this is the belief worth naming because it is the one most teams are actively relying on. The standard automated check on a retrieval-augmented answer is groundedness: does every claim in the response trace to something in the retrieved context? This answer scores perfectly on that, since the claim about SSO follows directly from the stored fact.

Look at what groundedness holds constant. It is a relation between the answer and the context, and it is exactly as satisfiable when the context is a year out of date as when it is correct — the score does not move at all as the underlying fact goes stale, because the fact is not one of its inputs. Meanwhile the property that decides whether the user is helped, namely whether the context is still true, varies freely. Faithfulness metrics, attribution scoring and citation checkers all reward this response, because they are built to catch a model that invents things and this model invented nothing. The defect is upstream of anything they inspect, and adding more of them changes nothing.

Human review of transcripts misses it as well. A reviewer reading that exchange sees a well-formed, appropriately specific, on-policy reply, and unless they independently look up the customer's plan they have no reason to flag it. In a sample of 400 transcripts scored by two reviewers on that product, inter-rater agreement on "is this answer correct" was 0.91, and both reviewers marked this class of answer as correct every time it appeared. High agreement is worth noting rather than reassuring: two reviewers agreeing means they were applying the same criterion consistently, and the criterion they were applying does not include the fact.

Read the right-hand column and the table splits cleanly in two. The first three rows have no latency at all, because they never detect. The last three do detect, and every one of them detects by the failure having already happened to somebody — a reviewer who happened to check, a user who happened to know, or a customer annoyed enough to escalate. So there is no row that catches this before the answer goes out, which is the argument for building detection deliberately: the ambient monitoring a well-run service already has is blind to this entire class.

It is also the sense in which a stale memory is worse than a retrieval miss. When retrieval returns nothing, the agent says it does not know and the user asks again, so the failure is visible and self-correcting within one turn. A confident wrong answer terminates the conversation successfully from the system's point of view, which is to say the system's notion of success and the user's have come apart and only the system's is being measured.

Checking the value against the thing that owns it#

The strongest control is available only for facts that have an authoritative source, and where one exists the useful reframing is that memory is a cache of it rather than a record of it. Cache entries carry an invalidation path; this one had none, and everything below is an attempt to give it one.

VERIFIERS = {
    "plan_tier":       lambda uid: billing.get_subscription(uid).tier,
    "seat_count":      lambda uid: billing.get_subscription(uid).seats,
    "shipping_city":   lambda uid: crm.get_contact(uid).city,
    "account_status":  lambda uid: billing.get_subscription(uid).status,
}
VERIFY_TTL = 60  # seconds; one call per user per minute at most

def resolve(fact, uid, now):
    verifier = VERIFIERS.get(fact.attribute)
    if verifier is None:
        return fact                      # nothing owns this; decay is all you have
    fresh = cache.get_or_call(uid, fact.attribute, verifier, ttl=VERIFY_TTL)
    if fresh != fact.value:
        store.supersede(fact, new_value=fresh, source={"kind": "system_of_record"})
        return fact.with_value(fresh)
    store.confirm(fact, now)             # resets last_confirmed_at, restores confidence
    return fact

Two branches of that function matter and neither is the obvious one. When the fresh value disagrees, the fix does not overwrite the row — it goes through supersede, which is the mechanism chapter 3 built, so the correction carries a system_of_record source and remains auditable afterwards. And when the fresh value agrees, the code does not simply return: it calls confirm, which resets last_confirmed_at and restores the confidence that decay had been eroding. That second branch is what stops the verifier and the retention policy from working against each other, since a fact that is checked against billing every day should not be quietly decaying as though nobody had looked at it.

Four attributes and a sixty-second cache is the whole implementation, and it costs about 8 ms amortised across a conversation because the first turn pays for the call and the rest hit the cache. The sixty seconds is doing the arithmetic that makes this affordable: at most one call per user per minute, whatever the conversation does, so a twenty-turn exchange pays for one lookup rather than twenty.

The window itself is the one number to choose deliberately, because it sets what the control can and cannot catch. A verify window of a minute means the agent can be wrong about a plan tier for at most a minute after a change, which for a value a user changes in a form and then immediately asks about is the difference between working and not. Lengthening it to an hour makes the call rate negligible and reopens exactly the gap the whole chapter is about, on a shorter timescale. The right way to pick it is from how quickly a user acts on a change they just made, rather than from the load the billing service can bear, since the load is already bounded by the per-user rate limit rather than by the window.

The design question this raises is whether such facts belong in memory at all. Storing a copy of something a service already owns buys a small latency saving and inherits a staleness problem; storing a pointer — "this user has a paid subscription, look up the tier in billing before answering plan questions" — buys correctness at the cost of a call. The working compromise is to keep the value with a short verify_after window on the class, so that plan tier is treated as fresh for a minute and re-read beyond it, while genuinely immutable facts like a signup date are stored outright. That is the same distinction chapter 5 drew between classes, applied to a different question: how long a value may be trusted without checking, rather than how long it may be kept.

Making the memory visible in the answer#

Most facts have no system of record. Nobody owns the truth about whether a user prefers concise replies, is learning Rust, or has a manager named Dolores, and for those the only source that can correct the store is the user. That correction is free when the answer shows its working.

You're on the Team plan (from our conversation on 14 January) — SSO is included, under Settings → Security.

The parenthetical costs nine tokens and gives the user something to disagree with. The mechanism is worth stating plainly, because it is not persuasion: without the date, the user reads a claim about their account and has no way to tell whether the agent looked it up or remembered it, so a correction requires them to already suspect something. With the date, the claim carries its own provenance, and a user who changed plan in March recognises January as too old at a glance.

Instrumented on that product for six weeks, in-conversation corrections of stored facts went from 0.4 to 3.1 per thousand turns, which is close to an eightfold rise, and the stale-fact ticket rate fell by about 40%. The corrections themselves are the more valuable output, because each one is a labelled example of a fact that had gone stale, which is the ground truth the retention policy from chapter 5 has no other way of getting. A product that attributes its facts is generating its own training data for the half-life it should have chosen.

It is not free of cost, and it should be gated by class and by confidence. Attributing every fact makes the agent read like a system printing its state, users stop noticing the parentheticals when they appear on every sentence, and low-stakes facts do not warrant the interruption. Applying it to facts whose effective confidence has decayed below the assert-plainly band, or to any fact that is about to drive an action with a cost, concentrates the disclosure where a correction is worth having. That is the same idea as the ASK action from chapter 4, moved from a separate question into the answer the user was already going to read, which is why it costs a fraction as much: a confirmation question spends a whole turn and a user's attention, and this spends nine tokens inside a sentence they were reading anyway.

The SSO case is worth generalising before leaving it, because entitlements are the class where all of this is sharpest. An entitlement is owned by a system, changes without any conversation, is exactly the kind of thing users ask about, and produces an answer that is fully checkable and therefore fully embarrassing when wrong. Preferences go stale too, and a stale preference produces a slightly annoying answer. A stale entitlement produces twenty minutes of a customer looking for a menu that does not exist. Ordering verifier work by that difference rather than by how often a class is retrieved is what keeps a small set of verifiers worth more than a large one.

Two offline mechanisms complete the picture. A periodic staleness audit samples facts per class, verifies them against a source or a confirmation question, and publishes a per-class wrong rate, which is the same measurement chapter 5 used to set lifetimes and is worth running continuously rather than once. And a stale-store eval fixes the problem in CI: build a fixture where the store deliberately contains facts the tools would contradict, and assert on behaviour — the agent must call the billing tool, or hedge, and must not assert the stored value. Asserting on the tool call rather than on the text matters, because a model can produce a hedged sentence while still acting on the wrong value, and hedged language is the easiest thing in this chapter to produce without changing anything that matters.

The class with no detector at all#

One class is still unsolved when everything above has been built. For facts with no authoritative source and no user correction, there is no detection mechanism in this chapter or anywhere else. If the user's manager changed in March and they never mention it, no verifier exists, no contradiction arrives, and the fact will keep being asserted until its class-level decay pulls it below the injection threshold — which chapter 5 set at 18 months for relationships based on how fast that class churns on average, and averages are exactly what an individual user is not.

Follow that through and the residual is unavoidable rather than merely unaddressed. A class-level lifetime is one number applied to a population whose members change at different rates, so for any setting there are users whose facts go stale long before the lifetime elapses and users whose facts are demoted while still true. Shortening the lifetime trades the first error for the second; it does not remove either. The honest position is that this class carries a residual error rate, that bounded lifetimes and periodic confirmation reduce it rather than eliminate it, and that the size of the residual is something you measure per class instead of assume.

That measurement is the thing to build first, and it changes what everything else is worth, because a per-class wrong rate tells you which of the mechanisms in this topic are worth their cost on your data. In rough order of return: attribute names and cardinality for the twenty or so facts that drive actions, because without them chapters 1 through 4 have nothing to operate on; supersession links, because they cost one column and make every later decision reversible; per-class retention with demotion rather than deletion; a constraint channel at read time; verifiers on the handful of attributes that have a system of record; and the stale-store eval wired into CI so the whole arrangement stops silently regressing.

That ordering is not arbitrary. Each item removes a specific failure with a name and a trace, and each depends on the ones above it: retention needs classes, which need attributes; verifiers need supersession to write their corrections through; the CI eval needs all of it to have something to assert against. And each is cheap next to the eleven days it took to answer one ticket about a menu item that was never going to appear.