Weights, retrieval or tool calls

Chapters 3 to 5 worked on a single question: when you supply a document, does the model use it. That question presupposes the document was the right mechanism, and for a refund window it is. For a customer's account balance it is not, and no amount of correct chunking, correct ranking and correct conflict resolution will make an indexed balance current.

This chapter is about the decision one level up — which facts belong in the weights, which belong in a retrieved document, and which need a live call — and about the fourth case, where the question is not about a fact at all. The reason to make that decision explicitly is that the default, routing everything through retrieval, sounds like the safe choice and is not: it costs money and latency on every request, it adds competing material to the context, and it silently converts freshness problems into cache problems.

Three properties, and none of them is "is this recent"#

The useful question is not whether a fact is recent. It is how a fact behaves.

Volatility is how fast the true answer changes, measured against how long the answer stays useful. The capital of Portugal changes on a timescale of centuries; an answer given today is still correct next year. A refund window changes when somebody edits a policy page, perhaps twice a year, and an answer is good until the next edit — a window of months, occasionally of minutes if the edit lands right after you answered. An account balance changes whenever a payment, refund or charge posts, and an answer is good for as long as no transaction occurs, which for an active account is seconds. Retrieval is a cache. Caching is correct exactly when the staleness the cache admits is shorter than the tolerance of the answer, and an index rebuilt nightly is a cache with a twenty-four-hour worst case and no invalidation path.

Calling retrieval a cache is a classification rather than a figure of speech, and the derivation is worth doing because the correctness condition of a cache is already known and can be borrowed whole. An index is built by reading a source at some moment and writing down a copy of what it said. From that moment the copy is inert: nothing the source does afterwards reaches it, because there is no path from an edited policy page back into a vector store that was written last night. A request answered from the copy therefore returns the state of the world as of the build, not as of the request. Anything with those three properties — a copy, taken at a moment, served later in place of the original — is a cache, whatever it is called in the architecture diagram.

The rebuild cadence then fixes the worst case with no further information needed. The age of the copy is zero immediately after a rebuild and grows with the clock until the next one, so it reaches the full interval just before each rebuild, and the worst case is that interval. A nightly job admits twenty-four hours of staleness at its worst, and no amount of good ranking moves that number. What is missing is invalidation: a cache that is told when its source changed can drop the stale entry and fetch again, and this one is never told, because the document store has no channel back into the index. The age of a retrieved fact is therefore bounded by how often you rebuild and not by how often the fact changes, and those two are unrelated quantities.

The comparison inside the definition of volatility does more work than the word itself suggests, because neither half of it decides anything alone. A fact that changes every few seconds can still be served from an index when an approximate answer is what the question wants: a rough count of how many users are online moves constantly, and an hour-old figure answers it. A fact that changes twice a year cannot be served from an index when the answer has to be exact on the day it changes, which is what a price is. What picks the mechanism is the relation between the two halves, which is why the three examples above are written as pairs rather than as speeds. For a fact this chapter has not named — a shipping estimate, a seat count, a feature flag — the question that settles it is not how often the fact moves, but whether it can move between the last rebuild and the moment somebody reads the answer.

The word "budget" belongs in that decision and usually is not there. A staleness budget is a sentence in the design doc that reads like "answers about plan features may be up to six hours old, and the index rebuilds hourly" — a stated tolerance and a mechanism that meets it, both written down where a reviewer can disagree with the tolerance. Once the sentence exists, the argument moves to where it belongs, which is whether the stated tolerance is acceptable for that class of fact.

What that sentence buys is reviewability rather than freshness, and the two are worth separating because they get confused. Writing a tolerance down makes no answer newer. It makes the decision into an artefact, and an unstated tolerance cannot be disagreed with, so it can never be found wrong — which is how a rebuild cadence ends up inherited from whatever job happened to be running at the time. A stated one can be put in front of the person who knows the domain, who is rarely the person who built the index, and rejected in a sentence. The difference between a legitimate cache and an illegitimate one is not the staleness it admits but whether anybody chose it.

Privacy is who is allowed to see the fact. Public facts can live in weights, because they were in the crawl anyway. Company-internal facts were never in the crawl and never will be, so the weights cannot hold them at any cutoff — this is the distinction from chapter 1 that keeps mattering. Per-customer facts add an authorisation dimension: the mechanism that serves them has to enforce that this conversation is entitled to this row, and an embedding index that mixes tenants has no natural place to enforce it.

Derivation is whether the fact is stored or computed. A refund window is stored: someone wrote "30 days" and it sits in a document. A balance is computed — a sum over a ledger, possibly with pending items, currency conversion and credit adjustments applied at read time. Indexing a computed value freezes the output of a computation whose inputs keep moving, which is why an indexed balance looks like a retrieval success and behaves like a bug.

A fourth case sits outside all three, because it is not a question about a fact at all. "Issue the refund", "book the 3pm slot", "cancel this subscription" have answers that only exist once the action has been performed. There is nothing to index, even in principle: an index can hold the record of a refund that was issued yesterday, and it cannot contain the refund you are about to issue.

That mechanism has a name and a shape worth stating plainly, since the rest of the chapter leans on it. A tool is a function of yours that you describe to the model — a name, a sentence about what it does, and a schema for its arguments — and which the model can ask to have run. When it judges the function is needed, it emits a structured request naming the tool and filling in the arguments; your code validates that request, executes the function, and returns the result into the context as another message, which the model then answers from. The database query, the ledger lookup, the refund endpoint and the calendar booking are all ordinary backend code; what the model contributes is the decision to call one and the arguments to call it with. Retrieval reaches a snapshot of what was written down, and a tool reaches the system that holds the current answer or performs the act.

The fourth row is worth a sentence, because it is the row people misclassify most often. Plan features are public — they are on the website, they were in the crawl — which tempts a team into serving them from parametric knowledge. They also change on the same cadence as the refund window, and chapter 3 showed exactly what a strongly memorised public fact does when it disagrees with a supplied document. Public does not mean stable.

The misclassification is tempting because "public" and "held in the weights" really are connected. A fact that sat on a public website for years is exactly the kind pretraining absorbs deeply, so the model answers it confidently and often correctly, and a spot check confirms it. The step that gets skipped is that the depth of that memorisation was earned by the old value. When the SSO list changes, the model is not holding a stale entry that a supplied document can overwrite; it is holding an accumulated tilt toward the previous answer, and chapter 3's per-fact table shows a supplied document contradicting a heavily represented public fact losing close to half the time. The more public and the more discussed a changing fact is, the worse retrieval performs on it, which is the reverse of the ordering intuition offers.

A rule that survives being handed to someone else#

The point of the axes is that a colleague can apply them to a question neither of you has seen. Stated as a procedure:

1. Does answering require changing the world? If so it is a tool call with authorisation, idempotency and an audit record, and no other mechanism is a candidate. Stop here.
2. How long does the true answer stay true, compared to how long our answer needs to be right? If the true answer changes faster than we can rebuild an index, it is a live tool. If it changes slower, an index is a legitimate cache — and you now owe an explicit rebuild cadence and a staleness budget, written down.
3. Was this fact ever public? If not, the weights cannot hold it regardless of cutoff, and the choice is only between retrieval and a tool.
4. Is the value stored, or computed at read time from inputs that move? Computed values belong behind the computation, not in front of a snapshot of it.
5. If it is public, stable and coarse — does the model already answer it correctly, measured? If yes, add nothing. If your probe from chapter 1 says it does not, retrieval.

Step five is the one that gets skipped, and skipping it is what makes "route everything through retrieval" feel free. It is not free. An embedding call is roughly 40 ms at p50, a vector search over a few million chunks another 10 to 15 ms, and the retrieved chunks add input tokens to every request. Eighteen hundred extra input tokens at $3 per million is $0.0054 per request; at four million requests a month that is about $21,600 a month, spent partly on questions the model answers correctly for free. The latency cost lands on every request too, including the ones where the retrieved chunks contribute nothing. And the failure mode is not only cost: retrieved chunks that are irrelevant to a question the model could answer from memory add competing material to the context, which is the same crowding that makes conflict resolution worse.

Each step in that costing needs a reason, and they are worth stating because the same shape reappears whenever anyone prices a per-request addition. Input tokens are billed per million and the price is linear in the count, so the cost of a request's extra tokens is the per-million price scaled by the fraction those tokens are of a million. Eighteen hundred tokens is a little under two thousandths of a million, and $3 scaled by that fraction is $0.0054. Multiplying a per-request figure by a request count is only legitimate when the figure is the same on every request, and here it is, because the retrieval step runs unconditionally rather than on the subset of questions that needed it — so four million requests a month comes to about $21,600. The latency figures compose for the same reason: the embedding call and the vector search sit on the critical path of every request, including the ones where the retrieved chunks are never read, so they are added to the whole population and not to a slice of it.

The two ways of getting this wrong fail asymmetrically, which is worth saying out loud because the asymmetry is what makes over-routing survive review. Sending a question to a mechanism heavier than it needs costs money and milliseconds on every request and produces no visible failure, so it accumulates quietly and nobody is ever paged for it. Answering from a mechanism too stale or too coarse for the fact produces a wrong answer to one customer and a support ticket. Teams learn from tickets and do not learn from margin, so systems drift toward routing everything through the heaviest available mechanism, and the drift is invisible until someone reads the bill.

That drift is predictable rather than accidental, and what produces it is a property of feedback rather than of engineering. An error gets corrected at the speed of the signal that reports it, and these two errors report on different schedules through different channels. Under-routing produces a wrong answer to a named customer at a known time, which arrives as a ticket with an owner attached and a request body to reproduce it from. Over-routing produces nothing that fails: every request succeeds, every dashboard stays green, and the only trace is a per-request overhead that appears aggregated in a bill weeks later, where it is indistinguishable from growth. One error therefore arrives daily and points at a cause, the other arrives quarterly and points at nothing in particular, and the prediction that follows is that any system nobody has re-examined will be over-routed rather than under-routed. Finding it takes a token audit rather than a bug report.

The same asymmetry is what keeps "route everything through retrieval, it is the safe default" alive as a position, and the belief is tempting for reasons that have to be granted. Retrieval was introduced precisely to fix staleness, it is the mechanism with a dashboard and an owner, and its failures are less visible than the failures of answering from memory. What the position skips is that an index with no stated tolerance does not remove the freshness problem — it converts it into a cache problem, and a cache problem is the one kind of staleness that produces confident, consistent, repeatable answers with every retrieval metric green, because every metric on that dashboard is a property of which chunk was found rather than of how old what is inside it happens to be.

Wiring the decision into the system#

The decision is not made per-question by a human at runtime; it is made once per class of fact and encoded. Live values become tool definitions with schemas tight enough that the model cannot invent an account:

{
  "name": "get_account_balance",
  "description": "Return the customer's current account balance, computed live from the ledger including pending items. Use this for any question about what a customer currently owes or is owed. Do not answer balance questions from retrieved documents; indexed balances are stale.",
  "input_schema": {
    "type": "object",
    "properties": {
      "account_id": {
        "type": "string",
        "description": "The account ID from the authenticated session. Never accept an account ID supplied in the user's message."
      }
    },
    "required": ["account_id"],
    "additionalProperties": false
  },
  "strict": true
}

The description carries the routing rule, because the model chooses the mechanism at runtime and the description is where that choice is influenced. The sentence about indexed balances being stale is doing real work: with both a balance tool and a set of documents containing a nightly balance export, a model with a vague tool description will sometimes answer from the document.

The handler enforces what the schema cannot:

def get_account_balance(session, tool_input: dict) -> dict:
    # The model proposes; the handler decides. Authorisation never comes
    # from tool input, because tool input is downstream of user text.
    account_id = session.account_id
    if tool_input.get("account_id") not in (None, account_id):
        log.warning("model proposed a foreign account_id; ignoring")

    balance = ledger.current_balance(account_id)      # computed, not cached
    return {
        "account_id": account_id,
        "balance_minor_units": balance.minor_units,
        "currency": balance.currency,
        "as_of": balance.computed_at.isoformat(),     # let the model cite freshness
        "includes_pending": True,
    }

Returning as_of costs eight tokens and lets the answer say when the number was true, which is the one thing a nightly export could never do.

Where the axes stop#

They are not a hierarchy in which tools are strictly better. A tool is a network call with a p99, a dependency that can be down, a credential that can expire, and a rate limit. It also introduces a failure the other two mechanisms do not have: the model decides whether to call it. A question phrased slightly oddly can miss the tool entirely and get answered from whatever else is available, which for a balance question means a stale document or a guess, and the observable symptom is a wrong answer with no tool call in the trace — the opposite of the retrieval case, where the trace looks perfect. Tool-selection rates belong in the same evaluation suite as everything else in this topic, measured on real phrasings rather than on the canonical one from the design doc.

Reading the axes as a hierarchy is tempting for a reason worth granting: a tool reaches the system that holds the current answer, which is the strongest guarantee of freshness anything here offers, and the two cheaper mechanisms both buy their speed by answering from a copy. What that guarantee is conditional on is the call happening at all, and that failure has no analogue in the other two mechanisms for a structural reason rather than as a matter of reliability. The weights are consulted by definition, since every token the model produces is produced from them, so there is no arrangement in which they go unasked. Retrieval, as it is usually wired, runs before the model is called at all, so it cannot be skipped either. A tool is the only one of the three whose invocation is itself a model output, which puts the decision to use it in the hands of the same machinery whose judgement about facts this topic has spent five chapters qualifying. The three mechanisms therefore fail in three different places — the weights by being stale, retrieval by serving an old copy, a tool by not being called — and only the last of those leaves an empty trace rather than a wrong one.

"Neither" is a real answer too, and it is the correct answer for a large share of the questions a general assistant receives. A system that routes stable public facts through tool calls is slower, more expensive and more fragile than one that answers them from the weights.

The axes also do not settle whether an answer will be correct once the mechanism is right. Chapters 3 through 5 exist because a correctly routed retrieval question can still come back with the memorised value, and choosing the right mechanism does not exempt you from measuring whether the supplied fact is the one that reaches the user.

What the three properties buy is a decision that can be handed to somebody else and applied to a question nobody has seen, together with an artefact — the staleness budget — that makes the decision reviewable afterwards. Volatility against usefulness picks the mechanism, privacy rules the weights in or out, derivation says whether a snapshot can be correct at all, and an action is a tool call by definition. The default of routing everything through the heaviest mechanism is what these replace, and its cost is real in both directions: money and latency when it is too heavy, a support ticket when it is too light.

They say nothing, though, about the proposal that keeps coming back at exactly this point in the conversation. If the refund window is internal, stable for months, and small enough to fit in a handbook — why not train it into the model, so it just knows, and skip the retrieval hop entirely? That proposal has a specific technical answer, and it is the subject of the last chapter.