Route per feature, and re-run the numbers when the model changes

Six months ago the platform group published a one-line architecture decision: always fill the window, retrieval is legacy. It was defensible at the time — context windows had grown by an order of magnitude, caching had arrived, and the retrieval pipelines in the product were genuinely a maintenance burden that nobody enjoyed owning. Eleven features migrated. The indexes were decommissioned in month three to stop paying for the vector store.

Today model spend is up roughly 7×, p95 is 11 seconds across the affected surface, and the loudest quality complaints all concern long documents. The obvious proposal on the table is to reverse the decision and move everything back, which would be the same mistake with the sign flipped: some of those eleven features are genuinely cheaper and better stuffed, a mandated migration back would regress them, and the organisation would spend another two quarters rediscovering feature by feature that the answer varies. The defect was that a single rule was applied to eleven workloads with different corpus sizes, session lengths, freshness requirements and latency budgets, and the direction it happened to point is close to irrelevant.

The damage that is not on the invoice#

The 7× is the easy part to explain and the least interesting. Three other things broke, and they are the ones that make the reversal proposal look attractive to people who have not looked closely.

Quality fell on precisely the features the policy was supposed to help most. The position and dilution effects of chapter 4 — where more context can lower accuracy and not only raise the bill — are functions of how many tokens are in the window, so the longer the document, the worse the migration went, which is why the complaint volume correlates with document length rather than with feature popularity. Nobody detected this during the migration because the migration had no evaluation gate; the policy said the new way was better, so measuring it would have been a formality.

Freshness became invisible. Several features hold cached prefixes over corpora that change during the working day, and the failure described in chapter 3 — a cache entry ages, and invalidation and TTL are terms in the cost model — applies directly: answers reflect a corpus as of the last cache write, with nothing to signal the lag. There is no incident for this because there is no signal for it, and the support tickets that result are logged as ordinary model errors.

The organisation lost the ability to change its mind. Decommissioning the indexes turned a routing decision into a rebuild project, so every subsequent conversation about a specific feature ran into "that would mean standing up retrieval again", and the answer was always no. This is the part worth generalising: an architecture policy that removes one of the options from the building also removes the ability to act on any future measurement, which is why keeping a thin retrieval path alive for at least one feature is a real operational requirement rather than an aesthetic preference.

Underneath all three sits one tempting belief: that a single well-argued architecture decision defends an organisation against drift, so that having settled the question once, carefully, with reasons, you are protected from settling it eleven separate times. It is tempting because the decision really was defensible when it was written. It is wrong because the decision is a constant while the thing it is applied to varies — corpus size, queries per session, freshness budget and latency budget all differ across eleven features, and a rule that does not read them cannot track them. The aggregate hides that perfectly. One organisation-wide spend figure can move by 7× while the per-feature verdicts underneath it point in opposite directions, and that single number is what both sides of the reversal argument are holding.

The two terms a routing decision compares#

Strip the argument down to what is being chosen. Each feature has a set of candidate routes — send retrieved passages, send a selected subset of documents, or send the whole thing and cache it — and the job is to pick the cheapest route that clears two bars: the answers have to be good enough, and the p95 has to fit the budget. In ordinary words, minimise cost subject to a quality floor and a latency ceiling, once per feature — and the per-feature part is forced, because every quantity in that sentence is read off one workload and none of them exists at the level of an organisation.

The cost side has a shape worth writing down, because it is simpler than people expect. Every route in this topic bills the same way: some number of input tokens goes into the model, each token is charged at a price per token, and a multiplier is applied depending on whether that token was written into a cache, read back out of one, or sent at full rate. Call the token count T, the price per token p, and the multiplier m. Then the cost of a route is T · p · m, and there is no fourth term — no per-request fee, no fixed overhead. Each of the three worked features below reproduces its own headline figure from those three numbers alone with nothing left over, which is the evidence that the model is complete rather than merely plausible.

That linearity licenses something specific, and it is why the routing table has no money column. Comparing two routes for one feature means comparing T₁ · p · m₁ against T₂ · p · m₂. The price per token is the same p in both, because both routes send input tokens to the same model, so it divides out of the comparison exactly and leaves T₁ · m₁ against T₂ · m₂. A uniform change in price therefore cannot flip a route. It scales both sides by the same factor and the cheaper route stays the cheaper route, whatever the price does.

The cancellation is exact and narrow, and the two things it does not cover are where the honest residual lives. Prices do not in practice move uniformly: the multiplier charged for reading a cached token can change on its own, and a change in m alone does not cancel, because m is the term that differs between the two routes. And the cancellation only applies where both sides of the comparison are costs. Where the decision is a quality gain weighed against a spend increase — which is what happens when the rule below returns measure — the price scales the money side while leaving the accuracy side untouched, so one term moves and the other does not, and decisions of that shape genuinely do flip when a price does.

Decisions of that shape also re-open all at once rather than one at a time, because p is a single shared input: the same number sits in the arithmetic of all eleven features simultaneously. A per-feature review cadence cannot catch a change in a shared input, since the change does not arrive feature by feature — which is the argument for watching the inputs rather than the features, and the job of the trigger list further down.

Why a cache write can make a session dearer#

The cache is where the multiplier m earns its keep, and also where the most confident wrong belief in this topic sits. A cached prefix is billed at two different rates: writing it in costs more than sending it normally, and reading it back costs much less. The chapter 2 arithmetic — a cached prefix only pays off if nothing mutable sits in front of it — gives the write multiplier as 1.25 and the record at the end of this chapter gives the read multiplier as 0.10. The belief to watch is that a discount of that size makes caching safe. It does not, and the reason is that the two multipliers are paid on different schedules.

Work it out with symbols first, because the arithmetic is short and the shape is what generalises. Let a session send the same prefix N times: once as a write, then N − 1 times as reads. Write the write multiplier as w and the read multiplier as r. The cached session costs T · p · (w + (N − 1) · r). The uncached session sends the same prefix at full rate every time, so it costs T · p · N. Divide the second by the first and both T and p cancel, exactly as they did above, leaving the saving as N / (w + (N − 1) · r). Caching wins when that ratio is above one and loses when it is below.

Two consequences fall out of that expression immediately. The first is that the saving depends on neither the price nor the size of the prefix — those cancelled — so a large prefix and a small one amortise at exactly the same rate, and a price cut does not make caching more attractive. The second is where the ratio crosses one. Setting N = w + (N − 1) · r and collecting the N terms gives N · (1 − r) = w − r, so the break-even count is (w − r) / (1 − r). With the write multiplier a quarter above full rate and the read multiplier a tenth of full rate, that lands a little above one query per session. Caching starts paying from the second query onwards, and at a single query it loses by exactly the write premium.

The stumble this sets up is precise. A team reads the read multiplier, sees a large standing discount, and treats it as a property of the cache. The multiplier is indeed constant — it applies at that rate to every read, always. What varies is the number of reads, and the number of reads is the dangerous quantity, because at zero reads the discount is never collected and only the write premium is paid. A blended cost-per-request figure across a product is the same trap one level up: it can hold perfectly steady while the mix of one-query and thirty-query sessions behind it shifts, because it averages exactly the quantity that decides the sign.

An analogy helps here provided its limit is stated with it. A cache write is a joining fee and a cache read is the member's rate: cheaper per use, but only worth paying if you use it enough, and the break-even count above is exactly the number of uses at which membership starts paying. Where the picture breaks is the horizon. A joining fee is paid once for the life of the membership, whereas a cache entry expires, so the write is paid again at every TTL boundary and again whenever the prefix changes. The amortisation window is the life of the cache entry, not the life of the feature, and that is precisely the correction chapter 3 makes to the cost model.

A rule an engineer can apply without asking you#

The replacement is a set of thresholds evaluated per feature, in order, with the first match winning. Every threshold below is a number rather than a judgement, which is what makes it usable by someone who has not read this topic.

Read the table by its gaps rather than by its rows. Rule 3 and rule 4 are the same quantity — queries per session — measured on either side of a hole, and the hole is where the teaching is. The break-even derived above sits a little over one query, which would put the threshold between one and two; the rule instead routes to retrieval at two or fewer and to caching at eight or more, leaving everything between three and seven to fall through to the later rules. The gap is a design decision and not a derived quantity. At two queries the cached session is cheaper, but by a margin thin enough that a single missed hit erases it, and the rule buys margin by refusing to act inside the band where the answer is close. That caching loses at one query is a requirement of the arithmetic; that the rule waits until eight is a choice about how much margin to demand.

The same distinction runs through rule 1. That a corpus larger than the window cannot be sent is a requirement of physics. That the ceiling is set at 60% of the window rather than at 100% is a decision, and it buys room for the question, the instructions and the output alongside the corpus, plus distance from the accuracy decay chapter 4 measures as the window fills. The upper edge of the selection band is a chosen line of the same kind: at 240% of the window, fitting the corpus means keeping a bit over two-fifths of it, and beyond that point selection is discarding more than half the material, which is retrieval wearing a different name.

Rule 5 is a margin of the same shape. Staleness is bounded by the TTL: an entry read just before it expires reflects a corpus as of one full TTL ago, which is the worst case rather than the typical one. Requiring the freshness budget to be at least twice the TTL leaves a full TTL of headroom on top of that worst case. The requirement is that staleness can reach one TTL; the factor of two is the decision.

Rule 2 is the one people most often leave out, and it is the reason a purely economic framing gets some features exactly wrong. A code review assistant asked whether a diff introduces a regression cannot be served evidence selected by relevance to the question, because the relevant thing may be the file that should have changed and did not. The same holds for summarising a document, checking a contract for missing clauses, and any comparison across a whole corpus. Retrieval answers questions of the form "what does this corpus say about X"; it cannot answer questions of the form "what is not in this corpus", and no amount of tuning changes that, because a retriever's output is defined by what matched.

Encode the rule as configuration rather than as a document, so that a feature's route is a value someone can read at three in the morning.

@dataclass(frozen=True)
class FeatureProfile:
    corpus_tokens: int
    window_tokens: int
    queries_per_session: float
    measured_hit_rate: float
    freshness_budget_min: float
    cache_ttl_min: float
    p95_budget_s: float
    requires_whole_artifact: bool

def route(p: FeatureProfile) -> str:
    if p.corpus_tokens > 0.6 * p.window_tokens:
        return "select_documents" if p.corpus_tokens <= 2.4 * p.window_tokens else "retrieve"
    if p.requires_whole_artifact:
        return "long_context_cached"
    if p.queries_per_session <= 2:
        return "retrieve"
    if p.freshness_budget_min < 2 * p.cache_ttl_min:
        return "retrieve"
    if p.queries_per_session >= 8 and p.measured_hit_rate >= 0.9:
        return "long_context_cached"
    if p.p95_budget_s < 4.0 and p.corpus_tokens > 100_000:
        return "select_documents"
    return "measure"

Count the fields on FeatureProfile and notice what is absent from them. There are eight inputs — two token counts, a query rate, a hit rate, two durations, a latency budget and a boolean about task shape — and not one of them is a price. That is the cancellation from the previous section showing up as a fact about the code: since p divides out of every route-versus-route comparison, a router that only ever compares routes has no use for it. Anyone expecting to configure this with a rate card was expecting to supply a term that cannot change the answer.

The diagram and the code agree with each other and disagree with the table, which is worth reading rather than smoothing over. Both the flowchart and the function evaluate the freshness condition before the amortisation condition; the table numbers them the other way round. The order matters materially: a feature running many queries per session over a corpus that changes faster than its TTL satisfies both conditions at once, and the two orderings then return opposite routes — cache it under the table's numbering, retrieve under the code's. The code and the diagram are right, for the principle the next section turns on. Staleness is a correctness condition, since serving an answer computed from a corpus that has since changed is serving a wrong answer, while amortisation is an economic optimisation, and an optimisation does not overrule a correctness condition. The table's row numbering is the error, left in place here because the fix is a change to the artefact rather than to the prose around it.

Note that measured_hit_rate is an input. A feature that has not measured it cannot pass rule 4, which is deliberate: the most common way this decision goes wrong is a team asserting a hit rate they have never read off the counters. It is the same failure as the blended figure above, in miniature — an assumed constant standing in for a quantity that varies.

Two features, one model, opposite answers#

Run the rule on the two features that started the argument in the retro.

Feature A fails at the first threshold and would fail the others anyway. Even in a hypothetical window that fit 12 million tokens, a single question per session amortises nothing: the session pays a 1.25× cache write and then ends, so caching makes it 25% more expensive than not caching. At $3 per million, one query would cost $36 uncached and $3.60 at a perfect cache-read rate that a one-question session can never achieve. Retrieval sends six thousand tokens, costs under two cents, and answers in three seconds.

Every figure in that paragraph is the general expression evaluated at one query. The uncached cost is twelve million tokens at $3 per million, which is $36 with nothing left over. The floor of $3.60 is that same $36 at the read multiplier of 0.10, which is what a session would pay if every token could be a cache read. The 25% is the ratio N / (w + (N − 1) · r) at N = 1, where the read term vanishes because there are no reads, leaving one over the write multiplier. And the retrieval cost is the stuffed figure scaled down by the ratio of the two token counts, since the price is common to both.

Feature B inverts every term. The 90,000-token prefix is written once at $0.34 and read twenty-nine times at $0.027, so the session costs about $1.12 against $8.10 uncached — and the prefix is stable by construction, because the diff does not change during the review. The task argument is stronger than the cost argument here. A reviewer's job includes noticing the test that was not updated and the migration that was not written, and a retriever ranking by similarity to "does this change break anything" will not surface an absence. Feature B would be the wrong shape for retrieval even if retrieval were free.

That breakdown closes exactly, and the closing is the check that the model has no missing term. One write at $0.34 plus twenty-nine reads at $0.027 each reconstructs the $1.12 session total with no residual, and the uncached $8.10 is thirty full-price sends of the same prefix. The ratio between them is the same expression evaluated at thirty queries, and it comes out a little over seven — so two features, sharing a model, a price and a cache, land on opposite routes from one formula, with the query count as the only input that differs in sign. The residual to name is what these figures exclude: they are input-side only, which is legitimate here for the reason chapter 1 gives — input tokens per session, not per request, decide the cost — and it does mean thirty answers are generated and billed as output tokens that appear in none of the numbers above.

When two rules fire, the order carries the decision#

The third feature in the retro is the one that shows why the rules are ordered rather than weighted, and what an ordering buys beyond a tie-break.

Every threshold here is an input that expires#

Every number in this topic is an input that expires. Window sizes grow, cache read multipliers change, prices fall, and a new model snapshot moves the accuracy comparison of chapter 6 — proving a long-context design is at least as good before you switch — by more than the design difference it was measuring. A decision recorded as a conclusion, "we use retrieval for the support assistant", is unfalsifiable and will be re-argued from scratch every six months. A decision recorded with its inputs is a calculation someone can re-run in an afternoon.

feature: support-assistant
route: retrieve
decided: 2026-05-04
inputs:
  model_snapshot: model-2026-04-17
  window_tokens: 400000
  corpus_tokens: 12100000
  queries_per_session: 1.1
  price_input_per_mtok: 3.00
  cache_read_multiplier: 0.10
  measured_hit_rate: n/a
  eval_set: support-eval-v2 (640 items)
  result: "retrieval 78.1% vs selection 77.4%, p95 2.7s vs 5.9s"
re_measure_when:
  - model_snapshot changes
  - window_tokens >= corpus_tokens * 1.7
  - price_input_per_mtok changes by more than 30%
  - measured_hit_rate on any cached feature drops below 0.90
  - p95 exceeds 4.0s for 7 consecutive days
  - 180 days elapse
owner: search-platform

One line in that list is not the arbitrary safety factor it looks like. The trigger window_tokens >= corpus_tokens * 1.7 is rule 1 turned inside out: rule 1 fires while the corpus exceeds 0.6 of the window, so it stops firing once the window reaches the corpus divided by 0.6, and 1.7 is that reciprocal rounded up to one decimal place. Rounding it up rather than down makes the trigger fire a little later than the rule strictly requires, which is the safe direction. Before assuming a figure in a config was picked by feel, check whether it is the reciprocal of a threshold stated somewhere else.

The trigger list is what stops the argument recurring, because it converts "should we revisit this" from an opinion into a condition. Wire the conditions that can be checked automatically — price, window size, hit rate, p95 — into a weekly job that opens a ticket against the owning team when one fires, with the decision record attached. The team then re-runs the chapter 6 comparison against the current snapshot and either updates the record or changes the route, and in both cases the artefact that comes out is a new set of inputs rather than a new opinion.

The list stops the argument recurring for the inputs it watches, and the algebra says which inputs those are. A model snapshot moves the quality side of the comparison and belongs there. The window-to-corpus relation is rule 1 and belongs there. The hit rate is the measured input rule 4 reads and belongs there. The price does not appear in any of rules 1 through 6 and cannot flip them, so it earns its place for a different reason: it decides the affordability of a feature outright, and it moves the money side of the rule 7 comparison while leaving the accuracy side fixed. Three inputs that can flip a route are not on the list at all. The cache read multiplier sets the break-even query count that rules 3 and 4 are built from, so a change in it changes where those thresholds should sit. The TTL does the same for rule 5. And queries per session is recorded once, at 1.1 for this feature, with nothing re-reading it if user behaviour drifts upward.

The record's own result deserves the same scepticism. It reports retrieval at 78.1% against selection at 77.4% on 640 items, and read as a level that looks like a win for retrieval. Read as a gap it is not a result at all. The uncertainty on a measured percentage is roughly √(p·(1−p)/n) where p is the score as a fraction and n the number of items, and with a score near four-fifths on a set of that size it comes out somewhere above a point and a half — larger than the gap between the two arms. The accuracy comparison did not separate them. What separated them was the p95, 2.7s against 5.9s, which is more than a doubling and nowhere near the noise. The route is right, and the reason recorded first is the weaker of the two reasons available.

One more pair of figures shares a number without measuring the same thing. Rule 6 tests whether a feature's p95 budget is under 4 s, while the trigger fires when the measured p95 exceeds 4.0s for a week. The contract assistant, at 5.1 seconds against a 6-second budget, is comfortably inside its budget and would trip that trigger on the first day the template was copied to it. Neither figure is wrong; they are thresholds on different quantities that happen to coincide, and this list is written against one feature's own measured latency rather than as a global rule.

Two habits keep the whole thing honest. Keep at least one feature on each route in production, so that neither capability rots and re-routing stays a configuration change; the retro's real cost was not the 7× but the three months it took to rebuild an index that had been deleted. And keep the evaluation sets alive as products in their own right, since every threshold in this chapter is only as good as the measurement behind it, and an evaluation set that has not been extended in a year is measuring last year's traffic.

From here the work is arithmetic on your own features. Take the two or three that send the most tokens, write down their corpus size, queries per session, freshness budget and p95 budget, run the routing rule, and where it returns measure, run the chapter 6 comparison on eight hundred questions with the grader blinded. The most likely outcome is that one feature is on the wrong route by a factor of five in spend, one is on the right route for a reason nobody had written down, and one turns out to need the middle design of chapter 5 — retrieve into the window, then cache what you retrieved — that nobody had considered because the argument had only ever had two sides.

What survives all of it is smaller than a policy and more durable than one. Cost is tokens times price times a multiplier, so the price cancels out of any comparison between two routes for one feature, and the decision is made in tokens, queries and seconds — quantities that are properties of a workload, and eleven workloads have eleven answers. Writing each answer down with the inputs that produced it, and with the conditions under which those inputs would have to be read again, turns a decision into something checkable rather than re-arguable.