The step that writes tool arguments is the one not to downgrade

Model tiering is assigning each step of a pipeline to the cheapest model that can carry that step, instead of running the whole agent on one model. It is the largest saving still available once the caching work of the previous chapter has been taken, and unlike caching it can change what the agent decides, which is what makes the interesting question which steps rather than how much.

The rule most teams start with is to keep the strong model for the customer-facing answer and move everything upstream to the cheap tier. That rule gets the assignment backwards on both ends, and the reason it survives is that its failures are invisible in the artefact anybody reads. A drafted reply written by a weaker model looks like a drafted reply. A tool call written by a weaker model also looks like a tool call — an object with the right fields, in the right shape, nearly every time. What changes is how often those fields are exactly right, and nothing about reading the output tells you that.

Prose degrades on a slope, structure degrades off a cliff#

The difference is in the shape of the acceptance test, and it is worth stating in those terms because it predicts which steps are safe without any measurement.

A summary is judged on a continuum. A weaker model produces one that is a little flatter and occasionally drops a qualifier, so the whole quality distribution shifts downward by some amount, and what matters is how much of the mass is still above the acceptance bar. Shift a distribution slightly and most of it stays where it was, which is why a weaker summariser remains usable roughly as often as a strong one.

A tool call is judged by a predicate. The schema and the downstream system accept the object or they do not, and there is no slightly-correct record id. An enum value is a member or it is a rejection, and a nested object with the amount at the top level instead of inside adjustment is a 400 regardless of how sensible the number was. A predicate has no continuum to shift along, so a model that is a little less reliable does not produce slightly worse calls; it produces the same calls, correct less often.

That difference is what makes the failure expensive out of proportion to its rate. A rejected call comes back into the loop as an error the model has to correct, which topic 10.6 handles and which costs a full model round trip over a context that is now larger. On the frontier tier one such step was rejected on 1.4% of runs; on the cheap tier, 11.9%. Each rejection added 8.8 seconds and roughly $0.03, and about a fifth of them failed to correct within the allowed attempts and became escalations.

The accepted-but-wrong cases are worse still, because nothing in the system reports them: the tool returned 200, the run completed, the evaluator saw a resolved ticket, and the only signal was a reconciliation report two weeks later. Rejections are self-announcing and therefore boundable; silent wrong actions are neither, and any comparison run over a window shorter than the reconciliation cycle will score them as successes.

An inventory before an assignment#

Assigning models step by step requires knowing what the steps are, which sounds obvious and is skipped constantly because the loop is written as one function. The useful columns are what the step reads, what it produces, and what a wrong output does — that last one is the column that decides the tier.

Read the fourth column and the tier column falls out of it, which is the point of arranging the table this way. The three steps that stay on the frontier tier are the three whose failure column contains either a wrong action or a silent one. The three that move are the ones whose failures are visible, recoverable, or merely mediocre.

Steps 3 and 6 are the honest downgrades: the input is long, the output is prose, and a mediocre result is a mediocre result rather than a wrong action. Step 1 is a nine-way classification with clean labels, which small models do well and which the evaluator catches when they do not. The three that stay are on the list for different reasons, and only one of them is obvious.

Step 4 is obviously judgement — it decides whether a customer gets money — and nobody proposes downgrading it. Step 2 is the one that gets misread. It looks like extraction: pull the shipment id the customer is talking about. What it actually requires is holding six shipments across three orders in mind, matching a customer's approximate description ("the box with the two lamps, ordered before Christmas") against them, and noticing when none of them fits well enough to proceed. That is long-context reasoning wearing the costume of a lookup, and the ablation below shows it is the second most expensive step to get wrong. Step 5 is the one this chapter is named for, and its mechanical appearance is the most convincing of the three.

The rule of thumb that puts the small model everywhere except the final answer gets this backwards on both ends. The final answer in a workflow agent is frequently the most mechanical step in the pipeline, because by the time it runs the verdict, the amount and the customer's name are all sitting in the context and the job is to render them into a house-style paragraph — which is close to templating with better grammar. Meanwhile the steps it protects with the strong model, the ones early in the run that gather and interpret, are where the run's outcome is actually determined. The rule is popular because the final answer is the only step a person reads, so it is the only step whose quality anyone can feel.

The price gap that motivates all of this is worth stating in absolute terms, since it determines how much any of these rows can possibly be worth. The frontier tier here bills $3 per million input tokens and $15 per million output; the small tier bills $0.80 and $4. A step reading 11,000 tokens of policy and writing 250 costs $0.0371 on the frontier tier and $0.0098 on the small one, so the ceiling on downgrading it is 2.7 cents per run. A step reading 14,000 tokens and writing a 90-token tool call costs $0.0434 against $0.0116, a similar ceiling.

Notice what that arithmetic is dominated by. Both steps write almost nothing and read a great deal, so the input price is nearly the whole cost, and the input price differs between the tiers by a factor of a few. Two steps with very different jobs therefore have almost the same ceiling, because the ceiling is set by how much they read rather than by what they do — which is the arithmetic that made step 5 look attractive, and it was correct about the saving and silent about everything else.

Measuring a downgrade on the thing you sell#

Eyeballing output cannot detect any of this. A reviewer comparing two drafted replies sees prose quality; the failure lives in an enum three steps earlier. What works is a per-step ablation, meaning a comparison in which one component is changed and everything else is held still, scored on the fixed success criterion from the first chapter: hold the whole pipeline constant, move exactly one step to the cheap tier, and run the same 2,000 replayed tickets. The caching change from the previous chapter had already brought cost per successful task to $0.152, so everything here is measured against that.

Step 5 is the row worth staring at. Moving it to the cheap tier made the agent more expensive per successful task, from $0.152 to $0.158, while dropping success by nearly nine points.

Both halves of that come from the same place. The token price of that step fell by about 73%, which is a real reduction in the numerator, and it is small in absolute terms because the step is a few cents. Meanwhile success falling by nearly nine points removes about a tenth of the denominator, and a tenth off the denominator raises the ratio by more than a few cents off one step lowers it. That is the general shape of a bad downgrade: the saving is real and local, the cost is diffuse and larger. The rejections it caused added correction round trips on the frontier model over a full-length context, and the runs that never recovered moved into the escalation queue where they cost $6.40 each.

The combined row is not the sum of the individual rows in success terms, and you should not assume it will be. Set the two columns side by side and they behave differently. The cost column composes exactly: the three savings taken alone are eleven, nineteen and seventeen tenths of a cent against the baseline, and the combined configuration lands at $0.105, which is the baseline less all three. The success column does not. Downgrading three steps at once can interact — a flatter policy summary from step 3 makes step 4's job harder — and here the interaction ran the other way, costing 0.3 points against the 0.9 that summing the singles would predict. Check that prediction against the table rather than taking it on trust: the three single-step arms give up nothing, two tenths and four tenths of a point, which adds to six tenths, so six tenths is the number the rows support. Either figure leaves the conclusion standing, since the combined arm is cheaper in success terms than its parts predict, and neither is the sum.

That the interaction was mild is a fact about this pipeline and licenses nothing about another. On agents where steps feed each other more tightly it has been worse, which is the argument for running the combined configuration as its own arm rather than composing deltas on a spreadsheet.

Wiring it so the assignment is visible#

The mechanism is unremarkable and its value is that the assignment stops being implicit. A step declares its tier and its output contract in the same place, so anyone changing one has to look at the other.

@dataclass(frozen=True)
class StepSpec:
    name: str
    model: str
    reason: str                    # why this tier, in one line
    validator: Callable[[dict], None] | None = None

PIPELINE = {
    "classify":     StepSpec("classify", SMALL,
                             "9 clean labels, evaluator catches errors"),
    "disambiguate": StepSpec("disambiguate", FRONTIER,
                             "long-context matching, failures are silent"),
    "summarise":    StepSpec("summarise", SMALL,
                             "prose out, degrades gracefully"),
    "eligibility":  StepSpec("eligibility", FRONTIER,
                             "decides whether money moves"),
    "credit_call":  StepSpec("credit_call", FRONTIER,
                             "exact ids and enums; rejection or wrong credit",
                             validator=validate_credit_args),
    "reply":        StepSpec("reply", SMALL,
                             "templated 120-word message"),
}

def validate_credit_args(args: dict) -> None:
    if args["reason_code"] not in REASON_CODES:
        raise ArgumentRejected(f"reason_code must be one of {sorted(REASON_CODES)}")
    if not ORDER_ID_RE.fullmatch(args["order_id"]):
        raise ArgumentRejected("order_id must match SO-###### (six digits, zero-padded)")
    if not orders.exists(args["order_id"]):
        raise ArgumentRejected(f"no order {args['order_id']}; re-read the context")

The reason field is the part that pays for itself. A tier assignment with no recorded justification is indistinguishable from an accident, so the next person to look for savings sees six models and no argument, and the cheapest thing they can do is try moving one.

The three checks in the validator are a ladder from cheap to expensive, and they catch different failures. The membership test rejects an invented enum using only local knowledge. The pattern test rejects an id of the wrong shape, which is what catches a five-digit number where the records are zero-padded to six. The existence check on the last line is what would have caught the wrong credits below, since a well-formed id for a real order that is not this customer's order passes every schema check there is, so the validator has to reach into the domain and compare against what the run established. Validators of that kind are worth writing before you tier anything, because they convert the silent failure class into the noisy one, and a noisy failure is a failure your ablation can measure.

What tiering is not, and where it stops paying#

It is not routing by task difficulty. Sending easy tickets to a small model end to end and hard ones to a frontier model is a different technique with a different risk, which is that the router has to judge difficulty before the work is done and is wrong in the direction that matters. Step tiering makes the same decision after the context exists, per step, and both can be used at once; conflating them tends to produce a system where nobody can say which model handled a given failure.

It is also not a substitute for a smaller prompt or a better tool schema. A cheap model that needs three attempts to produce a valid call costs more than one attempt on an expensive model, and the reason it needs three is usually an enum documented in prose rather than in the schema, or a field named id that could mean four different records. Fixing that raises the ceiling for every tier at once and often makes a downgrade that failed the ablation pass it on the second try — which is a reason to attempt the schema work before the ablation rather than after, since otherwise you have measured a model against an obstacle you were about to remove.

Its ceiling is set by the breakdown in chapter two. Tiering moves generation time and token price, which together were 24.6 seconds and most of the bill on this agent, so it pays here. On an agent dominated by tool waits it can cut the model bill by 60% and move the customer's wait by four seconds, and the honest report of that result says so — a 60% cut in a bill is a real result and a four-second improvement is a real result, and presenting the first as though it implied the second is how a latency complaint survives an optimisation that was never aimed at it.

All of this lowers the price of a step, and the ablation above assumed the number of steps was fixed while it did so. It is not fixed. The number of steps is whatever the loop decides it is, and the loop decides by asking the model whether it is done — which it answers, every time, with a fluent statement that it is.