Under pressure, drop the guardrail another one already covers

The feature adjusts list price on long-tail SKUs where the merchandising team has never set one manually: it reads the product description, competitor prices scraped overnight, and 90 days of conversion data, and proposes a price. It touches about 4% of orders. The marketplace does 320,000 orders a month, so roughly 12,800 orders a month carry an AI-set price, and at the measured 98% accuracy about 256 of those are wrong — between eight and nine a day, every day, at an average order value of £68.

What "wrong" means splits into two costs that behave differently. A price set too low sells the item; the revenue is gone and there is no mechanism to recover it, because you cannot go back to a customer who bought a £70 item for £46 and ask for the difference. A price set too high is worse in a different way: the customer either does not buy, which is an invisible loss you will never attribute to this feature, or buys and later discovers the same item listed at a different price, which produces a refund, a support contact, and — if the pattern is systematic rather than scattered — a consumer-protection question that arrives with a regulator's letterhead. Applying the error classes from chapter one gives four lines, and only the first has a percentage in it.

feature: longtail_price_adjustment
monthly_ai_priced_orders: 12800
budget:
  harmless:
    definition: proposed price rejected pre-commit by a rule or a reviewer
    threshold: <= 8% of proposals
    rationale: pure compute and reviewer cost; nothing reached a customer
  costly_reversible:
    definition: wrong price live but no order placed against it
    threshold: <= 150 SKU-days/month
    rationale: detectable by reconciliation within 24h; correcting it costs nothing
  irreversible:
    definition: an order completed at a price the model set wrongly
    threshold: <= 220/month, and <= £9,000/month of margin at risk
    rationale: 220 x £41 mean margin delta; sits inside the promotions budget
  regulated:
    definition: the same SKU sold at materially different prices to comparable
      customers in the same window, or a price below the advertised floor
    threshold: 0 — enforced by a deterministic floor rule, not by the model
    rationale: one instance is a pricing-integrity finding, not a rate

Five controls, and what each of them actually catches#

The obvious control set for this is a confidence threshold on the model's own score, a review queue for anything below it, a commit hold before the price goes live, a hard cap on how far a price may move in one step, and a nightly reconciliation against the previous price. They look like five layers of defence. They are five instruments with overlapping coverage, and the way to see the overlap is to write down the failure modes first and then mark which control catches each one.

The pattern in the first two columns is the finding. The confidence threshold and the review queue are not independent, because the queue is fed by the threshold — every item a reviewer sees is an item the score already flagged, so any failure that the score does not notice is invisible to both. That is a common-cause failure, in the sense safety engineering uses the term: two redundant components sharing a single point of failure, like duplicate sensors wired to the same power bus. The decimal error and the injected description are both cases where the model is wrong and confident, and both columns go dark together.

Meanwhile the cap and the reconciliation catch things without consulting the model's opinion of itself at all. The cap is a rule — no price may move more than 8% from the last human-approved price in one step, and none may fall below cost plus 2% — and it catches the decimal error, the injection and the bad deploy for the price of about fifteen lines of code and no human time. The reconciliation is the only control in the table that sees the shape of the whole day's changes rather than one item at a time, which is why it is the only one that catches drift, segment-wide errors and stale inputs.

The routing decision in that diagram is deliberately not the model's confidence. Items go to a human when the impact is large — a price delta above 3%, or an estimated monthly margin exposure above £500 — because impact is computed from numbers the system knows precisely, whereas confidence is an estimate of something the system is bad at estimating. Routing by impact also makes the queue size predictable: about 6% of proposals exceed those bounds, which is 770 items a month, roughly 35 a working day at three minutes each, which is under two hours of one person's time. That number matters because of the capacity arithmetic from chapter six; a queue that is affordable stays a real control, and one that is not becomes a rubber stamp within a month.

What each one costs to keep#

Coverage is half the decision; the other half is what each control costs to build and to run, because the schedule conversation is about engineering days and the operations conversation is about everything after.

The cap is about fifteen lines of code plus access to a cost feed, and its running cost is close to zero. The reconciliation is two to three days to build, and its real cost is ongoing: somebody owns the alert, and an alert that pages at 03:00 for a 40% category move needs a documented rollback procedure or it becomes an alert people acknowledge and ignore. The impact-routed review queue costs the two hours a day computed above, which at a merchandiser's loaded rate is roughly £3,000 a month, and it is the only control with a cost that scales with volume. The commit hold costs almost nothing here because prices are set by an overnight batch rather than at request time — a 30-minute hold on a nightly job is invisible, whereas the same hold in a checkout-time pricing path would be impossible, and that distinction is why the hold is affordable on this feature and not on a superficially similar one. The confidence threshold is the cheapest to build and the most expensive to reason about, since every later argument about the feature ends up relitigating whether the score means anything.

One failure mode deserves separate attention because it is invisible in the table and arrives on somebody else's schedule: the model underneath you changes. A provider deprecates the version you measured, or silently updates a pointer alias, and the 98% you validated in March is a property of a system that no longer exists. Pinning to a dated model version removes the silent case and converts it into a deadline, at which point you re-measure before migrating. What catches the case you did not anticipate is the reconciliation, because a version change that shifts pricing behaviour shows up as an unusual distribution of price moves the following night — which is the same detection path as a bad prompt deploy, and is another reason that particular control is doing more work than its build cost suggests.

Running a pre-mortem is the cheapest way to find the gaps before the launch rather than after. Assume it is three months from now and the feature has caused a serious incident, then write down the three most plausible stories. On this feature they were: a seller discovered the description trick and worked it across 400 listings; a prompt change shifted the whole Home & Garden category down 20% for a weekend; and a stale competitor scrape held prices at a bank-holiday promotion for four days. Reading those against the matrix shows the first is caught only by the cap, the second and third only by the reconciliation, and none of the three by the two model-confidence-driven controls that most of the design discussion had been about.

Which one goes#

When the date moves, the control to drop is the model-confidence threshold as a routing signal, because everything it catches is caught by the cap on the large end and by the impact routing on the consequential end, and it fails on exactly the cases — confident and wrong — that the other controls exist for. Removing it also removes the temptation to treat the score as meaningful in later decisions, which is a real cost of leaving a half-trusted number in the pipeline.

The cap is the last thing to go, and it is worth being explicit about why, since it is also the least sophisticated. It is deterministic, so its behaviour under an adversarial input is the same as its behaviour under a normal one, and chapter one's point about abusive cases applies directly: a seller who works out that a particular description makes the model price aggressively controls the rate at which that happens, and no measured accuracy constrains them. The cap is also the cheapest control to keep, so trading it away buys almost no schedule.

The reconciliation is the one teams actually drop, because it is a nightly job with an alerting story and a rollback path, and it feels like something that can be added in the next sprint. Dropping it removes the only detection for four of the seven failure modes, which means the feature would ship with no way to notice that an entire category has been mispriced for a week. If the schedule genuinely cannot accommodate it, the honest move is to reduce the scope of the feature to match the controls that exist — ship to one category instead of all of them, or cap the feature at 0.4% of orders rather than 4% — so that the missing detection is bounded by the exposure rather than by luck. Scope reduction is the underused alternative to control reduction, and it is available in almost every one of these conversations.

A dropped control should leave a written trace with three things in it: what was dropped, what compensates for it in the meantime, and the condition under which it comes back. "Reconciliation deferred; feature limited to Home & Garden and to 1,500 SKUs; ships before we expand to a second category" is a decision someone can hold you to. "We'll add it later" is not, and it is the form these decisions almost always take.

The part that stays unresolved#

The uncomfortable property of a layered control set is that you cannot tell which layer is load-bearing until something fires, and by then you have already been paying for all of them for months. A control that has never triggered is either unnecessary or broken, and the logs look identical in both cases. The countermeasure is the seeding technique from chapter six, applied to the machinery rather than to the reviewers: inject a proposal with a decimal error into the pipeline once a week and assert that the cap rejected it, push a synthetic 40% category-wide drift into the reconciliation input and assert that it alerted. Each of those is an integration test that runs in production, and the catch counts they produce are what turns "we have five guardrails" into a statement with evidence behind it.

Everything in this topic collapses into a short set of questions to ask about the next feature that puts a model in front of a customer. How many wrong outputs a month does the volume imply, and what does one cost. Which slice absorbs them. Where does a single wrong output land, and how many people does that reach. Is there a window, a cancel path and a prior version. Is the effect safe to repeat. Who reviews, how many items an hour, and how do you know they are still looking. What does the user see when the system is unsure, and can they act on it. Run those against a feature you are working on now, in that order, and the places where you cannot answer with a number are the places where the design is not finished.