Setting expectations on non-determinism

Read these first

• Designing around an imperfect component

What you will know

• Run a calibration exercise that turns a 90% success rate into expected counts for a stated workload

  At 100 requests, roughly 10 failures are normal, and small samples can make the observed rate fluctuate substantially.

• Refuse a demo being treated as a requirement

  A scripted happy path proves possibility, not coverage, repeatability, or behavior on adversarial inputs.

• Set expectations before a launch rather than after an incident

  Publish the target population, measurement window, failure handling, and escalation owner before users form an absolute expectation.

• Communicate a quality regression without losing trust

  Show the same metric and slice before and after, state uncertainty, and pair the disclosure with a rollback or mitigation plan.

• Reject a request that cannot be met reliably

  A clear boundary prevents users from silently depending on tail behavior while leaving room for a measurable alternative.

Chapters

1. 1

   Turning 90% into failures somebody has to handle on Tuesday

   A rate means nothing to an operations manager until it becomes a count against their workload: 90% over 100 requests a day is about ten failures, and day-to-day it will swing by several either side, so 16 is ordinary rather than alarming. Do that arithmetic with them, in their units, before launch. Then say what you need back, which is a correction path with capacity and someone who reports what went wrong.

   Read chapter
2. 2

   Nineteen successes out of twenty tells you almost nothing

   A demo that works 19 times in 20 is consistent with a true rate anywhere from roughly 75% to 99%, which is the difference between a shippable feature and an unusable one. It also proves possibility rather than coverage, repeatability, or behaviour on inputs nobody scripted. The skill is refusing the demo as a requirement without refusing the demo, by naming what a scripted happy path can and cannot establish.

   Read chapter
3. 3

   A launch criterion without a population is not measurable

   'The assistant answers policy questions correctly' cannot be passed or failed: it names no population of questions, no rate, no measurement method and no window. Rewriting it is the easy half; getting the stakeholder to accept the rewrite means showing that the vague version leaves them with no way to tell success from failure either. Publish the failure path and the escalation owner alongside it, before launch rather than after the first incident.

   Read chapter
4. 4

   Disclosing a regression while it is still yours to disclose

   A four-point drop found before any customer noticed is the best position you will ever be in, and quietly fixing it first converts that into a credibility problem if anyone learns the sequence. Report the same metric on the same slice before and after, state the uncertainty honestly, and arrive with a rollback or mitigation already attached so the conversation is about a decision rather than about alarm.

   Read chapter
5. 5

   Saying no by naming the adjacent thing you can measure

   Some requests cannot be met at a rate anyone would accept, and the damaging outcome is not refusal but a quiet half-delivery that people come to depend on in the tail. State the boundary plainly, then offer the version you can measure and stand behind, which is usually narrower in scope or paired with a human step. A clear no leaves the relationship intact in a way an unacknowledged 60% does not.

   Read chapter