Keep the business goal, narrow the scope until it is checkable

The VP of Product wrote it on a whiteboard in eleven words: an AI that reads every support ticket and tells us what to build. There are 41,000 tickets a quarter, nobody reads more than a sample of them, and the roadmap is currently assembled from whatever three loud customers said on the last quarterly business review call. The ask is not stupid. It is pointing at a real and expensive gap.

It is also unbuildable in the form written, and the reason is not model capability. Suppose the system returns "build bulk user provisioning, then SAML group sync, then a usage-export API". Nobody can say whether that is right. There is no reference roadmap, the decision it feeds is made by six people weighing revenue concentration, competitive pressure, technical debt and a board narrative, and by the time the quarter's outcome is known the counterfactual is gone. No owner of correctness exists either: if the recommendation is bad, the failure lands on the VP, who will simply stop using the tool rather than file a bug.

Separate the goal from the ask#

The move that saves proposals like this is to treat the sentence on the whiteboard as a statement of a business goal that has been prematurely compiled into a system design, and to decompile it. Three questions do the work. What decision changes if this exists? Who makes that decision, and on what cadence? What would they have to see for the decision to change?

For the ticket proposal the answers are specific. The decision is quarterly roadmap ranking. It is made by the VP with four staff engineers and the support director in a two-day session. What would change it is evidence that a category of pain is larger, growing, or concentrated in high-value accounts in a way nobody currently sees, because the current inputs are anecdotes and the loudest accounts are not the largest.

That reframing produces a different system. Cluster and label the tickets into a taxonomy that a named person maintains, publish volumes and trends by segment, and let the six humans read the ranking and argue about it as they always have. The model's output is now an input to a decision rather than the decision, which sounds like a smaller ambition and is actually the thing the sponsor asked for, since the two-day session was never going to be replaced by a text box.

Every property that was missing is now present. The taxonomy is a closed set of 34 categories, so an output is a label from an enumerable set rather than free prose. Correctness is checkable by sampling: 300 tickets a month labelled by two support leads independently gives both a model accuracy figure and an inter-annotator agreement figure, which is how often the two leads chose the same category as each other, and the second number bounds the first. Someone owns the taxonomy, which means someone owns being wrong about it. And the trend arithmetic is arithmetic.

{
  "ticket_id": "SUP-184203",
  "primary_category": "provisioning.bulk_user_import",
  "secondary_categories": ["auth.saml_group_mapping"],
  "confidence": 0.91,
  "evidence_spans": [
    {"start": 402, "end": 511, "quote": "we have to add all 900 users one at a time"}
  ],
  "abstained": false,
  "account_arr_band": "250k-1m",
  "taxonomy_version": "2026-Q2.3"
}

The abstained flag and the taxonomy_version field are what make this maintainable rather than a one-off. Tickets that match nothing at sufficient confidence abstain instead of landing in a junk category, and the abstention rate is the taxonomy's health metric: when it crosses 12% over four consecutive weeks, the taxonomy owner reads a sample of the abstentions and adds categories. That loop is the actual product. It is also the thing that would have been impossible in the original framing, because a system that outputs a roadmap has no equivalent of an abstention rate.

Hamel Husain's A Field Guide to Rapidly Improving AI Products argues the same case from the other end — that looking at your data in a structured way is what turns a vague product into a tractable one — and the taxonomy is what that looks like when the output has to be defended in a planning session.

Three proposals, ranked#

The same move applied to three asks that arrived together, phrased as agents. An agent that triages security alerts. An agent that writes the weekly executive summary from Jira. An agent that answers "is this customer allowed to do X" from our contracts.

Alert triage ranks first because it has the deepest verifier in the building. Every alert already carries an analyst disposition — true positive, benign, duplicate — recorded at close, going back four years and roughly 340,000 alerts. That is chapter 1's fourth row at a scale most teams never get. The error structure is sharply asymmetric: over-ranking a benign alert costs an analyst about four minutes, and suppressing a real one costs an incident, so the narrowing writes itself as a constraint on the output rather than a reduction in ambition. The system ranks and attaches context; it never closes anything. Three things would change my confidence. Whether the disposition field is filled honestly or by an end-of-shift bulk-close macro, which I would check by looking at the distribution of close timestamps — a spike in the last ten minutes of each shift means the labels are noise. What share of the 1,400 daily alerts are exact duplicates that a dedup rule removes for nothing, since chapter 4's arithmetic says the residual is what matters and the residual here may be 400 rather than 1,400. And whether alert payloads contain attacker-controlled text, because a hostname or user-agent field is an injection surface, meaning text the attacker wrote that arrives in the prompt and may be read as instruction rather than as data, and the adversary in this system is a real adversary rather than a careless user.

Contract entitlement ranks second and would rank first if the corpus turned out to be in better shape than I expect. It has real, blocking value — 60 questions a week, currently taking a median of two days, each one sitting in front of a deal — and it has a verifier, because the deal desk answers these in a ticket queue and the good answers cite a clause. The narrowing is to make citation and abstention mandatory: retrieve candidate clauses, draft an answer that quotes the governing language, and abstain when nothing is on point, with the deal desk approving before anything reaches the customer. That approval step is what bounds the error cost, and it is affordable at 60 items a week in a way it would not be at 6,000. What I would need to know is whether the past answers actually cite clauses or are free-text replies (which would mean building the labelled set by hand from perhaps 200 tickets), how concentrated the questions are (if 70% hit a dozen clause types, this is tractable; if they are a long tail of bespoke amendments, it is not), and whether the contract corpus is complete and current, since executed PDFs with linked amendments are a different corpus from a folder of Word drafts.

The instinct that puts contracts last because legal will slow it down inverts the situation. Legal's involvement is the asset, because they are the people who already own being right about entitlement, and a proposal whose correctness has an owner with authority is in a far stronger position than one whose correctness belongs to nobody. What legal will slow down is a design where the answer goes to the customer unreviewed, and that design should be slowed down.

The Jira summary ranks last, which is where the shallow ranking puts it in reverse. Being easy and low-risk is not a reason to build something; it is a description of what happens if nobody uses it. The summary has no verifier — no reader will ever say a summary was wrong, they will simply stop opening the email — no consumer decision attached, and a value that is difficult to state as anything other than convenience. Its version next door is worth naming, because the useful half is not a model problem at all: what slipped this week, by epic, with owner and days late, is a query against the Jira API and a template. Only the "why" half involves language, and it becomes bounded once it is scoped to the nine issues that actually slipped, summarising their comment threads with comment IDs attached so a reader can check. That is a two-day build with a deterministic core, and if the executives read it for six weeks, there is evidence for something larger.

Where narrowing goes wrong#

The failure mode is descoping until nothing anyone wanted remains, and the test that catches it is whether the sponsor's decision actually changes. A ticket taxonomy that ranks 34 categories by volume changes the roadmap session, because the session currently runs on three anecdotes. A ticket taxonomy that is never published, or that lands in a dashboard nobody opens, does not, and it will be indistinguishable from the wide version six months later when both are unused. Take the narrowed proposal back to the person who wrote the original sentence and ask them directly whether they would run the next planning session differently with it; the answer is usually immediate and usually honest.

Narrowing is also not the same as adding human review, which gets offered as a universal solvent and is a real control only where the arithmetic supports it. A deal desk approving 60 entitlement answers a week is inspecting; an analyst nodding at 1,400 ranked alerts a day is not, and calling both "human in the loop" hides the difference. What makes the alert design safe is the constraint that nothing gets auto-closed, which holds whether or not anyone is paying attention, and constraints that hold under inattention are worth more than review steps that do not.

Finally, the narrowed version is not a ceiling. It is the version that produces the evidence the wide version needs — labelled data, a measured accuracy, an observed usage rate, a demonstrated error cost — so the argument for expanding scope in two quarters can be made from numbers rather than from the same enthusiasm that produced the whiteboard sentence.

That evidence takes time to accumulate, and sponsors rarely offer a quarter to decide. The usual budget is a few days, and what gets built in those days determines whether the answer that comes back is worth anything.