The first wrong number in a leadership channel ends the product

The ask came out of a Monday leadership meeting, phrased the way these always are: why can't I just ask Slack what revenue was last week? The company has a BI tool with about 40 certified dashboards, and a two-person analytics team who field roughly fifteen ad-hoc requests a week. The executive is not wrong that something is slow. The question is whether the thing that is slow is SQL generation.

That question has an answer available before any code is written, and getting it is a morning's work. What makes a request slow is not one thing, and the kinds of slowness in play have completely different remedies — one is a search problem, one is an agreement problem, and only the third is a query-writing problem. Separating them means classifying real requests rather than reasoning about them.

What fifteen requests a week are actually made of#

Sixty of those ad-hoc requests, sampled from the last month of the analytics channel and classified by hand, came out like this: 34 were answerable from a dashboard that already exists and the asker did not know about or could not find, 18 were definition negotiations — "does this include partner revenue?", "why doesn't this match the invoice report?" — and 8 were genuinely new analysis requiring a query nobody had written. A text-to-SQL bot addresses the third bucket, which is 13% of the volume and the bucket that most needs a human's judgement. The first bucket is a search problem over dashboard titles and descriptions, which is two weeks of work with none of the failure modes in this topic. The second bucket is chapter 7's problem and is solved by a registry, with or without a bot.

Two things about that count are worth checking before leaning on it. The three buckets add up to the sixty exactly, with nothing left over, so the classification was exhaustive and its categories did not overlap. That matters because it means there is no residual bucket quietly holding volume, and the 13% is a share of the whole sample rather than of some part of it that happened to be classified.

The second check is that the sample size and the request rate were arrived at separately and agree. Sixty is a count of messages in a channel over a month. Roughly fifteen a week is an estimate of how much work the two analysts absorb, and four weeks at that rate comes to sixty. Two statements with no step in common landing on the same number is what licenses reading the sixty as close to a census of that month rather than as a subsample of it, which in turn is what allows the bucket shares to be read as shares of the real workload.

The residual under both of those checks is the same one and it is real. Everything above concerns requests that reached that channel. A question asked at a desk, in a direct message, or in the meeting where it came up is outside both instruments, and those are systematically the questions a Slack bot would also never see. The agreement between the two figures says nothing about how large that population is.

Read the buckets as gaps rather than as three items on a list and the proposal changes shape. The largest bucket is the one where the answer already exists and nobody could find it, so most of the perceived slowness is a discovery failure, and generating SQL does not touch it — a bot that writes a fresh query for a question a dashboard already answers has added a second number where there was one. The next largest is a disagreement about meaning, which generating SQL makes worse by contributing a fourth derivation to an argument that already had three. The bucket the bot genuinely addresses is the smallest of the three, and it is also the one where a wrong answer costs most, because a genuinely new analysis is by definition one nobody can check against something they already have.

That sample is the argument you bring to the meeting, and it is more persuasive than any position on model reliability.

Conditions that make this a project to refuse#

Several conditions make this a project to refuse rather than to scope down, and they are all about what exists around the warehouse rather than about the technology.

There is no owner for metric definitions. If nobody has the authority to say what revenue means when finance and sales disagree, the bot inherits the disagreement and gets blamed for it. The blame lands there for a structural reason rather than an unfair one: of the three parties producing numbers, the bot is the only one that can be switched off, so it is the cheapest place to resolve an argument that is really about two teams. Building the registry is a prerequisite that requires a decision-maker, and if that person cannot be named, the answer is no.

There is no path to an evaluation set. Three hundred questions with gold SQL costs an analyst something like eight days, spread over a month, and without it there is no schema-retrieval recall, no measured answer accuracy, and no way to tell whether a model upgrade helped. A team that will not fund that is asking for an unmeasurable system. The eval set is not a quality gate that a project passes once — it is the instrument every later decision is read off, and a project without an instrument does not get worse slowly, it gets opinion-driven immediately.

The warehouse has no curated layer. If the only objects are raw event tables and staging schemas, then the views, grains and column descriptions that everything in this topic depends on do not exist, and the honest estimate for the project starts with a quarter of data modelling.

The figures in scope are externally reported. Revenue that flows to the board or into a filing has an audit requirement that a generated query does not meet, and the right scope excludes those metrics regardless of accuracy.

That last condition is worth separating from the others, because it is the one where the usual engineering move does not apply. An audit requirement arrives from outside and is not a target that a higher accuracy figure eventually reaches; no measured percentage satisfies a control that asks for a reproducible, reviewed, human-signed derivation. Excluding those metrics from scope is the design decision that answers it. A proposal that treats the audit condition as a bar to clear has confused a requirement with a target and will spend a quarter finding out.

A useful test cuts through most of the discussion: would the sponsor still want it if every answer had to be reviewed by an analyst before it was sent? If the answer is that this would make it pointless, the value being asked for is speed without accountability, and it is worth surfacing that explicitly before anyone writes code.

The smallest version worth showing an executive#

Assume the conditions are met. Version one is narrower than the sponsor expects, and the narrowness is what makes it survivable.

Nine certified metrics, twelve mart views, no free-form SQL over raw tables in version one. The refusals are load-bearing: each one is a class of question where the system would produce a plausible answer with no basis, and a product that declines clearly is trusted more than one that always responds.

The line that table draws is not the line most people expect, and it is worth tracing. "Revenue by plan, Q2" is answered and "Revenue this month" is refused, even though the second is the easier query of the two by any measure — fewer joins, a simpler window, less to get wrong. What separates them is whether a defensible statement about the number exists. A completed fiscal quarter has a meaning that can be written down. A partial month's total has no meaning until somebody says how much of the month is in it, and a figure with no statable meaning is exactly the thing chapter 7 spent its length on. The same test explains the churn row: the series is a warehouse question and the cause of its movement is not, so one half is returned and the other is declined. The refusals track what can be said about a number, not what can be computed.

One row on that table is different in kind from the rest, and mistaking it for policy is a real error. The emails refusal holds at the grant level, because the view the role can reach has no email column on it. Every other refusal is a decision the system makes and could therefore fail to make — a routing bug, a prompt change, a model that reasons its way past an instruction. The grant refusal survives all of those, because there is nothing to select. When choosing where to put a defence, that is the distinction that matters: one class of protection depends on the system behaving, and the other does not.

Every answer carries its provenance in the message body rather than behind a link, because nobody clicks the link.

Active users, July 2025 (DE, AT, CH)     134,802

  metric      active_users (certified · owner @growth-analytics)
  definition  distinct users with create/share/export, account-local
              calendar month, excludes internal + trial
  data        fct_events, complete through 2025-08-06 14:00 UTC
  checks      grain ok · literals in domain · 3,412 rows · not truncated
  query       SELECT COUNT(DISTINCT e.user_id) AS active_users
              FROM analytics_marts.fct_events e ... (shown in full)
  answer_id   ans_01J9F2K4  ·  this is a warehouse figure, not a
              certified financial statement

The visible SQL matters more than it appears to. It converts every reader into a possible reviewer, and about one in twenty answers gets a "hang on, that date column is wrong" from someone who understands the business, which is a review channel that costs nothing to operate. It also sets the register of the whole product: this is a tool that shows its working, so a figure from it is checkable.

That review channel comes with a condition attached, and the condition is the one that goes on to decide the whole rollout. It works while the readers can read SQL and know the business well enough to spot a wrong date column, which describes an analytics channel almost exactly. It thins as the audience widens, and it is close to absent in a room of executives, none of whom will read the statement. So the free reviewer supply is largest where a mistake is cheapest and smallest where a mistake is most expensive, which is the opposite of the arrangement anyone would design.

Two operational choices belong in version one as well. Answers are posted in public channel threads and never in DMs, because a wrong number in a DM cannot be found later or corrected. And the bot is rate-limited per user, which matters less for cost than for the pattern where someone re-asks a question eight ways until they get the number they expected.

Launch where the reviewers already are#

The rollout that protects the product is to put it in the analytics channel first, with the two analysts as the audience, for four weeks. They ask it the questions they are already being asked, they see the SQL, and every answer they disagree with is a labelled failure with an expert explanation attached — which is the same evaluation data the eval set is made of, arriving for free from people who would have done the work anyway.

The number to watch during that pilot is the rate of corrections per hundred answers, and the target for moving to a leadership channel is that it stops falling. A rate that drops from 22 to 9 to 7 to 7 over four weeks means the remaining failures are the hard kind and further pilot time will not find them; a rate still falling means the cheap fixes are not exhausted. Expanding to a wider audience while the curve is still moving means the wider audience meets the errors the analysts would have caught next week.

The levels in that series carry almost nothing and the gaps between them carry everything, which is why the decision rule is written the way it is. A large fall, then a small one, then none: the first gap is a week of looking finding whole classes of failure that a day of work removes, and the last gap being zero is the claim that the fourth week turned up no class the third week had not already seen. Seven per hundred is not a target anybody chose and it is not being called acceptable. The decision rests on the curve having stopped moving, and a curve that had flattened at a different level would license the same move.

What a flat curve licenses is narrower than it first reads, and the narrowing is the point of running the pilot where the reviewers are. The curve flattens when this audience stops finding new classes of failure, and two analysts asking the questions they are already asked will exhaust their own distribution of questions well before they exhaust the space of questions. A leadership channel asks a different distribution. So a flat curve supports "the cheap fixes for this audience are exhausted", which is what the decision needs; it does not establish that every remaining failure anywhere is a hard one.

What the pilot also produces is the sponsor's expectations. An executive who has watched the analytics channel for a month has seen the bot decline questions and has seen an analyst correct it twice, so the first wrong answer in their own channel is a familiar event rather than a revelation.

Running cost against the cost of owning it#

Running cost is small and is not where the expense is. At roughly 4,900 input tokens and 400 output tokens per question, plus a repair attempt on 14% of them, the model cost is about $0.021 an answer; 400 answers a month is under $10, and the warehouse compute is a few dollars more. The real cost is one engineer's ongoing attention — the eval set re-run on model changes, the corpus of wrong queries, the verifier's false rejections triaged weekly — plus about half a day a week of analyst time approving metric definitions and table descriptions. A proposal that presents this as an infrastructure cost has mis-stated the project.

Those two money figures check against each other, and it is worth doing the multiplication rather than accepting both. Four hundred answers at that per-answer cost lands under the ten-dollar line quoted, so the monthly figure and the per-answer figure are consistent. What that closure licenses is only the arithmetic between them: the per-answer figure itself rests on a price per token that is not stated here, so it cannot be re-derived from the token counts, and a reader who wants to check it against their own contract has to supply that price.

The per-answer figure is also a mean with the repair attempts already inside it, which is the right way to build it and the wrong way to read it as a bound. Roughly one question in seven gets a second pass, and that surcharge is spread across every answer in the average. A week of unusually long questions does not violate the figure; it just is not described by it.

The reason any of that matters is what it removes from the argument. If inference were the dominant cost, the project would be scoped by a budget line somebody could approve. It is not, so the project is scoped by an engineer's continuing attention and half a day a week of an analyst's, and neither of those appears on an infrastructure forecast. A sponsor who approves the cheap number and does not commit the expensive one has approved nothing.

Making a wrong answer recoverable#

It will happen. The eval set says accuracy on the certified path is 94% with abstention on 8%, so roughly one answer in seventeen is wrong. Those two percentages do not sit on the same base, and reading them as shares of one total gives more than a hundred per cent, which is the tell. Accuracy is measured over the answers the system gives. Abstention is measured over the questions it is asked. So of a hundred questions some are declined and the rest are answered, and of the answers given, ninety-four in a hundred are right — leaving six wrong in every hundred answers, which is about one in seventeen.

An abstention does not belong in that error count. It is the behaviour the refusal table was built to produce, and charging the system for it converts its best feature into a defect on the same line as a wrong number. The two rates are worth tracking together and never worth adding. Eventually one of the wrong answers lands in a leadership channel and gets quoted, and whether the product survives that depends entirely on what exists before it happens.

An answer record, written for every answer, is the foundation:

{
  "answer_id": "ans_01J9F2K4",
  "asked_at": "2025-08-07T09:14:22Z",
  "channel": "C04REV", "thread_ts": "1754557..", "asked_by": "U03KD",
  "question": "active users in DACH last month",
  "route": "semantic_layer",
  "metric": "active_users", "metric_version": "2025-03-11",
  "retrieved_tables": ["fct_events", "dim_accounts", "dim_plans"],
  "prompt_sha256": "8f2c...", "model": "claude-opus-4-6", "temperature": 0,
  "spec": {"metric": "active_users", "time": {"grain": "month", ...}},
  "sql": "SELECT COUNT(DISTINCT e.user_id) ...",
  "warehouse_query_id": "01b8-0a04-...",
  "row_count": 3412, "truncated": false,
  "checks": {"grain": "pass", "literals": "pass", "freshness": "pass"},
  "watermark": "2025-08-06T14:00:00Z",
  "answer_value": 134802,
  "latency_ms": 2410, "cost_usd": 0.021
}

The fields that do the work in an incident are the metric version, the prompt hash, the warehouse query id and the thread reference. The first two let you reproduce the exact conditions three weeks later, when the prompt has changed twice and the metric definition once. The query id lets a DBA pull the actual execution from the warehouse's own history, including the plan and the bytes scanned, which settles arguments about whether the data changed underneath. The thread reference is what makes the correction reach the people who saw the original.

Each of those four is doing a job that the others cannot, and the reasoning is worth having explicitly. Without the prompt hash and the metric version, a failure to reproduce is uninterpretable — the prompt changed, the definition changed and the data changed in the interval, and no evidence distinguishes them. The query id settles a different kind of dispute, and it settles it because the warehouse's execution history is a second instrument that shares no step with the code under suspicion: it did not go through the retriever, the model or the compiler, so it cannot be wrong in the same way they are. The thread reference is the only field concerned with people, and without it a correction gets published to whoever is reading rather than to whoever read the original.

Both branches of that diagram converge on the same step, and the convergence is the design. Whether the answer reproduces or not, the next move is to find every other answer that touched the same metric or table. A fault that reproduces is a fault in the current system and every past answer through the same path shares it; a fault that does not reproduce means something changed underneath, and every past answer from before the change shares that instead. Neither case is ever about the one answer somebody complained about.

Blast-radius search is the step teams forget to build and then need at the worst moment. When active_users turns out to have excluded a plan tier since March, the question is not what one answer said; it is which forty-one answers over five months touched that metric, who received them, and in which threads. That is a SELECT over the answer log, and it takes three minutes if the log has the metric name on it and three days if it does not.

The gap between three minutes and three days is not a difference in effort, and it is decided months earlier by a schema choice. Writing an answer record at all is the requirement. Writing the metric name into it as a queryable column is a design decision, and it is the one that turns the blast-radius question into a filter. Without that column the same question is answered by reading the SQL text of every answer in the window or replaying its prompt, which is the three days — and it is three days of work that has to happen while the wrong figure is still circulating.

Two defences that cover different windows#

Two more things belong in place before launch. A kill switch that an on-call engineer can flip in under a minute, which posts a fixed message and stops the bot answering, so a definition bug does not keep producing answers while you investigate. And a published accuracy figure, stated before anyone uses the product: 94% on 300 evaluated questions, abstains on 8%, not a source for certified financial reporting. Publishing it changes the meaning of the first wrong answer from a betrayal into a known and quantified event, and it is the difference between an incident and the end of the project.

Those two defences and the blast-radius search cover different windows and none of them substitutes for another. The kill switch acts forwards: it prevents answers that have not been given yet, and it does nothing whatever about answers already sitting in threads. Blast-radius search acts backwards over exactly the answers the kill switch cannot reach. A team that has built one of the two usually believes it is covered, and it is covered for half the timeline.

The published figure is the one defence whose effectiveness depends on when it was published rather than on what it says. Stated in advance, it makes the first wrong answer an instance of something already agreed. The same sentence produced after a wrong answer is an excuse, and reads as one, whatever its arithmetic. As an analogy — and it is an analogy, not a mechanism — this works the way a stated error bar on an instrument does: a reading inside the band is the instrument behaving as specified, and a reading outside it is a fault. The place the analogy breaks is worth marking, because it breaks where the misconception forms. An instrument's error is a distance, so being wrong by a little is a small failure. A wrong answer here has no distance to the right one — it is a different number, and how bad it is depends on who read it and what they did next, which no error bar can express.

That last dependency is the belief this whole chapter is arranged against. It is tempting to treat a measured accuracy as the thing that decides readiness, because accuracy is the number that improves when the work goes well and it is the number a pilot produces. But accuracy is a property of the system alone, and it does not move at all when the bot is added to a leadership channel: the same eval set returns the same figure, the same one answer in seventeen is wrong. What changes is everything on the other side — a wrong figure in an analytics channel is corrected in the thread by someone who reads SQL, and a wrong figure in a leadership channel is quoted into a board pack by someone who does not. A constant sitting next to a variable that has moved by an order of magnitude is not evidence of readiness, and reading it as evidence is how a product that measured well ends anyway.

The failure being managed is a trust failure and not a metrics one, which is why "connect the model to the warehouse and iterate on accuracy from real usage" is the wrong opening move even when the components are all correct. Iterating from real usage means the training signal is a stream of unverified answers already given to executives, and the accuracy curve looks fine right up to the moment it does not matter any more.

The order the components get built in#

The order in which these components get built is close to the order they appeared here, for a practical reason: each one makes the next measurable. The evaluation set and schema-retrieval recall come first, because they are what turn every later change into a measurement. Then the schema block with real column values, then the structural validator and the read-only role together, then the grain check and the result checks, then the registry for the metrics that people argue about, and only then a surface anyone outside the team can reach.

The corpus of wrong-but-valid queries is the artefact that persists across all of it. It starts with the fan-out from chapter 5, gains an entry every time someone finds a plausible number that was wrong, and is run against the verifiers on every change. A system that can name the twenty-two ways it has been wrong, and demonstrate that it now catches them, is one you can defend in a leadership meeting when the twenty-third arrives.

That corpus is also the clearest statement of what the whole arrangement is for. Nothing above raises the accuracy figure by a point. The sample of sixty requests decides whether the project should exist, the refusals decide which questions never get a plausible-looking answer, the pilot decides when the audience widens, and the answer log decides how long a bad definition stays uncorrected — and every one of those is about what happens around a wrong answer rather than about how often one occurs. The rate is what the eval set measures. The consequences are what gets designed, and they are designed before the first wrong number, because afterwards there is no time to build any of it.