Per-call records are what turn complaints into a metric

The spreadsheet has 47 rows and four columns: date, account, "what the customer said", and a free-text note from whoever took the ticket. It lives in a support engineer's personal drive, it has been maintained for three weeks, and it exists because there was nowhere else to put the observation. Reading it end to end takes twenty minutes and produces one clear pattern — 31 of the 47 mention a document the customer describes as a scan, a fax, or "the PDF from the registry" — and one useless one, because the notes say things like "answer was about the wrong section" without any way to find the answer in question.

That is the state of a system in its second year with a gateway, an orchestrator, two providers, a managed vector store, Postgres, full distributed tracing, SLOs on latency and error rate, an on-call rota, and no incidents ever. The spreadsheet is a quality signal that has been running in production for three weeks with a sample size of 47, and the reason it cannot be acted on is that none of its rows can be joined to anything the system knows.

The record, field by field#

What makes the spreadsheet actionable is not more complaints. It is a durable per-call record that a complaint can be joined to, and the join key has to exist on both sides.

CREATE TABLE call_records (
  request_id        UUID PRIMARY KEY,
  occurred_at       TIMESTAMPTZ NOT NULL,
  tenant_id         TEXT NOT NULL,
  request_class     TEXT NOT NULL,          -- 'chat' | 'extract' | 'summarise'
  -- what was asked
  prompt_template   TEXT NOT NULL,          -- 'qa_with_context'
  prompt_version    TEXT NOT NULL,          -- 'sha:8f31c0a'
  prompt_variables  JSONB NOT NULL,         -- redacted, not the rendered string
  -- what it was asked with
  index_name        TEXT,
  index_version     TEXT,                   -- '2026-03-14T02:11Z'
  embedding_model   TEXT,
  retrieved         JSONB,                  -- [{chunk_id, doc_id, score, rank}]
  -- who answered
  model             TEXT NOT NULL,          -- snapshot id, never an alias
  params            JSONB NOT NULL,         -- temperature, max_tokens, tools
  -- what came back
  completion        TEXT,                   -- sampled; null when not retained
  finish_reason     TEXT,
  input_tokens      INT NOT NULL,
  output_tokens     INT NOT NULL,
  latency_ms        INT NOT NULL,
  -- how the document got classified, for slicing
  doc_type          TEXT,                   -- 'native_pdf' | 'scanned' | 'html'
  attempts          INT NOT NULL DEFAULT 1
);

Three choices in that schema carry most of its value. Storing prompt_variables and the template version rather than the rendered prompt string keeps the row small and keeps personal data out of a table you will want to retain for ninety days, while still allowing exact reconstruction by re-rendering the pinned template. Storing retrieved as chunk identifiers with scores and ranks, rather than chunk text, makes the row roughly 400 bytes instead of 9 KB and preserves the only thing you actually query — which chunks came back and in what order. And doc_type is denormalised onto the record deliberately, because the slice you need to cut by is a property of the input, and joining to a documents table at query time means the slice breaks when a document is deleted.

The record is written by a consumer of the call_completed event from chapter 2, so nothing here is on the request path. At 40,000 calls a day with completions sampled at 10%, the table grows about 24 MB a day and 2.2 GB over ninety days, which is a rounding error against the vector store.

The join is a product change, not a data change#

The record is inert until an outcome label can be attached to it, and that is where most implementations stop. Three sources of labels are worth wiring, in increasing order of cost and decreasing order of noise.

The cheapest is an explicit signal in the product: a thumbs-down control that submits the request_id already rendered in the response payload. Response rates run around 0.2–2% of responses, heavily skewed toward negative, which is fine because you are detecting movement rather than estimating absolute quality.

The second is the support join, and it is the one that converts the spreadsheet. It requires that the request id be visible to the customer or to the support agent — a small ref: 6f2b0f6e under the answer, or a copy-to-clipboard on hover — so that "the answer was wrong" arrives with a key attached. That is a two-day product change with a larger effect on this plane than any modelling work, and it is routinely skipped because it looks cosmetic.

The third is implicit acceptance, which is the strongest signal when the product allows it. If the output lands in an editable field, record whether the human edited it before submitting, and how much: an edit distance over the accepted text — a count of the character insertions, deletions and substitutions between what you produced and what they submitted — gives a continuous quality signal on 100% of traffic at zero inference cost.

With labels joined, the spreadsheet becomes a query.

SELECT doc_type,
       count(*)                                        AS calls,
       count(*) FILTER (WHERE label = 'bad')           AS bad,
       round(avg(judge_score)::numeric, 3)             AS mean_score,
       index_version
FROM call_records r
LEFT JOIN outcome_labels l USING (request_id)
LEFT JOIN judge_scores  j USING (request_id)
WHERE occurred_at >= now() - interval '14 days'
GROUP BY doc_type, index_version
ORDER BY bad DESC;

Four rows, and the last one is doing the work: the same slice under the previous index version scored 0.90, which dates the regression to an index rebuild rather than a model change or a prompt edit. Chapter 5 showed why the same information is invisible in an aggregate — the arithmetic of a small slice moving a large mean — and this table is what the aggregate is hiding.

Retention is a design decision, not an afterthought#

A table containing prompt variables and completions is a table containing customer content, and treating it as ordinary application data is how this plane gets switched off six months after it is built. Three decisions keep it viable. Completions are sampled rather than retained wholesale, at a rate that gives each slice enough rows to read — 10% of general traffic plus 100% of anything with a negative label or a finish_reason other than stop, which biases retention toward the calls anyone will actually want to read. Prompt variables are redacted at write time by the same routine the application already uses for logs, and the redaction runs in the consumer rather than in the request handler so it costs the user nothing. Retention is tiered: metadata and scores for a year, because year-over-year slice comparison is the whole point, and content for ninety days, which comfortably covers the interval between a regression starting and support noticing it.

Tenant-level opt-out is worth building before a customer asks for it, and it is one boolean consulted by the consumer. What the opt-out must not remove is the metadata row, because a tenant with content retention disabled still needs to appear in slice counts; losing them from the denominator silently biases every score you compute.

The judge, and the version of it that fails#

A judge model scoring sampled completions is the right way to get a continuous signal on traffic that has no natural label, and the naive implementation of it is the most common way this whole plane is built wrong. Four properties separate the two.

Sampling rather than scoring everything. At 40,000 calls a day, a judge prompt of roughly 1,200 input and 200 output tokens costs about $0.0066 per call at $3 per million input and $15 per million output tokens, so 100% coverage is $264 a day and a stratified 5% sample is $13. The sample is not merely cheaper; it is also the version you can afford to run with a stronger, slower judge model, which matters more for agreement than volume does.

Stratification rather than uniform sampling. A uniform 5% of traffic gives 159 scanned-document calls a day against 1,097 native PDFs, so the slice you most need to watch has the widest confidence interval. Sampling to a fixed count per slice — 200 calls per doc_type per day — gives every slice the same statistical power and costs less in total.

Pinning the judge. The judge is a model call with a prompt, so it drifts exactly like the system it monitors, and an unpinned judge produces score movements that are indistinguishable from product regressions. The judge's model snapshot and prompt version belong on every judge_scores row, and changing either is a migration: score a held-out set — a fixed batch of records set aside and never used to tune either version — with both versions and record the offset before switching.

Calibrating it against humans, once, and then periodically. Take 200 records stratified across slices, have two people label them independently, and measure agreement between the judge and the human majority. Cohen's kappa, which measures how often two raters agree after discounting the agreement you would get from chance alone, runs from 0 at chance to 1 at perfect agreement; around 0.6–0.7 the judge is a usable monitor, and below about 0.4 it is measuring something else, so the honest response is to narrow the rubric — "does the answer cite a passage that contains the claim" is a question a judge answers reliably, while "is this answer good" is not.

Averaging an unversioned judge over unsliced mixed traffic and alerting on the mean fails all four, and it fails in a way that looks like success for months: the number exists, it goes on a dashboard, it moves a little, and nobody can act on any movement because they cannot tell whether the product changed, the traffic mix changed, or the judge changed.

What it feels like to use during an incident#

The value of the record shows up in the shape of the investigation rather than in any dashboard. A complaint arrives with a reference, so the first step is a single-row lookup that returns the prompt version, the model snapshot, the index version, the five chunk ids with their scores, and the completion. In the case above, that lookup showed the top-ranked chunk for a scanned filing had a similarity score of 0.71 against a corpus where the median top-1 score was 0.86, and that the chunk spanned the end of one section and the start of another. The second step is a slice query asking how common that pattern is, which returns the table above. The third is a version comparison, which dates it to the rebuild. Twenty minutes, and none of it requires reproducing anything.

Compare that with the same investigation without the plane, which is what the team had been doing: read the complaint, ask the customer which document, request the document, try to guess the question they asked, run it against the current system, get a plausible-looking answer, and close the ticket as not reproducible. Three of the 47 rows in the spreadsheet had that phrase in the notes column, and it was accurate each time.

What one engineer builds in one quarter#

The order matters, because the components have a dependency structure and the expensive one is last.

Build the record and the event that populates it, with pinned prompt, model and index versions stamped on every row. Build the request id into the response payload and into the support tooling, so the existing complaints join. Build the slice query as a scheduled report — a table in an email, not a dashboard product — cut by doc_type, tenant_id, request_class and version fields. That is most of a quarter and it is the part that cannot be bought.

Deliberately do not build: a judge scoring 100% of traffic, an annotation UI when a spreadsheet with a request-id column works, alerting on score thresholds before you know the score's noise floor, a full offline evaluation harness, and a real-time quality dashboard. Every one of those is defensible in year two and every one of them competes for the same quarter as the record, which is the only component whose absence is unrecoverable — a complaint about an answer from three weeks ago cannot be investigated later if the call was never written down.

The case for building none of this#

Everything in the last two chapters argues toward more machinery, and applied without a threshold that argument justifies a vector store nobody queries and a judge on an internal tool with four users. The plane described here costs a quarter of an engineer's time and a table that grows forever, and there are real systems where the correct amount of it is one log line. Working out which systems those are turns on variables that have very little to do with how many requests a day they serve.