The eval set is your definition of correct, so it needs an owner

Over five months the billing assistant's suite pass rate climbed from 76% to 91%, which the team reported in every monthly review as steady progress on quality. The wrong-answer tag rate in production over the same five months stayed between 1.1% and 1.4%, exactly where it had been at the start. Somebody eventually ran git log over the cases directory to work out where the improvement had come from.

$ git log --diff-filter=D --name-only --since="5 months ago" -- evals/cases/ \
    | grep -c '^evals/cases/'
23
$ git log --diff-filter=D --format='%h %an %s' --since="5 months ago" -- evals/cases/
a91c4e2 rk  remove flaky refusal cases, they fail nondeterministically
77b0d13 mn  case no longer representative of current product
3e0aa87 rk  drop annual-plan cases, corpus doesn't cover this yet
c02f118 jd  clean up eval set before release
...

Twenty-three cases had been deleted. Nineteen of them had failed on the most recent run before their deletion. The suite had not got better at anything; it had stopped asking the questions the system got wrong. Every deletion was made by someone acting in good faith, in a small commit, usually while trying to get a red pipeline green so an unrelated change could merge, and none of them went past anyone whose job it was to say no.

Deleting a failing case is an edit to the definition of correct#

An evaluation suite is not a test suite that happens to be statistical. It is the written form of what your team means by a correct answer, and every case in it is a claim: this input should produce something like this output, and if it does not, we have a problem. Deleting a case retracts the claim. The commit message drop annual-plan cases, corpus doesn't cover this yet is a decision that the assistant is no longer expected to handle annual-plan questions, taken by one engineer, in passing, on a Tuesday. That is a product decision with a support cost attached, and chapter 1's incident is what it looked like four months later when the annual-plan slice ran at 9.8% wrong.

The dynamic is self-reinforcing, because a failing case imposes a cost on whoever is trying to merge and imposes nothing on anyone else. Adding a hard case is unrewarded work that makes the dashboard worse; removing one is a two-line commit that unblocks a release. Under that gradient an unowned suite drifts monotonically towards the cases the system already passes, and the pass rate becomes a measure of how many awkward questions the team has retired rather than how well the product works. This is the same failure that made database schemas rot in organisations where every service could add a column and nobody could refuse one, and the remedy that worked there works here: a named steward who reads every change to the artifact and can decline it.

A team label is not an owner#

Most teams believe they have ownership because their CODEOWNERS file has entries. The entries usually look like this, and they do not give you what the previous chapter's gate needs.

# Before: every path owned by a group, so any member's approval satisfies it
prompts/            @acme/ai-platform
config/             @acme/ai-platform
evals/              @acme/ai-platform

With a group as the owner, GitHub is satisfied by any one member's review, so the approver of a change to the definition of correct is whoever happened to be online. There is no person who can be asked why the annual-plan cases went away, no one whose calendar contains time for the question, and no one for whom a drifting suite is a visible failure. Replacing the group with individuals changes the property.

# After: a person per artifact class, with a named deputy for cover
prompts/                @rk-priya      @deputy-tomas
config/retrieval.yaml   @serving-marc  @deputy-priya
config/decoding.yaml    @serving-marc  @deputy-priya
config/models.lock      @serving-marc  @deputy-priya
skills/                 @rk-priya      @deputy-tomas
evals/cases/            @eval-sofia    @deputy-marc
evals/rubrics/          @eval-sofia    @deputy-marc
evals/harness/          @serving-marc  @deputy-sofia

The split inside evals/ is deliberate and is the distinction the Braintrust guide Evals for PMs draws: the harness is engineering code with engineering failure modes, while the cases and the rubrics encode what the product considers a good answer, and those are two different jobs with two different kinds of expertise. Sofia owning the rubric does not mean Sofia writes every case; it means no case enters or leaves the definition of correct without her reading the reason.

The last row is where the ownership boundary is least obvious and most consequential, and chapter 4 is about the workstream behind it.

Choosing which name goes in the file#

The instinct is to assign each artifact to whoever writes it most, and for prompts and config that is usually right, since the person making the changes has the deepest model of how they interact. For the eval data it is the wrong instinct, because the author of a prompt change has a direct interest in which cases are in the suite that judges it. When the same engineer both ships the prompt and approves the removal of the three cases it fails, nothing dishonest needs to happen for the outcome to be bad; the removal simply looks more reasonable to someone who has spent two days convinced the change is an improvement.

So the eval-data owner should be someone whose work is measured by production quality rather than by shipped changes. On the billing team that was Sofia, who came from the support side and had spent two years reading escalations, which meant she had opinions about what a correct answer was that predated any particular prompt. A product manager with domain depth is often the right person; an engineer is fine as long as they are not the primary author of the prompts being evaluated. What does not work is assigning it to the tech lead as one more thing, because the tech lead's incentive is delivery and the whole point of the role is to have someone for whom a shrinking suite is a personal failure.

Every owner needs a named deputy with the same authority, and the deputy should be someone who genuinely reviews rather than a rubber stamp on holidays, which in practice means the deputies are drawn from the other owners. Marc deputising for Sofia works because he already reads slice tables; a deputy who has never opened the rubric will approve whatever arrives.

What an owner's authority actually consists of#

Authority to reject is worth stating concretely, because "owner" in most organisations means "person who gets pinged". The eval-data owner's authority is a short list of powers that someone else cannot exercise: a case is removed only with a written rationale and her approval; a rubric criterion is loosened only with her approval; a case whose failure the team has decided to accept is marked rather than deleted; and a change that would invalidate existing labels triggers a re-labelling plan rather than a silent redefinition.

That last set of rules is easier to enforce when the case format leaves room for the decision.

# evals/cases/refusal/kb-annual-proration.yaml
id: refusal-0142
suite: refusal
provenance:
  source: production_trace
  trace_id: "01JQ7F3M2ZK4T8"
  captured: 2026-01-14
  labelled_by: sofia
  adjudicated_with: tomas
input:
  question: "If I switch to annual halfway through a month, what happens to the
             remaining monthly credit?"
expect:
  behaviour: refuse_or_escalate
  rationale: "Proration rules are not in the corpus; a confident answer here is
              the failure mode that generated three refund disputes in Q4."
status: active          # active | quarantined | accepted_failure
review:
  accepted_failure_until: null
  last_reviewed: 2026-03-02
  owner: sofia

status: quarantined exists so that a genuinely broken case, one whose expected output is wrong or whose input no longer parses, can be taken out of the scoring without leaving the repository, which keeps the record of what was once expected. accepted_failure_until exists so that a case the team has consciously decided not to fix this quarter stops blocking merges while remaining visible, with a date that forces the decision back onto someone's desk. Both of them make the honest action cheaper than the dishonest one, which is most of what a review standard can do. Under this regime the billing team's suite went the other way: over the following quarter it grew from 457 to 612 cases, the pass rate fell from 91% to 84%, and the production tag rate fell from 1.2% to 0.7%.

Making a refusal go somewhere#

Authority to reject is only usable if refusing has a defined next step, since otherwise the owner is choosing between approving something they disagree with and blocking a colleague indefinitely. The rule the billing team settled on has two parts: a refusal is written into the pull request as a reason rather than delivered in a meeting, and if the author disagrees, the engineering lead settles it within two working days.

The written reason is the load-bearing half. When Sofia refused the removal of the annual-plan cases, the comment said that the cases described a behaviour support was still receiving complaints about, that the corpus gap was the actual defect, and that the correct action was status: quarantined with a review date and a ticket against the content owner. That comment is now the record of why those cases exist, which is what makes the decision survivable when the same suggestion arrives from someone new eight months later. Rubric changes get the same treatment in a changelog next to the rubric, since a criterion loosened without a written reason is indistinguishable from a criterion that was never meant seriously.

The two-day time-box exists because stalling is how a strong ownership regime usually fails. An owner who is uncomfortable and not certain will let a request sit, and a request that sits for three weeks teaches everyone to route around the owner next time. Hamel Husain's write-up on how engineers and PMs should collaborate on evals is largely an argument about where that boundary sits, and its practical answer is that the domain expert owns the criteria while the engineer owns the machinery that applies them.

Ownership is what makes an escalation land#

The other thing a name buys is a destination. The postmortem from chapter 1 produced three follow-up items: pin and deliberately accept model builds, close the annual-plan corpus gap, and get out-of-corpus refusal cases into the suite. In the version of that team without owners, all three went into a shared backlog labelled AI quality, where two of them were still untouched at the next quarterly review, because a backlog item with a team label has no one for whom it is late.

With the ownership map above, each item has a person the moment it is written: the model lock is Marc's, the corpus gap is the content owner's, the refusal cases are Sofia's. None of that guarantees the work gets done on time, and it does guarantee that being behind is attributable, which is the difference between a follow-up system that functions and one that circulates. It also changes what a review meeting is for. When each item carries a name and a date, the meeting is four short status answers and one real argument about the item that slipped; when the items carry a team label, the meeting is a discussion about whether the team is prioritising quality, which produces nothing that can be checked a fortnight later.

Where naming an owner does nothing#

Ownership does not make one person the author of the artifact. Anyone on the team should be adding eval cases, and a suite that only its owner contributes to is a suite that reflects one person's imagination. The owner's job is the accept-or-refuse decision on changes and the standing responsibility for whether the artifact still describes the product.

It is also not a bottleneck if the volume is honest about itself. The billing team's eval directory takes about nine pull requests a week, most of them additions with clear provenance, and Sofia's median review time on them is under twenty minutes. If a single owner cannot keep up, the artifact is too large for one owner and needs splitting by suite, which is a different fix from removing the gate.

The failure that naming owners genuinely does not address is the one underneath. Sofia can refuse every unjustified deletion and still preside over an eval set that means nothing, because the cases arriving for her approval were labelled by engineers between tickets, with no guidelines, no calibration and no measurement of whether two of them would label the same output the same way. An owner with authority over data that nobody was funded to produce properly is a name attached to a number that describes the labellers rather than the system.