Tribal knowledge fails the new-engineer test

Dana joined on a Monday and was given a small, real task on Wednesday: add a sentence to the billing assistant's system prompt requiring it to state the refund window in days before explaining it. She had the repository, commit access, and a laptop that built. The change itself took her eleven minutes. Getting it merged took two days and consumed about nine hours of other people's attention, distributed across a channel where nobody noticed the total.

Every gate this topic has described existed on that team and functioned correctly. The policy file was in the repository, the owners were named individuals, the suites were calibrated and the annotation workstream was funded. None of that reached the person doing the work, because the connective tissue — which file to read first, what the words in it mean for the change in front of you, what to type — had never been written down. It had accreted in six people's habits over eighteen months, and to those six the system felt fully documented, since every question Dana asked had an answer somebody could give from memory in under a minute.

Onboarding is the only cheap instrument you have#

There is no way to audit this from inside the team. Ask the six engineers whether their AI practices are documented and they will say yes and be sincere, because they are checking whether they know the answers rather than whether the answers are reachable. A new joiner is the only person in the building who cannot substitute knowledge for documentation, and their first week is the one window in which their ignorance is a feature.

So run it deliberately. Give the joiner a genuine behaviour-affecting change in their first few days, tell them the standard applies to them exactly as it applies to everyone, and add one constraint: they may read anything in the repository or the docs, and they may not ask a human. Every point at which they get stuck is a defect, and unlike most documentation feedback it comes with a location, a timestamp and the specific sentence that should have existed. Dana's list above is the output of running the test accidentally; running it on purpose costs the joiner's time, which you were spending on onboarding anyway, and about an hour of yours to read the log afterwards.

The instinct that destroys the measurement is the onboarding buddy. Pairing a joiner with a veteran for their first change is kind, produces a merged pull request on day two, and tells you nothing, because the veteran supplies from memory exactly the connective tissue you were trying to detect the absence of. The compromise that keeps both properties is a stuck rule: the joiner works alone, and when they have been stuck for twenty minutes they ask, in a channel, and the question gets logged before it gets answered. Nobody is abandoned, and every answer that had to come from a person is recorded as a gap with the file they were looking at when they hit it.

The change should be real, since a synthetic exercise routes around the parts that matter. It should also be small enough that the failure mode is confusion rather than difficulty, because the test is measuring reachability of the process and not the joiner's grasp of the domain. A one-sentence prompt change and a one-integer config change are both good candidates, and running one of each is better than running two of the same kind, since chapter 2's asymmetry means they exercise different parts of the map.

The map the joiner needs#

What Dana was missing is a route from I want to change this file to this is merged with evidence. That route is a sequence of decisions with a defined artifact at each step, which is the shape a diagram serves better than a page of prose.

Written prose describing that sequence will drift within a quarter, because the suite names change, the owners change and the commands change, and nobody updates a page they never read. The version that survives is generated from the files that CI already enforces, which is the same review-policy.yaml and CODEOWNERS from chapters 2 and 3.

$ ./scripts/change-plan config/retrieval.yaml

  config/retrieval.yaml
  class     serving_config
  owner     @serving-marc   (deputy @deputy-priya)
  suites    billing_core, refusal, multi_doc
  evidence  eval_run, slice_table, cost_latency_delta

  run:      make eval SUITE=billing_core,refusal,multi_doc BASE=main
            (~11 min, ~$3 in tokens, writes evidence.json)

  notes     refusal/refusal-0142 is accepted_failure_until 2026-06-30
            cost_latency_delta comes from the same run; see docs/eval.md#cost

  If no suite covers the behaviour you are changing, open an eval-data
  request:  ./scripts/eval-request --slice <name> --why "<one line>"
            owner @eval-sofia, 200 stratified items, 48h

Twenty lines of script output replaces five of the seven questions in Dana's table, and it stays correct because it reads the same source of truth the CI job reads. The remaining two questions — which prompt directory serves production, and whether a red refusal suite is her fault — are answered by the same principle applied elsewhere: the serving config should name the prompt path so that a joiner can trace it, and a suite failure caused by an accepted-failure marker should say so in the failure output rather than looking like a regression.

The other half of the map is what the CI check says when it blocks. Chapter 2's script prints the missing evidence, and the difference between an error message that lists what is absent and one that tells the reader how to produce it is most of the onboarding experience for anyone who never reads documentation until something breaks.

Behaviour-affecting change without evidence:
  - serving_config: no run for suite 'refusal'
  - serving_config: missing 'cost_latency_delta'

  Produce both with:
      make eval SUITE=billing_core,refusal,multi_doc BASE=main
  Then commit evidence.json. Owner for this path: @serving-marc.
  Why this is required: docs/ai-practices.md#config-is-behaviour

That leaves one genuinely hand-written document, and the mistake is to fill it with instructions the script already generates. Commands and owner names belong in the generated output, because prose copies of them go stale silently and a stale command is worse than no command. What belongs on the page is the reasoning that no tool can emit: why a config value counts as a behaviour change when it looks like a tuning knob, why removing a failing eval case is an edit to the product's definition of correct, why a spot check of five outputs cannot see a doubled error rate, and why the model build is pinned when the provider's alias would work fine. The billing team's version runs to about 900 words under four headings, and it is the only file in the repository whose purpose is to make the other files' rules make sense. Reading it takes six minutes and it changes maybe twice a year, since the arguments are stable even when the suite names are not.

When the billing team put the script, the failure message and that one page in place, the next joiner made the equivalent change on his second day in forty minutes with no questions in the channel. The one question he did ask afterwards was whether the accepted-failure marker on refusal-0142 was still deliberate, which is the kind of question the test is supposed to surface, since it came from reading the artifact rather than from being unable to find it.

What passing the test does and does not tell you#

It tells you your gates are reachable. It says nothing at all about whether they are the right gates, and this is the misreading worth guarding against, because a team can document a bad standard perfectly and then point at a smooth onboarding as evidence of health. A joiner who lands a change in forty minutes on a suite of 120 cases labelled by one person with no guideline has demonstrated that the process is legible and has demonstrated nothing about whether the process produces trustworthy answers. Documentation cannot manufacture the authority chapter 3 assigned or the funded hours chapter 4 argued for; it can only make them findable once they exist.

The test is also a wasting asset. A joiner is a clean instrument exactly once, so a team that hires twice a year gets two measurements a year, and the practices drift faster than that. Two things extend it. Every question asked in the team channel about how to land a change is a defect report against the map, and treating it that way — with a small commit that answers it in the generated output or the docs page — keeps the map current between hires. And an engineer moving onto a surface they have not worked on, a backend engineer touching retrieval for the first time, is a partial instrument for that surface, which is worth using when hiring is slow.

There is a version of this that goes too far. Not everything a team knows can or should be written down, and a team that responds to this chapter by writing a 40-page process manual has produced something no joiner will read and no one will maintain. The billing team already had a Confluence space of that size, largely accurate, and it did not help Dana for a minute, because none of it was reachable from the file she had open. What survived was one page, one script, and a set of error messages that teach at the moment of failure.

Where to start with your own team#

Take the next behaviour-affecting change that comes through and write down, honestly, what a person who joined last week would have had to ask to land it. That list is your backlog, in priority order, and it is usually four to eight items long rather than forty.

Then look at what the list implies about the four artifacts this topic has treated as separate. If the joiner cannot find which suite is affected, you have a policy file problem. If they cannot find who approves, you have the team-label problem from chapter 3. If they find the suite and it does not cover the behaviour they are changing, you have the annotation-latency problem from chapter 4, and the documentation fix will not touch it. If they can find everything and the evidence they produce is a spot check of five outputs, the standard itself is the thing that needs rewriting, and chapter 2's arithmetic is the argument to make.

The exercise that grounds all of it is looking at outputs directly. Arize's error-analysis homework for the AI evals course walks through a session of reading fifty traces, naming the failure modes and turning them into criteria, and putting a new engineer through that in week one gives them the thing no process document conveys, which is a concrete sense of what this system gets wrong and why the gates are shaped the way they are. Everything in this topic is downstream of someone having looked carefully at what the system actually produced, and the practices exist so that the looking turns into decisions somebody owns.