Attributing a bad outcome to one specific config line

The events finding went to the platform review with the pg_indexes output, the migration date, and the two EXPLAIN ANALYZE runs attached. It took nine minutes to be deflected. Nobody disputed that the index existed; what they disputed was the story about how the agent came to write those queries. The tech lead's position was that the agent writes odd queries about once a fortnight, that two people on the team had seen it do the same date-range-then-filter thing in a service whose config file says nothing about indexes at all, and that removing a line because it happened to be present during a bad session is the same reasoning that leads people to blame the deploy that was closest in time. He asked, reasonably, what would be different if the line were not there, and the honest answer at that moment was that nobody could say.

That is the tie the rest of this chapter is about breaking. The claim you can now support is that the line is false. The claim you need is that the line is operative — that it entered the agent's context, that the agent reasoned from it, and that the action followed from that reasoning. Those are separable, and a stale line lives comfortably in the gap between them, model variance — the ordinary spread in what a model produces when the same task is run more than once — makes "the model is sometimes wrong" an explanation that covers every individual observation and can never be exhausted by adding more observations of the same kind.

Four artefacts, taken from one session#

Handling this like a chain of custody is the right instinct. A physical exhibit is worthless in evidence unless you can account for it continuously from the scene to the courtroom, and it is worthless whether or not anybody suspects tampering in a particular case, because a chain with a gap in it cannot be distinguished from one that was interfered with. A causal claim about a config line has the same structure. Four artefacts, all from the same session, all identified by the same session ID:

{
  "session_id": "01J9WQ7K3D8E",
  "started_at": "2026-07-31T09:12:04Z",
  "repo_sha": "e4b1c77",
  "model": "claude-opus-4-6-20260615",
  "request": "Add GET /customers/{id}/events returning the last 30 days of events for that customer.",
  "config": {
    "sha256": "b41f0c9a8d2e5f7031ac6b8e94d5f2a1c0e7b3d6a9f4128e5c7b0d3a6f9e2c48",
    "bytes": 6142,
    "sources": [
      "/home/agent/.config/agent/AGENTS.md",
      "/repo/AGENTS.md",
      "/repo/services/events/AGENTS.md",
      "mcp:postgres-tools/server-instructions"
    ]
  },
  "artifact": "PR #4471",
  "outcome": "seq scan over 30d window, 2.9M rows filtered in Python"
}

The request matters because the task has to be one where the right answer is not in dispute. "Return a customer's events for the last 30 days" has an obvious correct query, so nobody can argue that the agent faced a genuinely hard design choice and picked a defensible branch. If the task is ambiguous, the trace becomes an argument about taste.

It is worth spending a few minutes choosing which session to build the chain from, since the incident that made everyone angry is often the worst candidate. Incidents tend to be tangled: a long session, four files touched, a production symptom with several contributing causes, and a config line somewhere in the middle of it. What you want is the smallest task that the rule can plausibly govern, ideally one that was already done correctly somewhere else in the codebase so the correct output is visible for comparison. In the events case the endpoint had a sibling — GET /customers/{id}/invoices, written by a human eight months earlier, filtering on an indexed column in a single query — which makes the generated handler look wrong to a reader who knows nothing about Postgres. A small task also means a short trace, and a reviewer who has to read forty lines will read them.

The configuration matters in its assembled form rather than as the file in git. What the agent was actually given is a concatenation of several sources, and the file you read this morning is not necessarily the bytes that were injected on 31 July. Someone may have edited the repository file since; a subdirectory file may have added or overridden an entry; an MCP server's instructions may carry a claim about the database that nobody thinks of as configuration at all. Recording a hash of the exact assembled text, together with the list of sources that produced it, is what makes the rest of the chain refer to something specific.

The trace matters because it is the only place where the line and the action are joined by something other than your inference. What you are looking for is the agent naming the rule while deciding:

[09:12:41] assistant (thinking): The events table has no index on customer_id and
            AGENTS.md says never to filter by it directly. So the query should
            restrict on created_at, which is indexed, and the customer filter has
            to happen after the fetch. Thirty days is a bounded window so the
            result set should be manageable.
[09:12:58] tool_call: write_file("services/events/api/handlers.py", ...)

The output matters because a trace without an artefact is a plan. The two together give you a line, a stated reason, and a committed consequence:

# services/events/api/handlers.py — from PR #4471
async def list_customer_events(customer_id: int) -> list[Event]:
    since = datetime.now(UTC) - timedelta(days=30)
    # events has no index on customer_id (see AGENTS.md); filter in app code
    rows = await db.fetch("SELECT * FROM events WHERE created_at >= $1", since)
    return [Event(**r) for r in rows if r["customer_id"] == customer_id]

The comment in the generated code is a gift, because it carries the attribution into the artefact itself and survives after the session transcript has been rotated out. It is not something you can rely on appearing, which is why the trace is captured separately.

What each missing artefact leaves open#

The chain is only as strong as its weakest link, and each gap has a specific objection waiting in it. This is worth knowing in advance, because the objections arrive in exactly this form and they are not obtuse.

Aggregate metrics do not close these gaps either, though they look rigorous. Counting that the agent produced this query pattern in 6 of 9 sessions on this service over three weeks establishes a rate, and the reply is that the service is unusual, or the tasks were similar, or the base rate for odd queries is unknown. Nine sessions of correlation is a weaker instrument than one session with a complete chain, because the chain contains the agent stating the mechanism rather than you proposing one.

The instinct to reach for volume comes from the wrong analogy. When you are estimating how often something happens, more samples are the only thing that helps, and a single incident is worthless. When you are establishing that a particular input produced a particular output through a stated mechanism, one fully documented instance carries the argument and the tenth adds almost nothing, in the same way that one reproducible bug report with the exact input, the version, the stack trace and the wrong value beats fifty reports saying the feature is flaky. Which of those two questions you are answering determines what you should be collecting, and teams investigating config rot usually think they are doing the first when the thing blocking them is the second.

Capturing this before you need it#

Reconstructing a chain three weeks after the fact is mostly luck, and the luck runs out at the configuration hash. Doing it prospectively costs very little. A hook — a script the agent runtime runs by itself at a fixed point in a session, without the model being asked or able to skip it — is all the machinery required, one of them at session start to snapshot what was assembled and one at session end to file the transcript alongside it:

#!/usr/bin/env bash
# .agent/hooks/session-start.sh — runs before the first model call
set -euo pipefail
out="$HOME/.agent/sessions/$AGENT_SESSION_ID"
mkdir -p "$out"

# The assembled prefix as it will be injected, not the files on disk.
agent config render --assembled > "$out/config.txt"
sha256sum "$out/config.txt" | cut -d' ' -f1 > "$out/config.sha256"

jq -n --arg sha "$(cat "$out/config.sha256")" \
      --arg repo "$(git rev-parse HEAD)" \
      --arg model "$AGENT_MODEL" \
      '{config_sha:$sha, repo_sha:$repo, model:$model,
        started_at:(now|todate)}' > "$out/meta.json"

Two hundred sessions of this on a busy service is a few megabytes of text. The value shows up the first time an argument like the one in the platform review happens, and it also has a use nobody plans for: when two sessions on the same repository behave differently, the first thing to compare is the two config hashes, and about a third of the time they differ for a reason that explains the whole discrepancy — someone's user-level file, a directory-scoped file that only loads under services/events, an MCP server that changed its instructions in a background update.

The events investigation turned on exactly that. Two engineers tried to reproduce the bad query and one of them could not; the agent wrote the indexed query without complaint on her machine and the team briefly concluded the whole thing was noise. The two config hashes differed by 214 bytes. Her checkout predated a services/events/AGENTS.md that had been added six weeks earlier and carried the index line in a directory-scoped file, while the root file said nothing about indexes at all — which also explains why everybody who had gone looking for the rule in the obvious place had failed to find it. Without the hashes that is an unresolvable disagreement between two competent people, both of whom are reporting accurately.

There is one detail worth getting right in the snapshot. Render the assembled text the agent will actually receive rather than concatenating the source files yourself, because loaders trim, deduplicate, expand imports and sometimes truncate at a size limit. A config file that is 8 KB on disk and 4 KB in context is a live possibility, and if you hash the wrong object you will spend an afternoon proving that a line was present when it was silently dropped.

The chain is not the counterfactual#

An honest account has to include what this does not establish, and it does not establish that the agent would have done better without the line. It establishes that the line was present, that the agent cited it, and that the action followed. Someone can still say that the model would have written a mediocre query anyway and merely reached for the nearest available justification, and there is nothing in the chain that refutes that.

The concern has a real basis rather than being lawyerly obstruction. A model's stated reasoning is a generated text, produced by the same process that produced the code, and it is not a log of the computation that produced the answer. The research literature on chain-of-thought faithfulness — whether the reasoning a model writes out before answering is a true account of what drove the answer — has repeatedly found cases where models are influenced by a cue in the prompt, act on it, and give a different account of why — so a trace citing a rule is evidence that the rule is doing work, and it is not a mechanical proof. The stronger version is where the model does not cite the rule and is nonetheless steered by it, which the chain will never catch.

There is also a subtler failure that the chain hides rather than reveals. In the session above the rule was operative and visible. A rule can also be operative by omission — the agent never considers an approach, says nothing about why, and the absence leaves no trace to capture, which is the discovery problem from chapter 1 reappearing at the level of evidence. Prohibitions that suppress an option produce silence, and you cannot photograph silence.

What both of those point at is the same missing experiment. Everything in this chapter describes one arm: the world with the line in it. The objection from the tech lead — what would be different if the line were not there — is a question about a second arm, and it can be answered rather than argued about, by running the same task on the same commit with the line removed and putting the two traces side by side. That comparison is the next chapter.