A diagnostic is a record with a location, not a line of a log

The loop runs make build, captures everything the command writes to stdout and stderr, and hands the result to the model with the instruction to fix the build. On a repository with generated protobuf clients — source files that a tool writes out from a schema, rather than code anyone edits by hand — a TypeScript project graph and a bundling step, a failing run looks like this:

$ make build
protoc --plugin=protoc-gen-ts -I proto --ts_out=gen proto/billing.proto
gen/billing_pb.ts written (1 file, 214 KB)
tsc -b packages/billing packages/api packages/worker
packages/billing/src/proration.ts(88,14): error TS2532: Object is possibly 'undefined'.
node_modules/@acme/telemetry/dist/index.d.ts(12,5): warning: 'Span' is deprecated
packages/api/src/routes/invoice.ts(41,7): error TS2345: Argument of type
  'number | undefined' is not assignable to parameter of type 'number'.
tsc -b: 2 errors, 1 warning (3 projects, 2 up to date)
esbuild packages/worker/src/index.ts --bundle --outfile=dist/worker.js
✘ [ERROR] Could not resolve "../../billing/dist/proration"
make: *** [Makefile:31: build] Error 2

The agent's next patch edits packages/worker/src/index.ts. Nothing in that file is broken. The bundler failed because packages/billing never emitted, and packages/billing never emitted because of the error on line 88 of proration.ts — the first diagnostic in the log and the only one that is a cause rather than a consequence. From the model's position, what arrived was a wall of text in which the last named file was the worker, the loudest marker was a red cross next to the worker, and the two lines that actually mattered were sandwiched between a deprecation warning from a dependency and a summary line counting projects. It patched what the text emphasised.

Three kinds of program produced the complaints in that transcript, and because the whole topic rests on what they emit, they are worth naming plainly before anything else is built on them. A compiler turns source code into something that can run, and refuses when the code cannot be turned into anything runnable; a build is the whole sequence of such steps that produces a deployable artefact from a repository. tsc is the TypeScript compiler, and the two error TS… lines above are it declining. A type checker does the part of that work that verifies the claims the code makes about the shapes of its values — that a function declared to return a number is never handed to something expecting a string, that a value which might be missing is not read as though it is certainly there. In some languages the type checker is inside the compiler, and in others, such as mypy for Python, it is a separate program run over code that would have executed regardless. A linter checks for style and for constructions that are legal but suspicious, such as an unused import or a database write that nobody waited for; ESLint is the usual one in this ecosystem. Every individual complaint that any of the three makes is a diagnostic: one finding, carrying a severity, a message, and a location in the source. That location is what separates a diagnostic from a line of a log.

Diagnostics are a familiar thing under an unfamiliar name — the red underline in an editor, the wall of output from a build that failed, the annotations attached to a pull request. What makes them worth a topic of their own is that they are the cheapest correctness signal a repository already contains, produced deterministically, in seconds, at a named line, by tooling nobody has to write. An agent loop — a model proposes a patch, a program runs checks over the result, the findings go back to the model, and the cycle repeats until the checks are clean — needs a verdict on every attempt, and the compiler gives one for free. The program that runs those commands, reads their output and decides what the model sees next is the harness, and this topic is about what the harness should put in front of the model. The failing run above shows what happens when the answer is all of it.

Topic 13.6 made the equivalent argument about test runners: raw stdout is a gate rather than a signal, because it tells the loop that something failed without telling it what to change, and the fix was to extract a structured failure record per failing test. Compilers and linters make that job dramatically easier, because unlike a test runner, most of them already hold the diagnostic as a data structure internally and will hand it to you in that form if you ask. You are not writing a parser to recover structure from prose; you are switching off the human-facing renderer.

What the compiler is throwing away when it prints#

Inside rustc, a diagnostic is a struct with a level, a message, a code, and a list of spans, each span carrying a file, a byte range, and a label. Inside tsc, it is a ts.Diagnostic with a file, a start offset, a length, a category, a numeric code, and a possibly nested messageText. The coloured, underlined, arrow-annotated output an engineer reads at a terminal is produced by a renderer that walks that structure and flattens it into lines. Piping the flattened form into a model asks it to reconstruct, from text, a structure that existed a few milliseconds earlier and was deliberately discarded.

That the structure is available at all is a fairly recent accident of history, and it was not built for agents. Through the 1990s a compiler's diagnostic was a printf at the point of failure, which is why the file:line: message convention exists — it was the format Emacs' compilation mode could jump from, and being greppable by an editor was the entire specification. Clang's diagnostics work in the late 2000s pushed the internal representation towards ranges and fix-it hints so that an IDE could underline the exact tokens and offer a one-click repair, and the Language Server Protocol then standardised a diagnostic as an object with a range, a severity, a code and related information, because a server and an editor written by different people had to agree on one. Every checker that speaks LSP had to hold diagnostics as data to do so, and exposing that data on the command line became a small additional step rather than a redesign. The loop is borrowing an interface built for editors.

Most toolchains expose the pre-render form directly. The Rust compiler has emitted JSON diagnostics since 1.0 and the format is documented alongside the rest of the error infrastructure in the Rust Compiler Development Guide; Cargo forwards them when you ask for a machine format:

cargo build --message-format=json 2>/dev/null \
  | jq -c 'select(.reason == "compiler-message")
           | .message
           | {level, code: .code.code, message,
              path: .spans[0].file_name,
              line: .spans[0].line_start,
              col:  .spans[0].column_start,
              suggestion: .spans[0].suggested_replacement}'

{"level":"error","code":"E0308","message":"mismatched types","path":"src/proration.rs","line":88,"col":14,"suggestion":null}
{"level":"error","code":"E0599","message":"no method named `as_cents` found for struct `Money`","path":"src/invoice.rs","line":41,"col":7,"suggestion":"to_cents"}

Every field the loop needs is there and nothing has to be inferred. The suggested_replacement field is worth noticing because it is the compiler telling you what the fix probably is; for a whole class of typo-shaped errors, the harness can apply that itself and never spend a model call at all.

The story is similar elsewhere and not uniform, which is the part that costs a day of integration work. The table below covers the toolchains most repositories in this position actually run.

TypeScript is the awkward one, and since much of this topic uses it as the worked language it is worth showing what the alternative looks like. tsc --pretty false emits exactly one line per diagnostic in the form path(line,col): severity TSxxxx: message, which is stable enough to split on, though multi-line messages and errors with no associated file both break naive splitting. Driving the compiler API gives the objects directly:

import ts from "typescript";
import path from "node:path";

const host = ts.sys as unknown as ts.ParseConfigFileHost;
const config = ts.getParsedCommandLineOfConfigFile(
  "packages/billing/tsconfig.json",
  {},
  host,
);
if (!config) throw new Error("could not read tsconfig");

const program = ts.createProgram(config.fileNames, config.options);
const raw = [
  ...program.getSyntacticDiagnostics(),
  ...program.getSemanticDiagnostics(),
];

const records = raw.map((d) => {
  const pos = d.file
    ? ts.getLineAndCharacterOfPosition(d.file, d.start ?? 0)
    : { line: -1, character: -1 };
  return {
    tool: "tsc",
    severity: d.category === ts.DiagnosticCategory.Error ? "error" : "warning",
    code: `TS${d.code}`,
    path: d.file ? path.relative(process.cwd(), d.file.fileName) : null,
    line: pos.line + 1,
    column: pos.character + 1,
    message: ts.flattenDiagnosticMessageText(d.messageText, " "),
  };
});

process.stdout.write(records.map((r) => JSON.stringify(r)).join("\n"));

Roughly forty lines of harness code replaces a regular expression that will eventually meet a message containing a parenthesis and produce a diagnostic pointing at line zero of a file that does not exist.

The record the rest of the loop routes on#

Different tools give different fields, so the harness normalises everything into one shape before anything downstream sees it. The shape that has to survive is small:

{
  "tool": "tsc",
  "severity": "error",
  "code": "TS2532",
  "path": "packages/billing/src/proration.ts",
  "line": 88,
  "column": 14,
  "end_line": 88,
  "end_column": 31,
  "message": "Object is possibly 'undefined'.",
  "related": [
    {"path": "packages/billing/src/rates.ts", "line": 12,
     "message": "'lookupRate' declared here"}
  ],
  "suggestion": null
}

Paths are repository-relative, because absolute paths differ between the agent's sandbox and CI and the model will otherwise be handed a location it cannot open. Severity is normalised to a small vocabulary rather than passed through, since ESLint says 2, rustc says error, and mypy says error for things that do not block a build. The code is kept as a stable identifier because it is the only field that reliably means the same thing across runs, which matters for counting.

Once diagnostics are records, three things become possible that were not possible with a log, and each of them is a chapter of this topic. The harness can route: send the agent the errors in the files it touched, and know which errors it did not send. It can count: forty-seven errors is a number, and the fact that thirty-one of them carry code TS2339 in eleven files is a shape you can act on. It can attribute: given the same records from a run before the edit, the set difference is exactly what this patch caused. None of those operations exist over a byte stream. They are all one-liners over a list of records.

The volume difference is worth a number, because it changes the arithmetic of the loop as much as the accuracy does. The make build transcript above is abbreviated; the real one on that repository runs about 640 lines and 5,200 tokens once protoc's per-file chatter, the up-to-date notices for thirty-seven unchanged projects, and esbuild's bundle summary are included. The same failure as normalised records is two objects and roughly 180 tokens, and rendered back to the model as two annotated source excerpts with five lines of context each it is under 400. At three attempts per task that is the difference between fifteen thousand tokens of build log per task and about twelve hundred, on top of which the second version does not contain a single line inviting the agent to look at the worker package.

There is a companion signal that structure alone will lose, and it takes one line of harness code to keep. The exit status and the diagnostic list are independent facts, and the interesting cases are the ones where they disagree.

set -o pipefail
node tools/collect-tsc.js > "$out" 2> "$err"; status=$?
count=$(wc -l < "$out")

if [ "$status" -ne 0 ] && [ "$count" -eq 0 ]; then
  # No diagnostics and a non-zero exit: the checker itself failed.
  # OOM kill, missing binary, corrupt tsbuildinfo, network-mounted
  # node_modules gone. Not a patchable defect - halt and report.
  exit 70
fi

A loop that reads only diagnostics will treat an out-of-memory kill as a clean run and merrily accept the patch, and a loop that reads only the exit status will treat a clean compile with a non-zero exit from a later make target as a compile failure. Keeping both, and having a third outcome called harness error that is neither pass nor fail, is what stops the agent being asked to fix an infrastructure problem it cannot see.

Structure is not relevance, and it is not a fix#

Machine-readable output solves exactly one problem: the loop now knows where each complaint points and how severe the tool thinks it is. It does not make the diagnostics true, or relevant to the task, or ordered usefully. A repository with three hundred standing lint warnings emits three hundred perfectly well-formed records, every one of which is about code the current patch never touched. A rename that breaks one shared interface emits sixty perfectly well-formed records, fifty-nine of which will disappear when the first one is fixed. Structure is what makes those two problems addressable; it is not what addresses them.

The other misreading is that a structured diagnostic is a fix. It is a location and a claim about that location, and the mapping from claim to edit is frequently not one-to-one. Object is possibly 'undefined' at proration.ts:88 can be resolved by adding a guard at line 88, by tightening the return type of the function on line 12 that the diagnostic's related field points at, or by adding a non-null assertion that silences the checker and changes nothing. All three make the diagnostic go away, and only two of them are engineering. The compiler's suggested_replacement field, where a tool provides one, is the narrow case where the tool really does know the fix — a misspelled method name, a missing &, an import that needs adding — and treating that as a general property of diagnostics is how loops end up with patches full of assertions.

Finally, this structure is a property of the checkers, not of the build. Shell scripts, Makefiles, Docker builds and bespoke codegen steps emit prose and always will, and a repository whose build is a sequence of thirty shell commands will have a handful of structured checkers embedded in an unstructured pipeline. The practical response is to invoke the checkers directly from the harness rather than through the build wrapper, so that the loop's feedback comes from tsc and eslint and ruff as separate, individually captured runs, and the wrapper build is something that happens later, further from the agent, where prose is survivable.

With diagnostics arriving as records, the next question is which checker should be producing them at all. The type checker and the test suite disagree about what they can see, and the disagreement runs in both directions — which is what the next chapter works through on a fixture where an eighty-eight percent covered, fully green suite says nothing whatsoever about the branch the agent just added.