Agent observability

Read these first

• The agent loop implemented from scratch
• Cost tracking per request, user, feature and tenant

What you will know

• Trace a full trajectory with a span per step

  Without parent-child relationships, retries and tool calls collapse into an unreadable transcript, while payloads may require deliberate redaction.

• Attribute cost to individual steps

  Token, tool, and retry costs often sit outside the final model call, so per-step totals must reconcile to the complete run.

• Replay a failed run from its trace

  Replay is only useful when nondeterministic model and tool responses are captured or stubbed; otherwise the supposedly identical run diverges.

• Find the step where a run went wrong, from the trace alone

  The first visible error may be downstream of a bad tool result or earlier state mutation, so diagnosis must follow causal links rather than the last failure.

• Compare a conventional log stream with a structured trace when diagnosing an injected retry failure

  Flat records discard parent-child relationships and per-attempt state, making a retry look like a duplicate or an unrelated error.

Chapters

1. 1

   Spans keep the parent-child structure that log lines discard

   Inject a failure that causes a retry, then ask each system how many attempts there were. The flat stream shows two similar records and cannot say whether they are one call retried or two unrelated calls; the span hierarchy nests the second attempt under the step that owns it. Start here, with a span per step and per tool call, because everything later in this topic reads that structure.

   Read chapter
2. 2

   Redaction is decided before the first payload lands in a trace

   Full tool payloads carry customer data into a system with far wider access than the production database, so what gets stored is settled now, before cost attribution, causal walking and replay are designed on top of an assumption the policy may forbid. Work the constraint through: which fields survive, which become structural metadata such as lengths, types and identifiers, and which capability dies — faithful replay is the first casualty, and the backwards causal walk degrades with it. Hashing the content passes the review and destroys the field's diagnostic value entirely; a short-retention, tighter-access full-fidelity tier is the compromise worth arguing for.

   Read chapter
3. 3

   Per-step costs that do not reconcile to the bill are wrong

   When your instrumented totals come to a third of what the provider charges for the same runs, the missing two-thirds usually sit where the final model call never looks: retried and failed attempts nobody recorded, and model calls made inside tools. Attribute cost at the step, then reconcile the sum against the invoice, because a per-step figure that does not add up to the run is decoration.

   Read chapter
4. 4

   A p95 eight times the median is a step count, not a price

   Given traces and nothing else, the instinct is to sort model calls by unit cost, and that almost never finds it. Compare step counts and span shapes across the expensive tail against a median run and you tend to find loops, retry storms, and a tool whose empty result sends the agent round again — a small number of runs doing very many ordinarily priced things.

   Read chapter
5. 5

   The first visible error is rarely where the run went wrong

   A validation failure at step nine means a bad value entered the trajectory earlier and travelled there. Diagnosis follows causal links backwards — which span produced that argument, which tool result it came from, what state mutation preceded it — instead of reading the last stack trace. Which links a trace has to carry for that walk to be possible at all is the design question here.

   Read chapter
6. 6

   Replay without captured responses is only a second run

   A replayed trajectory that takes a different path and succeeds is not a broken feature; it is a run whose model and tool responses were never recorded, so nothing was replayed. Pinning the model version and setting temperature to zero does not fix it, since providers vary and tool responses diverge far more. Decide what you capture or stub, and whether determinism or fidelity is the goal — under the redaction rule set in chapter two, which is what forces the answer here: replay over structural metadata with stubbed content, full-fidelity capture confined to a short-retention tier, or an explicit decision that some classes of run are simply not replayable.

   Read chapter