LLM tracing

Read these first

• Reference architecture
• Agent observability

What you will know

• Emit a span per model call with prompt and completion attached

  Include model identity, parameters, token counts, latency, status, and correlation context so a single call can be reconstructed and compared.

• Trace a full agent trajectory as a span hierarchy

  Nested tool calls, retries, retrieval steps, and model calls should preserve parent-child relationships even when work is asynchronous.

• Redact sensitive content without losing debuggability

  Use field-aware masking, stable placeholders, or controlled references so investigators can follow entities and compare attempts without exposing raw secrets.

• Find a production failure from a trace alone

  The trace should provide enough timing, inputs, outputs, tool results, and errors to identify the failing step without requiring a local reproduction.

• Compare a request log with a trace to isolate an internal model failure

  A request record establishes the external symptom, while correlated spans reveal retries, branching, malformed tool calls, or upstream latency inside the request.

Chapters

1. 1

   A span that records a call is not a span that explains it

   Model name, latency and status prove a call happened; they say nothing about what was asked or returned. The comparable span carries the resolved model version and sampling parameters, the rendered prompt and the completion, input and output token counts kept separate, plus the correlation ids that tie it to a request and a tenant. Those are the fields that let you hold one call against another during a quality regression.

   Read chapter
2. 2

   One span per request works until the second model call

   A single span carrying total latency, total tokens and the model used is honest exactly when the request makes one call and never retries, because then the span and the call are the same thing. Add a retry, a tool step or a retrieval hop and every attribute becomes a sum of things you can no longer separate. The trajectory has to be a tree of nested spans: agent step, tool call, retrieval, model call.

   Read chapter
3. 3

   Orphaned spans are a propagation bug, not a latency bug

   When an agent fans out with asyncio.gather and each tool call turns up as its own root, correctly nested internally, the active span context failed to cross the task boundary. Durations stay right and no dashboard moves, which is why it survives six weeks unnoticed. Threading a trace id through the tool signature as a string looks like a fix and rebuilds half of a propagation mechanism that already exists.

   Read chapter
4. 4

   Splitting 45 seconds into provider, retry and tool time

   The access log has one line — 200, 45.2s, one request id — and it establishes the external symptom and nothing else. The span tree says whether that was a single slow generation or four attempts with your own retry loop eating the timeout budget. It also explains why head sampling loses exactly the requests worth keeping: the keep-or-drop decision is made before anything interesting has happened.

   Read chapter
5. 5

   Every span green and the user still saw a failure

   A trajectory of three model calls and two searches, all ok, ending in "I could not complete that" already tells you the failure is retrieval: the agent looped on empty results and conceded. What it cannot tell you is whether the query was bad or the index was empty, because the search span recorded a result count and not the query. Turning empty results into exceptions to make them alertable destroys a legitimate answer.

   Read chapter
6. 6

   Masking that preserves identity keeps a trajectory followable

   Legal wants PII out of the trace store; support follows the account number the agent looked up across a dozen spans. Field-aware masking that maps each entity to a stable placeholder derived from it satisfies both, because the same value reads the same everywhere. A regex scrubber in the collector fails twice over: it misses free text inside tool results and mangles order ids, and it makes one customer a different blob in every span.

   Read chapter