Walk the trace forward from the source page, because the prompt log lies last

A trace is the record of what one request did at every stage between a document and an answer. It is not a log of errors, because the failure this chapter is about raises none. It is a record of what each stage was handed and what it produced, kept so that a later reader can check whether the evidence survived each step.

The occasion for one is a ticket. It says an assistant reported EMEA operating margin at 31% for Q2, and the chart in a quarterly report shows 24%. The post-mortem thread opens on a Tuesday. The parsing owner points out that the page rendered fine and OCR ran clean. The retrieval owner points out that the correct page was returned at rank 2, which is in the trace. The applied-AI engineer points out that the model was given the page and got the number wrong, which is a model problem. Everyone is looking at a different artefact, each artefact supports its owner's position, and nobody can produce the one that connects them. The thread runs until the following Monday and closes with "increase k from 10 to 30", which ships.

The reason it took six days is not that anyone was unreasonable. It is that the team defaulted to the artefact that is easiest to add and least diagnostic: the prompt log.

The prompt is the last thing in the chain and the least informative#

Full prompt logging is the standard first response to "we can't tell what happened", and it produces a specific kind of misleading evidence. By the time the prompt exists, every upstream stage has already had its chance to discard the answer, and none of them leave a mark. A chart whose values were flattened into axis labels arrives in the prompt as ordinary text. A crop that clipped the y-axis arrives as a perfectly valid PNG. A table whose merged header shifted every value by one column arrives as a clean, well-formed table. The prompt shows what the model was given and says nothing about what it should have been given, so the only actor visible in it is the model, and the model is the one stage that definitely operated on whatever it received.

The property that makes a prompt log feel like proof is exactly the property that disqualifies it. A full log is complete by construction — every token, every image, nothing sampled away — and completeness is what people are reaching for when they add one. But that completeness holds steady whatever happened upstream. It reads the same when the crop was perfect and when the crop cut off half the chart, because a bad crop is still a complete, valid, well-formed image. A measurement that does not move when the thing you care about moves cannot be evidence about that thing, however trustworthy it looks.

That asymmetry is what keeps the post-mortem stuck. A full prompt log is an argument for the applied-AI engineer's opponents and not an argument the applied-AI engineer can rebut, because rebutting it requires an artefact from three stages earlier that does not exist. Adding more prompt logging makes the thread longer rather than shorter.

The alternative is to start at the source page and walk forward, checking at each stage whether the evidence is still present and still correct. Every step then rules something in or out, and the walk terminates at the first stage where the evidence is gone or wrong. That stage is the defect, and every stage after it was innocent.

The direction matters for a reason worth spelling out. Each stage's correctness is a claim about the relation between its input and its output, so it can only be judged when both are in hand. Walking forward, every stage's input is the output you just verified, and each check is decidable. Walking backward, the first unverified input is the last stage's, and a stage whose input is unknown cannot be cleared or convicted. The uncertainty then propagates: if stage four cannot be judged, stage five's input is an unknown too, and so on down the chain. One missing artefact does not blur one stage; it makes everything downstream of it unjudgeable.

The artefacts that have to exist before the incident#

Each arrow is labelled with what has to be recorded for the step to be checkable. If any label is missing, the walk stops there and the remaining stages become unfalsifiable — which is how a post-mortem turns into a negotiation between owners.

In practice that is one record per request, keyed by request id:

{
  "request_id": "req_9f3c21",
  "query": "What was EMEA operating margin in Q2 FY24?",
  "pinned": {
    "index_version": "filings-2024-09-14",
    "encoder": "page-v3.2", "detector": "layout-v4", "reranker": "xmod-v2",
    "render": {"long_edge_px": 2048, "dpi": 200}
  },
  "pools": {
    "bm25":       [{"unit_id": "10k_fy24#p88",       "rank": 4, "score": 8.1}],
    "dense_text": [{"unit_id": "10k_fy24#p88",       "rank": 7, "score": 0.66}],
    "page_image": [{"unit_id": "10k_fy24#p88",       "rank": 2, "score": 0.71}],
    "figure":     [{"unit_id": "10k_fy24#p88#fig1",  "rank": 1, "score": 0.83}]
  },
  "fused": [
    {"group": "10k_fy24#p88", "rank": 1, "members": ["#p88", "#p88#fig1"], "rrf": 0.0491}
  ],
  "slots": [
    {"unit_id": "10k_fy24#p88#fig1", "type": "image", "long_edge_px": 2048,
     "bbox_norm": [0.52, 0.31, 0.95, 0.58], "tokens": 1240}
  ],
  "answer": {
    "text": "EMEA operating margin was 31% in Q2 FY24.",
    "citations": [{"evidence": 1, "region": [0.10, 0.14, 0.88, 0.79]}]
  }
}

The pinned block is what makes the trace replayable rather than merely readable. Without it, the first hour of every investigation goes to establishing which index the request hit and whether the detector has been redeployed since, and "I can't reproduce it on the current index" ends the investigation without resolving anything. That sentence is ambiguous between two states of the world — the defect was fixed, and the system under test is no longer the system that failed — and nothing in an unpinned trace can separate them.

One feature of the ticket narrows the search before the walk begins, and it is worth reading off the record deliberately. The answer says 31%. The chart shows EMEA at 24% and APAC at 31%. So the reported figure is a value that is genuinely on the page, attached to the wrong label. That is not what an invented number looks like; an invented number is a number the page does not contain. A value correctly read and wrongly attributed points at a stage that could separate a value from the label belonging to it, and exactly one stage in the diagram above does that — the one that decides what region of the page becomes a unit.

Walking it#

Step 1 — the source page and its render. Open 10k_fy24#p88 at 2048 px and look at it. This rules out a whole family of defects that have nothing to do with retrieval: the wrong page number, a page rotated 90°, a render that came out blank because an embedded font failed to load, a page that is a scan at 96 dpi. Here the render is fine and the chart is legible, showing EMEA at 24% and APAC at 31%.

Step 2 — parser and detector output. Layout detection found one figure on the page at score 0.88, and cell extraction is not involved because the evidence is a chart rather than a table. Nothing wrong yet, and this step is where the trace would have ended if the detector had missed the figure entirely — a case that presents as "the system never finds anything on this page" rather than as a wrong number.

That score is worth not over-reading, because it is the second quantity in this chapter that stays put while the thing that matters varies. A detector's score is a confidence that a figure is present at all. It says nothing about whether the box drawn around it is the right box. Presence and extent are two different properties reported by one number, and only the first of them is in it, which is why step 3 exists as a separate step rather than as a footnote to step 2.

Step 3 — the indexed unit. The record says the figure unit's bbox_norm is [0.52, 0.31, 0.95, 0.58]. Re-crop the page render to those coordinates and look at the result. The crop contains five bars and their value labels and does not contain the y-axis, the axis title, or the category labels along the bottom — the detector's box hugged the plot area, and chapter 4's padding (the margin added outward from a detector's box before the crop becomes an indexed unit) was configured at 6% horizontally, which was not enough on this template because the axis labels sit 9% of page width to the left of the plot.

That comparison is the whole finding, and it is a comparison of two numbers rather than a judgement about an image: the padding has to reach at least as far out as the labels sit, and six does not reach nine. The box's left edge is a little past halfway across the page and the labels sit nine hundredths of the page width further left, so six hundredths of padding stops short of them — short by more still if that six is a fraction of the box rather than of the page, since the box is under half the page wide. The crop excludes the labels by construction under either reading, on every page built from this template, whatever the render looked like.

That is the defect, found at step 3, and the walk can stop. What the model received was five unlabelled bars in an unlabelled chart, and it attributed the 31% bar to EMEA because EMEA appeared in the surrounding text candidate. Everything downstream — fusion, slot allocation, the model — behaved correctly on the input it was given.

The finding is confirmed by two facts that arrived by different routes and agree, which is what makes it a diagnosis rather than a hypothesis. The record's geometry says the labels are outside the crop. The ticket's symptom says a value present on the page was attached to the wrong name. Neither observation is derived from the other — one is a measurement on a stored bounding box, the other a comparison of two strings — and a missing label column is the only condition that produces both. What the agreement licenses is stopping the walk. What it does not license is a claim that stages four through seven are correct in general; it says only that they cannot explain this symptom.

Two things are worth noticing about how the remaining steps would have gone. Step 4 would have shown the figure unit at rank 1 in its own pool, exonerating retrieval, which is exactly what the retrieval owner had been saying all week and could not prove was the whole story. Step 5 would have shown the dedup group merging the page and figure units and promoting them to fused rank 1, exonerating fusion. Step 6 would have shown the figure crop taking an image slot at full resolution, exonerating the slot budget. And step 7, the prompt, would have shown a legitimate-looking image of a bar chart, which is precisely why starting there was useless: the crop's defect is invisible unless you know what the uncropped page contained.

Step 4 also disposes of the week's other constant. Rank was the retrieval owner's defence, and rank is a number that holds still while the thing that matters varies. The figure unit sat at rank 1 in its own pool and at fused rank 1 in the merged list, and the evidence was missing anyway. A perfect rank on a unit whose crop dropped the labels is indistinguishable, from the rank column alone, from a perfect rank on a unit that carries everything.

Elapsed time for the walk, with the artefacts in place: about forty minutes, most of it spent re-rendering the crop and comparing it to the page. That figure and the six days are not two measurements of the same quantity. Forty minutes is hands-on work by one person; six days is how long a thread stayed open while nobody could produce a decisive artefact. What the trace buys is not a faster investigation so much as an investigation that can end.

The fix that follows is specific and testable. Padding goes from 6% to 12% horizontally, or — better — the crop is expanded to include the nearest axis-label text blocks rather than a fixed fraction. Then the figure-slice number from chapter 7's slice table — the score on questions whose evidence lives only in a figure or table image — gets re-measured, and the fix is accepted if that slice moves and the text slice does not.

Naming which of those two is a requirement and which is a design decision is worth a moment, because the padding number depends on something the configuration does not state. The requirement is that the crop reach the labels. Twelve percent satisfies it if the padding is a fraction of page width, since twelve exceeds nine with room to spare. If the padding is instead a fraction of the box's own width — and this box is under half the page wide — then twelve percent of it is still under nine percent of the page, and the fix does not reach the labels at all. A percentage with an unstated denominator is the kind of error that is a word rather than a digit, and it is why the second option is better: anchoring the crop to the nearest axis-label text blocks satisfies the requirement without depending on which denominator anyone meant.

Why a proposed fix has to name a stage#

The generalisable rule for sign-off is that a proposed fix has to name the stage it operates on and predict which slice moves. "Raise k" names no stage — it is a knob adjacent to the problem rather than a change to the thing that broke. The same test rejects most of the other reflexive fixes in this area: raising resolution when the defect is a crop boundary, swapping encoders when the defect is detector recall, adding prompt instructions when the defect is that the evidence never arrived.

Raising k is tempting for a reason worth naming rather than dismissing. It is a real defence against a real failure — the right unit ranked below the cutoff — and most teams have seen it work, because that failure is common and the knob usually needs no redeploy of anything. What makes it wrong here is that the failure it defends against is not the failure that occurred, and the trace says so in one line: fused rank 1. A defence aimed at a failure mode that the record rules out is not a cautious choice, it is a change with cost and no mechanism.

The cost of keeping traces, and what to keep#

Full traces are not free, and the expensive part is the images. A trace with four 2048 px PNGs attached is around 2.4 MB, and at 40,000 queries a month that is 96 GB monthly of trace storage, most of which is never read.

The workable arrangement stores the record for every request and the bytes for almost none. The JSON above is about 4 KB, which is 160 MB a month and trivially retained for a year. The choice between those two turns out not to depend on how much traffic you have. Both monthly totals are the same query count multiplied by a per-request size, so dividing one by the other cancels the query count and leaves the ratio of the two artefacts — a record against a set of images, some six hundred times apart. That number is a property of the artefacts rather than of the workload, which is why the decision can be made once, at design time, without a traffic forecast. The cancellation holds only while the per-request sizes hold: a workload with many more candidates per query, or larger renders, moves the image side, whereas the record grows only with how many pools and slots are configured.

Images are reconstructed on demand by replaying from the pinned versions: the page render is deterministic given the document, page number and render parameters, and the crop is deterministic given the render and the recorded bbox_norm. That works exactly as long as the pinned artefacts remain available, which turns index and detector retention into an incident-response requirement rather than a storage-cost line — keeping the previous two index versions and every detector checkpoint is the concrete form of that requirement.

Replay also has to be one command rather than a notebook someone assembles under pressure. The gap between "we have the artefacts" and "we resolved it in an hour" is almost entirely tooling: a trace replay req_9f3c21 --stage 3 that pins the recorded versions, re-renders the page, draws the recorded boxes on it, and writes the crop to a file turns step 3 into thirty seconds of looking rather than an afternoon of reconstructing render parameters from a config repository. Build it the first time you walk a trace by hand, while you still remember which lookups were tedious. The same command doubles as the reproduction step in the bug report, so the next person does not start from prose.

Sample a small fraction — one or two percent — with images attached, for the cases where replay is not deterministic and for spot-checking that replay still matches what actually shipped. Determinism is an assumption rather than a guarantee, and the sample is the only instrument that can test it: stored bytes and replayed bytes are two measurements of one thing by different routes. When a replayed crop and a stored crop disagree, that discrepancy is itself the most valuable finding in the system, because it means something in the render or detector path changed without anyone versioning it. A sample of that size finds a change touching most requests within a day and finds one touching a rare template slowly or never, so it is a smoke alarm rather than a proof.

Everything in this topic has been about making one class of evidence — the kind that lives in pixels — reachable, sendable, measurable and traceable. The pieces now fit together in a specific order: choose the indexed unit for what you need to cite, keep enough resolution that small type survives the trip, fuse on ranks with representations of the same page grouped, measure against distractors that look alike, and keep the artefacts that let you walk a failure forward before you need them. The last of those is the one that has to be built while nothing is wrong, since every artefact the walk needs is one that had to exist before the request that failed.

What remains is the part no chapter can do for you, which is running it on your own corpus. Start with the slice proportions from chapter 7, because they determine whether any of this is worth building for your documents; the answer for a contracts archive is different from the answer for a filings archive, and both are knowable in an afternoon of hand-labelling. Then build the smallest adversarial question set that can tell resemblance from evidence, and let it referee every decision that follows.