Summarising a payload with a model

A summariser in a tool path is a cheap model placed between the raw payload and the tool result: the handler fetches the rows, hands them to a small model with instructions about what matters, and returns its output to the agent instead of the rows. It is reached for as a general-purpose way of making a large result small, and it is not that. It is the answer to one specific gap that server-side aggregation leaves open, and it pays for that answer with a property none of the earlier options have.

The gap is real. Consider a checkout service whose requests to an analytics replica fail in bursts of two or three, forty-one times over twenty minutes. A rollup reports them accurately — ConnectionReset, count 41, first at 09:12:44Z, last at 09:33:10Z, ranked fourteenth by frequency — and nobody knows why. The answer sits in the raw lines in a form no group-by could have produced: every reset was immediately preceded, within forty milliseconds and on the same trace id, by a WARN from a vendor telemetry SDK about renegotiating TLS after a cipher downgrade. Neither line means anything alone. The pairing is the finding, and the pairing is a relationship between adjacent records rather than a value in a column, so there was no dimension to aggregate on and no query anyone would have thought to write beforehand. A junior SRE found it in eleven minutes by scrolling; a summariser handed the lines finds it because it reads them and decides what mattered.

That is worth stating as a property of the data rather than as a lucky observation, because it is what marks the edge of the previous chapter's technique. A group-by can only count values that sit in a column. ConnectionReset is such a value and so is the vendor SDK's warning, which is exactly why the rollup reports both of them accurately and reports them separately. What no group-by can report is that the two interleave — that each reset is preceded on the same trace, within forty milliseconds, by one of those warnings — because interleaving is a property of pairs of adjacent rows and there is no column holding it. You could of course add one, by writing an aggregation that looks for this pairing, and then you would own a tool that finds this incident and not the next one. The gap is not that the rollup is coarse. It is that the aggregation has to be written before the question is known.

What sits between the payload and the window#

The naive picture — 41,300 lines in, 400 tokens out — is not implementable as stated, because 41,300 lines is about 3.96 million tokens and no model's context window holds that. What actually runs is a map-reduce: chunk the lines, summarise each chunk independently, then summarise the summaries. With Claude Haiku 4.5 as the summariser and its 200,000-token window, chunks of 1,500 lines come to roughly 144,000 tokens each, leaving room for the instructions and the output, which means 28 map calls and one reduce call for a one-hour window.

Both of those numbers come off the per-line token cost, and the direction the chunk size is chosen in is the part worth noticing. 1,500 lines at 96 tokens a line is 144,000, and that figure was not picked and then checked — it was worked backwards from the window. A 200,000-token window, less the instructions, less the room the output needs, less a margin for lines longer than the median, leaves what a chunk is allowed to be, and 1,500 lines fits inside it. Dividing 41,300 lines by 1,500 then gives a little over twenty-seven, and a partial chunk still costs a call, so it rounds up to the 28. The reduce call reads 28 chunk summaries rather than any lines at all, which is why it can be left out of the cost without distorting it.

import anthropic

client = anthropic.Anthropic()
SUMMARISER = "claude-haiku-4-5"

MAP_PROMPT = """These are consecutive log lines from the checkout service.

Report anything that would help explain a latency or error incident:
errors and their surrounding lines, unusual sequences, values that change,
and any line that appears only once in this chunk.

Quote every line you mention verbatim, including its timestamp and trace id.
Do not paraphrase a log line. If nothing here is notable, say exactly:
NOTHING NOTABLE."""

def summarise_chunk(lines: list[str]) -> str:
    response = client.messages.create(
        model=SUMMARISER,
        max_tokens=1024,
        system=MAP_PROMPT,
        messages=[{"role": "user", "content": "\n".join(lines)}],
    )
    return "".join(b.text for b in response.content if b.type == "text")

Two instructions in that prompt are load-bearing rather than decorative. Requiring verbatim quotes with timestamps is what makes the output evidence instead of hearsay — a summariser that writes "several connection errors around 09:12" has produced something the agent cannot act on and cannot verify, while one that reproduces 09:12:44.118Z trace=b41f9c2a7d ConnectionReset: analytics-ro has produced something an engineer can grep for. Requiring an explicit NOTHING NOTABLE rather than allowing an empty response is what keeps the reduce step from silently treating a failed call as a quiet chunk.

The costs are worth stating exactly, because the summariser is often reached for as though it were free. Twenty-eight map calls over 3.96 million input tokens at $1 per million is $3.96, plus roughly 20,000 output tokens at $5 per million; the reduce call is negligible against that. Run with a concurrency of eight, the map phase takes about 12 seconds of wall clock, and the whole thing lands near $4.00 and 14 seconds per invocation of get_service_logs.

The input side of that is the same 3.96 million as before, and chunking has not reduced it. Every line is still read once by some call, so the map phase reads the hour in full and the chunk size only decides how the reading is divided. 3.96 million at $1 per million is the $3.96. The output side is 20,000 tokens at $5 per million, which is a tenth of a dollar, and adding the two gives the $4.00. The wall clock behaves the same way: 28 calls run eight at a time is four waves, so the 12 seconds is four summarisation calls in sequence rather than twenty-eight, and the couple of seconds beyond it to 14 is the reduce call and the fetch. Raising the concurrency shortens the map phase and does not change the cost at all, since the cost is the number of tokens read and that is fixed by the window.

The last two rows are the whole argument, and they are why the summariser is not simply a better truncation. Everything above them is a trade you can reason about with a spreadsheet.

The loss is not the same loss twice#

Replaying one archived log window through the same summariser twice produces two different summaries, and on the window behind a checkout latency spike the difference was the causal line: WARN config_reload: pool 'orders-ro' max_size 8 (was 32) source=canary-overlay at 14:05:02Z. On one run it was there and an agent given the summary reached the right answer immediately. On another it was not, and the agent produced a confident, well-structured account naming the wrong cause.

The line had not been truncated away, because the chunk containing it was well within the window. It had not been aggregated away, because no aggregation ran. It was read by the summariser, judged less notable than the 47 pool timeouts it caused, and left out of a 380-token summary of a chunk that contained 1,500 lines. Nothing in either trace distinguishes the two runs: same tool, same input, same model, same prompt, the same temperature setting that governs how much randomness the sampler is allowed.

Turning that setting down is the first move anyone proposes, and it is worth being exact about what it would buy, because the answer is not nothing and it is also not the thing wanted. Lowering the temperature narrows how far the sampler may wander, so repeated runs over identical input come to resemble each other much more closely. What it cannot do is produce the boundary. The summariser would still be deciding, on its own judgement, which of 1,500 lines are notable, and a line judged less notable than the 47 pool timeouts it caused is dropped at a low temperature exactly as it is at a high one — dropped every time now, rather than sometimes. That removes the confusing variance and leaves the defect where it was. In either case nothing in the system emits a record of what was left out, so there is still nothing to put in the payload. Determinism is a property of the process; stating the loss requires a record of the loss, and no sampler setting produces one.

Non-determinism of that kind is the property that makes a summariser categorically different from the two bounded results earlier in this topic, rather than merely worse on some axis. A capped result loses rows past position 25 — a boundary you chose, can state in the payload, and can reason about, which is what makes an incompleteness contract possible to write. A rollup loses everything outside the aggregation you specified, which is a boundary you also chose and can describe in a basis block. A summariser's boundary is decided per call, inside a model, and is not available to be written down. You can tell the agent that the result was summarised; you cannot tell it what the summary omitted, because nothing in the system knows.

A summariser loses information invisibly and inconsistently, and those are two distinct defects wearing one name. Separating them is what makes each of them addressable. The loss is invisible: nothing in the result marks what is absent, so no consumer downstream can compensate for it. The loss is also inconsistent: it falls somewhere different on each call, so a run that came back complete is not evidence about the next one. Invisibility is what a kept copy of the raw payload addresses, by leaving something to compare a summary against. Inconsistency is what repetition addresses, by turning a single run from a verdict into a sample. Fixing either on its own leaves the other exactly where it was, which is why the two mitigations below arrive together.

Catching a loss that leaves no trace#

A summariser can go three weeks without anyone noticing it has dropped something load-bearing, because the tool result contains no evidence of what was discarded and there is nothing on disk to compare it against. Both are fixable, and together they turn a mystery into an ordinary measurable defect.

The first fix is to keep the raw payload and give the result a handle to it. The 41,300 lines cost about 8 MB in object storage, which is negligible, and writing them under a key before summarising means the summary is a lossy view of something that still exists rather than the only surviving artifact.

The grep_raw call at the bottom is the summariser's version of a rollup's drill-down, and it changes what a partial summary costs. An agent that has been told the answer might not be in the summary, and that it can search the raw window for a term it forms a hypothesis about, recovers from a dropped line in one extra call and about 200 tokens. Without it, the summary is the entire world and a dropped line is unrecoverable regardless of how good the agent's reasoning is.

The second fix is a regression suite built from real incidents, where each case pins a line the summary must contain.

NEEDLES = [
    ("logs/2026-03-17T14.jsonl", "config_reload: pool 'orders-ro' max_size 8"),
    ("logs/2026-03-09T09.jsonl", "cipher downgrade, renegotiating"),
    ("logs/2026-02-28T03.jsonl", "FATAL: schema version mismatch"),
]

def retention_rate(path: str, needle: str, runs: int = 20) -> float:
    lines = open(path).read().splitlines()
    hits = sum(needle in summarise_window(lines) for _ in range(runs))
    return hits / runs

Running each case twenty times is the part people skip, and it is the part that matters, because a single run of a non-deterministic component measures nothing.

Twenty is not a magic number, and the reasoning that would pick any other one is the same. What the suite estimates is a rate, and the precision of an estimated rate improves with the square root of the number of runs rather than in proportion to it, so doubling the runs buys rather less than twice the confidence and the returns flatten quickly. Twenty runs separates a case that almost always survives from one that survives most of the time, which is the distinction that decides whether to ship. It does not separate a case that would survive nineteen times in twenty from one that would survive every time, and a suite result should not be read as though it did: a case that passed twenty times has a small failure rate rather than none.

The decision lives at the tool boundary#

When an agent names the wrong cause from a summary, the wrong answer is the tool's bug in the same sense that a capped result reporting an empty search is: if the result is indistinguishable from the result you would return when the causal line genuinely was not there, no consumer can tell the two apart. The agent reasoned correctly from what it was given.

What is different here is where the decision sits. Nobody chose to drop that line; the choice was made inside a model call that nobody was watching, on a boundary nobody had specified. But the decision to put a non-deterministic component in the path between the data and the model was made deliberately, by engineers, in the tool's implementation — and that is the decision a review should be about. It belongs in the same place as the page size, the cap and the aggregation: in the tool's design, owned by the team that ships the tool, tested by them, and declared in the result, so that an agent holding a summary knows it is holding a lossy view produced by a judgment it cannot inspect.

The compounding version is worse than the single-call version. In a session that pages through six windows of logs, each page passes through an independent summariser call, so the probability that at least one page drops something load-bearing rises with the number of pages, and the agent's confidence does not fall to match. The failure is uncorrelated across pages, which means it is also invisible in aggregate: your eval suite run once looks fine, run again looks fine, and the production failure rate is a few percent that never reproduces on demand.

The shape of that rise deserves working out, because "rises with the number of pages" sounds gentler than it is. The calls are independent, so the chance that every page comes through intact is the per-page chance multiplied by itself once for each page. Suppose each of the six windows carried a line as easy to lose as the one this summariser was measured on — fourteen runs in twenty, seven in ten. Multiplying seven-tenths by itself six times leaves the session at something near one chance in ten of every page having kept what mattered. A figure that reads as a respectable pass rate on a single call reads as a near-certainty of loss across a session, and the session reports neither the loss nor which page it happened on.

A summariser earns its place when the salient content of a payload cannot be specified in advance, which is a genuine and recurring case, and it earns it at about $4.00, fourteen seconds, and a retention rate you have to measure rather than assume. Everything that makes the earlier options defensible — a stated boundary, a payload that describes its own loss, the same answer twice — is what you are trading away, so the mitigations are not optional extras: the raw window behind a handle, a search call into it, a needle suite run twenty times per case, and a result that says it passed through a summariser.

Stated that way, the summariser stops being an alternative to the earlier techniques and becomes the fourth option on the same shelf, with the widest coverage and the worst guarantees. Offset paging, cursor paging, bounded truncation and pre-summary each fail differently, and the payload in front of you decides which failure is tolerable.