Chapter 7 of 8 · 15 min read

Step two cannot fetch what step one's condensation threw away

On this page

The compression was tuned for a question it never saw
What the boundary has to specify
Handling the intermediate that arrives wrong
Giving step two the source does not make step two read it
Lossy is not malformed

A stage that hands forward a condensation of its input has made a choice on behalf of every stage after it. Summarisation is a lossy encoding — one that discards material it has no way of restoring later — and its loss function is "what a reader would want in general". A downstream consumer with a particular need is relying on a compression optimised for someone else, and no amount of prompt work on the consumer brings back what the compressor dropped.

That is why a two-stage pipeline can score below the single prompt it replaced while every component looks fine on its own terms. The summaries read well, the answering step answers the question it was asked correctly, and the information left the system at the boundary. A handoff contract is the set of decisions that stops this happening, and it is a design activity rather than something to discover in an incident review.

The compression was tuned for a question it never saw

Pull failures and read the intermediate next to the question, and the pattern arrives immediately.

text

Q: "How many days do I have to appeal a denied claim?"

Source (§7.3): "A member may request reconsideration of an adverse benefit
determination within 180 calendar days of receiving written notice of denial.
Expedited appeals for urgent care must be filed within 72 hours."

Step 1 summary (excerpt): "The plan provides an appeals process for denied
claims, with expedited handling available for urgent cases. Members should
file promptly after receiving notice."

Step 2 answer: "You should file your appeal promptly after receiving the
denial notice. For urgent care, an expedited appeal is available."

Step two did nothing wrong. Given a summary that says "promptly", the answer it produced is the correct answer to the question it was asked, and it has no mechanism for knowing that 180 and 72 ever existed. Step one was asked to summarise a document without being told what the summary was for, so it had to compress for the average of every possible question, and a good general-purpose summary of a policy document keeps the structure and drops the specific numbers. Taking 8,300 tokens down to 400 is a 95% reduction with no knowledge of which 5% matters.

Why the numbers in particular are the first thing to go is worth one more step, because it is not an accident of this summariser. A summary is chosen to be useful to a reader whose question is unknown, so it keeps what most questions would want. What most questions want is the shape: that there is an appeals process, that urgent cases are handled differently, that a deadline exists. A specific figure is the opposite kind of content — it answers exactly one question and is dead weight for every other, and it cannot be inferred from anything else in the document if it is dropped. So a compressor optimising for the average question systematically discards precisely the material that a single specific question needs, and it does this most reliably when it is working well. The failing intermediate above is not a bad summary. It is a good summary of the wrong thing.

The reduction figure follows from the two token counts: 400 out of 8,300 is under one part in twenty, so nineteen parts in twenty were discarded. The choice of which nineteen was made by a step that had never seen the question.

A one-line change tests the diagnosis, and it is the cheapest thing to try before redesigning anything:

text

Summarise the sections of this policy document that bear on the member's
question. Preserve every number, deadline, dollar amount and named condition
verbatim. Question: {question}

What the boundary has to specify

Question-conditioning treats the symptom. The design decision underneath is what the intermediate is for, and four questions have to be answered explicitly.

Contract dimension	The failing design	A design that holds
What the intermediate carries	Free prose, unspecified length	Typed fields plus provenance: section ids and verbatim quotes
Whether it is validated	No; whatever comes back is passed on	JSON Schema, validated before step two runs, retried on failure
Whether step two sees the source	No	Yes — the intermediate points into a document step two also holds
What step two does with a bad intermediate	Undefined; it answers anyway	Defined: abstain — return no answer and say why — escalate, or fall back to the single-call path

The third row is the one that changes the accuracy number, so it is worth being concrete about what it means. Step two does not need the intermediate to contain everything; it needs the intermediate to be a pointer into something it can still read. Restructured that way, step one's job stops being compression and becomes selection with evidence:

json

{
  "relevant_sections": [
    {
      "section_id": "7.3",
      "why": "states appeal deadlines",
      "quote": "within 180 calendar days of receiving written notice of denial"
    }
  ],
  "insufficient_evidence": false
}

Written as a schema the intermediate is enforced rather than requested, and the constraints that matter are not the obvious ones:

json

{
  "name": "policy_evidence",
  "schema": {
    "type": "object",
    "properties": {
      "relevant_sections": {
        "type": "array",
        "maxItems": 4,
        "items": {
          "type": "object",
          "properties": {
            "section_id": {"type": "string", "enum": ["1.1", "1.2", "7.3"]},
            "why":        {"type": "string", "maxLength": 120},
            "quote":      {"type": "string", "minLength": 20, "maxLength": 400}
          },
          "required": ["section_id", "why", "quote"],
          "additionalProperties": false
        }
      },
      "insufficient_evidence": {"type": "boolean"}
    },
    "required": ["relevant_sections", "insufficient_evidence"],
    "additionalProperties": false
  }
}

The enum on section_id is generated per request from the document's own table of contents, abbreviated above, which moves the check from validation into decoding: the model cannot emit a section number the document does not have, because those tokens are not available to it. That change alone removed the 0.9% of requests where step one cited an invented section, at the cost of compiling a schema per document, which measured at about 8 ms and is cached on document hash. The maxItems cap matters for a different reason — an unbounded selection step drifts towards returning most of the document, and capping it at four keeps step two's context predictable in size and cost.

The insufficient_evidence flag is not decoration. It is the field that gives step two something to do other than answer, and without an explicit way for step one to say "I found nothing", a selection step will always return its best guess.

Any abstention field needs calibrating against the queue it feeds, because the cost of the flag is entirely downstream of the model. A flag that fires too readily converts answerable questions into human work at whatever a human minute costs; one that fires too rarely leaves the confident wrong answers the flag existed to prevent. Neither error is visible in an accuracy number, because an abstention is scored as neither right nor wrong, so the measurement to make is the false-abstention rate — sample the flagged questions, have a reviewer say which of them the pipeline could in fact have answered, and multiply through by traffic. That converts a threshold into a monthly figure in hours, which is the unit the organisation absorbing the escalations reasons in.

The conversion is four multiplications long and every one of them is the kind a review can check on a whiteboard, which is what makes it usable in a meeting with people who do not read code. Start with the flag's fire rate. Sample the flagged questions and have a reviewer say which of them genuinely had no answer; the remainder is the false-abstention share of the flagged population. Multiply that share by the fire rate and you have the false-abstention share of all traffic, which is a much smaller number and the one that matters. Multiply that by monthly volume for a count of unnecessary escalations, and by the handling time for a figure in hours. Each step changes the unit, and only the last unit is one anybody outside engineering can act on.

Handling the intermediate that arrives wrong

Every handoff needs a defined behaviour for a malformed intermediate, because "step two answers anyway" is the default and it is the worst option available. Validation gives you a branch point:

Two things in that diagram are load-bearing beyond the validation itself. The fallback path keeps the original single-call prompt alive as a degraded mode, which costs you the maintenance of a prompt you hoped to delete and buys you a pipeline that cannot fail closed on a schema error; it fired on 1.1% of requests in the first month. And the check that the quotes actually appear in the source document is a cheap string containment test that catches step one inventing a quote, which happened on 0.6% of requests and is the failure that would otherwise produce a confidently wrong answer with a fabricated citation attached to it.

A health insurer's member-support tool answers questions about plan policies. Each policy document runs about 6,000 words, or roughly 8,300 tokens, and the original design put the whole thing in one prompt with the member's question and got 88% on a 1,200-question eval set at a 1.9 s p95. The pipeline that replaced it summarised the document in step one and answered from that summary in step two. It scored 71%.

Seventeen points below the thing it replaced, from a change that added a call and was expected to help. The eval sets were identical, the model was the same, and the summaries read well — an editor asked to grade them on their own terms gave them 4.4 out of 5. Question-conditioning the summary took the pipeline from 71% to 84%, and the remaining four points below the mega-prompt were cases where step one judged a section irrelevant and it was not. Rebuilding the intermediate as selection with quotes, with step two holding the document as well, scored 89% — one point above the mega-prompt, which at n=1,200 and a standard error near 0.9 points is a tie rather than a win.

The abstention flag had to be calibrated against the queue it feeds, because its cost is entirely downstream of the model. It fired on 6.2% of questions; a reviewer read 200 of those and agreed that 142 genuinely had no answer in the document, so about 1.8% of all traffic was being escalated when the pipeline could have answered it. At 400,000 questions a month that is 7,200 unnecessary escalations at roughly four minutes of agent handling each, or about 480 agent-hours a month spent on questions the system knew the answer to. Tuning it was a conversation with the support organisation about which error they would rather absorb, not a threshold an engineer sets from the score distribution.

Laid out against each other on the same eval set, at 400,000 questions a month and $3 per million input tokens, the five designs this endpoint has worn are:

Design	Accuracy	p95	Input tokens	Model calls	Input spend / month
Single mega-prompt	88%	1.9 s	8,400	1	$10,080
Summarise, then answer from the summary	71%	3.3 s	8,900	2	$10,680
Question-conditioned summary, then answer	84%	3.4 s	8,950	2	$10,740
Select with quotes, then answer with the source	89%	3.1 s	17,100	2	$20,520
BM25 selection, then the original prompt	90%	1.2 s	2,100	1	$2,520

The fourth row is the design the contract work produced, and it is the most expensive row in the table by a factor of eight against the row that beats it, because it is the only one that sends the whole document twice.

The spend column is one multiplication repeated, which is why these rows can be compared on their token counts alone. Input tokens times 400,000 questions times $3 per million produces each figure: the single prompt's 8,400 tokens give the $10,080, the contract design's 17,100 give the $20,520, and the BM25 design's 2,100 give the $2,520. Volume and price are identical in every row, so the ratio between any two rows is exactly the ratio of their token counts — 17,100 against 2,100 is the factor of eight, arrived at without consulting the money column at all.

The second and third rows are cheap for the same reason they are inaccurate, since a 400-token summary is a small thing to send and a small thing to reason from. Read as a whole, the contract work bought eighteen points over the naive split and one point over the design it was meant to replace, at about $10,000 a month and 1.2 s of p95 — and the bottom row, which needs no second model call at all, beat it on every column. The endpoint shipped the bottom row. :::

Giving step two the source does not make step two read it

The third row of the contract table says step two holds the document as well as the intermediate, and the natural reading of that is more generous than the measurement. Handing step two the source creates a route back to the detail step one dropped, and it does not create any inclination to take that route.

The failure that survives the containment check is the one where step one returns a quote that is real, verbatim and from the wrong section. Containment passes, because the string is genuinely in the document. On 120 requests constructed that way — the quote pulled from §7.9 on grievances when the question was about appeals under §7.3, both sections present in the document step two held — step two answered from the handed quote in 104 cases and consulted the document for itself in 16. The intermediate was being read as the answer rather than as a pointer to where the answer might be, which is the same deference a relay stage produces when it turns a draft into a premise.

Instruction moved it a long way and did not eliminate it. Rewording step two's prompt to say that the quotes indicate where to look and the document is authoritative, and requiring the answer to cite a section it verified itself, took the 104 down to 61. Adding a cheap consistency check — does the cited section actually contain the numbers the answer states — caught 29 of the remaining 61 at the cost of one extra string scan per request. The residual 32 out of 120 is the honest floor on this design, and the way to keep it small is to keep step one's selection accurate rather than to keep instructing step two to distrust it.

Those three interventions compose by subtraction, and tracking the remainder is more instructive than any single rung of the ladder. Of the 120 constructed cases, step two deferred to the handed quote in 104. Rewording the prompt removed a little over forty of those. A consistency check on the numbers removed twenty-nine of what was left. What survives is the 32 that neither instruction nor a mechanical check reaches, and those fail invisibly: the answer is fluent, it cites a real section, and the quote it rests on genuinely appears in the document, so there is nothing for a validator to catch and nothing for a reader to notice. Each rung costs less than the one before it and removes less than the one before it, and after the last rung there is no further rung of the same kind — the only lever left is making step one's selection right to begin with.

Lossy is not malformed

The temptation, once accuracy drops at a boundary, is to reach for stricter typing, and stricter typing would not have moved the original 71% by a point. The summaries were valid strings. They parsed. A JSON Schema requiring summary: string, min_length: 200 would have passed every one of the failures above, because the defect was informational rather than syntactic — the right shape carrying the wrong content. Schema validation catches malformed handoffs and says nothing about lossy ones, and the two get conflated because both present as "the pipeline broke at the boundary".

The distinction has a practical consequence: informational sufficiency has to be tested with an eval on the step-two side, using real intermediates. The question that eval answers is not "did step two answer correctly" but "was the correct answer derivable from what step two received", and that is a separate label an annotator can apply to the intermediate alone. On the original pipeline, 78% of intermediates were sufficient, which caps step two at 78% no matter how good its prompt is, and knowing that number before rewriting step two's prompt saves a sprint.

That cap is worth being precise about, because it is either the most useful number in the project or a slightly wrong one depending on how it is read. Where an intermediate does not contain enough to derive the answer, no prompt on step two can produce the answer from the intermediate, because the material is not in it. So sufficiency bounds what step two can do with what it was handed. It does not bound what step two might produce out of what it already believes, and a measured accuracy above the sufficiency rate would be a signal of exactly that — a step answering from its prior on the questions where the handoff failed it, which is right on general questions and wrong on the specific ones a policy document exists to settle. At 71% the original pipeline was well inside its bound, which is what made the bound the thing to attack rather than the prompt.

None of this argues the split was right. Selecting the same policy's 38 sections with BM25 and feeding the existing single prompt scored 90% at a 1.2 s p95 with 2,100 input tokens, beating the two-call contract design on every dimension including the one it was built for. The contract work above is what you do when the selection genuinely needs judgment; when it does not, the best-designed handoff in the world is still a handoff you did not need. That is worth saying at the end of a chapter about doing handoffs well, because the effort a good contract takes creates its own gravity — once four dimensions have been argued about, a schema compiled per document and a fallback path written, the design has sponsors, and the cheaper option that removes the boundary entirely stops being raised in review.

Decide the four dimensions before the split ships. What the intermediate carries — pointers and verbatim evidence rather than prose, wherever the source can stay available. Whether it is validated, and by a schema whose enums are generated from the source so that invention is undecodable rather than merely detectable. Whether the second stage holds the source, which costs input tokens and buys a recovery path. And what happens when the intermediate arrives wrong, which needs a written answer — retry, abstain, escalate, or fall back — because the default answer is that the second stage answers anyway.

As context windows grow and prompt caching makes re-sending a large document cheap, the reason to compress before answering weakens, and more pipelines can afford to carry pointers and keep the source available to every stage. What does not ease is the discipline of deciding what the intermediate carries, because a stage that hands forward a summary is deciding what the rest of the system is permitted to know.

At two steps, all of this is tractable. The contract fits on a page, the fallback has one path, and the eval sets are two. What happens to the same reasoning when a team has built eleven steps and adds one after every incident is a different problem, and it is not obviously a worse design.

Prompt decomposition — read the current version at https://daviddominguez.dev/learn/en/prompt-decomposition/step-two-cannot-fetch-what-step-one-s-condensation-threw-away

The compression was tuned for a question it never saw#

What the boundary has to specify#

Handling the intermediate that arrives wrong#

The compression was tuned for a question it never saw

What the boundary has to specify

Handling the intermediate that arrives wrong