Decomposing an output-bound job

A 60-page commercial contract of 214 numbered clauses cannot be rewritten in one call by a model that emits 4,000 tokens, and cannot be read at all by the model that emits 16,000. The useful observation about that document is that clause 137 does not depend on clause 12. It depends on the definitions in Part 1, on the governing-law clause, and on the four clauses in its own section that cross-reference it. Everything else is material the model rewriting clause 137 does not need in order to rewrite it well.

Budgeting backwards — reserving the tokens the answer needs before the packer fills anything, which is the previous chapter's rule — has nothing to offer here. It divides a window rather than enlarging one, and the constraint that makes this job impossible is a ceiling on the reply alone. No allocation of the window moves that ceiling, so the only remaining move is to make the reply smaller by making the job smaller.

The clause-137 observation — that a clause depends on its definitions and a few neighbours rather than on the whole document — is what turns an impossible job into nine possible ones, and the technique has a name: decomposition, meaning a partition of the work into groups that each fit inside one call's limits, run separately and assembled afterwards. This chapter derives the group size from the output cap, prices the context that has to be duplicated across groups, sets four ways of dividing the work against each other on input cost, wall clock and coherence, and then spends its second half on the seams — because everything a single generation kept consistent for free is now something you either state in every call or verify after assembly. The two models referred to throughout are the pair from the previous chapter: Model A, a 128,000-token window with a 4,000-token output cap, and Model B, a 32,000-token window with a 16,000-token cap.

The property that the clause-137 observation depends on is worth naming in general terms, because it is the test that decides whether any job can be split at all. Treat the work as units with dependencies between them: a job decomposes exactly when that dependency graph is sparse enough that each unit's dependencies fit inside one call alongside the unit itself. Clause 137 has few dependencies and they are small ones, so it passes. The shared context is the mechanism for satisfying that condition rather than a convenience — it is how the dependencies every unit has in common get into every call. Everything later in this chapter about seams is the bill for the dependencies a split did not preserve, and the section at the end about globally dependent jobs is not an appendix but the same test applied to a job that fails it.

What one call becomes#

The arithmetic that made a single call impossible now works: nine groups of roughly 24 clauses each, about 4,500 tokens of source per group, 3,800 tokens of shared context, and a rewrite of about 5,300 tokens per group — which fits inside Model B's 16,000-token output cap with room to spare, and inside its 32,000-token window once the input is 8,300 rather than 42,000.

Every figure in that sentence is a division, and doing them on the page is what makes the group size something a reader can re-derive for another document. The 214 clauses spread across nine groups is about twenty-four apiece. The 40,500 tokens of source spread the same way is about 4,500 per group. The shared context is not divided at all — it is added whole to each group, which is where the 8,300 of input per call comes from, 3,800 plus 4,500. And the 48,000 tokens of rewrite the finished document needs, spread across nine groups, is about 5,300 each.

The last two of those go against different limits, and keeping them apart is the whole exercise. The 8,300 of input is checked against the window, which bounds what a request may occupy in total. The 5,300 of output is checked against the output cap, which bounds the reply alone and does not enlarge when the input is small. The cap is the constraint the design is organised around, because an output ceiling smaller than the deliverable is what made a single call impossible in the first place, and a group is therefore sized so that its rewrite clears the cap with margin rather than so that its source clears the window.

The shared context is what keeps the pieces from drifting apart, and it is a deliberate selection rather than "as much as fits". For a contract rewrite it is the definitions section, the parties and recitals, the governing-law and interpretation clauses, and a style guide stating the drafting conventions to apply. That block came to 3,800 tokens, and because it is sent with all nine calls, the job's total input is 9 × 3,800 + 40,500 = 74,700 tokens against 42,000 for a single pass. The shared context is the multiplier, and its size is the number to watch: a 12,000-token shared block across nine calls costs 108,000 tokens of input before any clause is sent.

The shape of that sum is what makes the shared block the term to watch rather than the source. The source is a constant: 40,500 tokens of clauses go out once whichever way they are grouped, because each clause belongs to exactly one call. The shared term is a multiplication — nine copies of 3,800, or thirty-four thousand two hundred tokens — and it is the only part of the bill that grows when the document is cut into more pieces. A 12,000-token shared block is therefore not worse in some vague sense; it is the same multiplication with a larger multiplicand, which is how it reaches 108,000 tokens across the same nine calls. The grouping loop that holds both limits at once is short:

SHARED = load_shared_context()          # 3,800 tokens: definitions, parties, style
CAP    = 16_000                         # Model B's output ceiling
WINDOW = 32_000

def group_clauses(clauses: list[Clause]) -> list[list[Clause]]:
    groups, current, out_budget = [], [], 0
    for c in clauses:
        projected = int(c.tokens * 1.2)              # rewrites run ~20% longer
        in_used   = SHARED_TOKENS + sum(x.tokens for x in current) + c.tokens
        over_out  = out_budget + projected > CAP * 0.6      # leave headroom
        over_in   = in_used > WINDOW - CAP * 0.6
        if current and (over_out or over_in or c.starts_section):
            groups.append(current)
            current, out_budget = [], 0
        current.append(c)
        out_budget += projected
    groups.append(current)
    return groups

The c.starts_section term is doing more work than the budget terms. Splitting on a token count alone will eventually cut a group boundary through the middle of an indemnity clause and its carve-outs, and the model rewriting the second half has no idea what it is carving out from. Splitting on structural boundaries costs a little packing efficiency and removes an entire class of incoherent output, so the budget terms should be understood as an upper bound applied to a list of natural units rather than as the thing choosing where the cuts go.

Each call is also told what it may not do. The instruction that matters most in practice is that the call must return exactly the clause identifiers it was given, in order, with no additions — which makes assembly a mechanical merge and makes a drifting model detectable rather than silently creative:

returned = {c["id"] for c in response.clauses}
expected = {c.id for c in group}
if returned != expected:
    raise SeamError(f"missing={expected - returned} extra={returned - expected}")

Four ways to divide the work#

Decomposition is a family rather than a single technique, and the members trade off differently. The figures below are for this contract on Model B, at $3 per million input and $15 per million output, assuming 60 output tokens per second.

The wall-clock column is worth deriving, because its two headline figures come from different arithmetic rather than from different speeds. At 60 output tokens per second, a call producing 5,300 tokens takes about eighty-eight seconds. Nine parallel groups all run at once, so the job finishes when the largest of them finishes rather than when their durations add up, and about eighty-eight seconds is the ~95 s in the table. Sequential carryover cannot overlap anything, since each call needs the call before it to have finished, so its wall clock is the whole 48,000 tokens of rewrite divided by sixty, which is about eight hundred seconds and is the ~14 min.

Parallelism turned a sum into a maximum, and it could only do that because the groups do not depend on one another. That is the sparse dependency graph paying out a second time: the property that made the split correct is the same property that makes it fast, and any design reintroducing dependencies between groups gives the speed back whether it meant to or not.

Hierarchical map-reduce is the one name in that table that is borrowed rather than descriptive: it means running a call per section, then a call over those results, and so on up a tree until one call assembles the whole, which keeps each individual call small and puts the wording of neighbouring sections at some distance from each other. Sequential carryover is the version people reach for first because it sounds like the most faithful simulation of a single pass, and it is the one that scales worst. Feeding each call the rewrites produced so far means call nine carries 3,800 tokens of shared context, 4,500 of its own clauses and roughly 42,000 tokens of prior output, which is past Model B's window entirely, and the wall clock is the sum of nine generations rather than the maximum.

The intuition behind reaching for it is not wrong about what it is aiming at. Coherence really does come from one generation seeing what the ones before it produced, and sequential carryover is the only member of the family that supplies that. Where it is wrong is in whether the aim is reachable. The same mechanism that supplies the coherence — carrying every prior output forward into the next call — is what pushes the ninth call past the window, so the arrangement stops working at exactly the scale that made decomposition necessary in the first place.

The two-pass row is the one that usually ships, and it exploits the asymmetry between the two models instead of fighting it. The nine writing calls go to Model B, which can emit enough. The consistency pass goes to Model A, which can read all 48,000 tokens of assembled rewrite plus the 40,500-token original in one 128,000-token window and only has to emit a short defect report — perhaps 900 tokens listing terminology inconsistencies, broken cross-references and clauses whose numbering drifted. Model A's 4,000-token output cap, which made it useless as a writer, is entirely adequate for a reader whose job is to produce a list. Only the groups named in that report are re-run, so a typical repair costs one more call rather than a second full pass.

How many pieces#

The number of groups is a tuning parameter with pressure from both directions, and it is worth deriving rather than inheriting from whatever the first working version happened to produce. Larger groups mean fewer calls, so the shared context is duplicated fewer times and there are fewer seams to go wrong; the ceiling on group size is the output cap, since a group whose rewrite exceeds 16,000 tokens cannot be emitted at all. Smaller groups mean more headroom against a mis-estimated rewrite length, more parallelism, and a smaller unit of retry.

Running the same contract at three granularities makes the trade concrete. Six groups of about 36 clauses costs 22,800 tokens of duplicated shared context and produces five seams, with each rewrite projected at 8,000 tokens against the 16,000 cap. Nine groups costs 34,200 and eight seams at 5,300 tokens per rewrite. Eighteen groups costs 68,400 and seventeen seams at 2,700 tokens per rewrite, which is the safest against overrun and nearly doubles the input bill for the privilege. The middle option is where this job sits because the projected rewrite length is an estimate that has been wrong by 40 percent on individual clauses, and a group at half the cap survives that error while a group at 90 percent of the cap does not.

Those three granularities are one rule applied three times, and having the rule is what lets the trade be carried to a different document. The number of seams is one fewer than the number of groups, because a seam is a boundary between neighbours. The duplicated context is 3,800 multiplied by the number of groups. The per-group rewrite is the 48,000 tokens the finished document needs divided by the number of groups. Six, nine and eighteen groups check out on all three counts: five, eight and seventeen seams; 22,800, 34,200 and 68,400 tokens of duplication; and rewrites of eight thousand, five thousand three hundred and two thousand seven hundred tokens apiece. Cutting into more pieces therefore divides the output each call must produce and multiplies the input bill by the same factor, which is a trade with no free direction in it.

Persistence changes the shape of the failure more than any of these numbers. Writing each group's result to durable storage keyed by its clause set, as it arrives, means a job that dies after seven of nine calls resumes by running two calls rather than nine, and it means a user can be shown the first sections while the rest are still generating. That last property is worth more than it sounds for a job with a 95-second floor on wall clock: partial delivery in clause order turns a minute and a half of nothing into a document that fills in from the top. It also sets the retry semantics, which deserve a decision rather than a default — nine calls means nine chances to fail, and at a 0.5 percent per-call error rate a nine-call job fails somewhere 4.4 percent of the time, so the grouping key has to be stable across runs for a failed group to be retried on its own.

That percentage is a multiplication rather than an addition, and doing it explicitly is what makes it believable. If a call fails half a percent of the time it succeeds with probability nought point nine nine five. The nine calls do not affect one another, so all nine succeed with that number multiplied by itself nine times, which comes to a little over nought point nine five. Taking that away from one leaves the 4.4 percent, and the reason it is slightly less than nine times the per-call rate is that the small chances of two calls failing on the same run get counted once rather than twice.

The stability requirement follows from the retry rather than from that arithmetic. A failed group can only be re-run on its own if the grouping is a function of the document rather than of the run, since a retry that regroups the clauses differently produces groups that correspond to nothing the first run wrote down. At that point the only safe move is to repeat all nine calls, and the persistence that was supposed to make a failure cheap has bought nothing.

Where splitting is the wrong instrument#

Decomposition looks like retrieval and is not. Retrieval selects a small relevant subset and answers one question, accepting that most of the source material is never seen; decomposition must cover the whole document exactly once, so its correctness criterion is a partition rather than a ranking. Building a decomposition on top of a similarity search is a common and expensive mistake, because similarity search has no notion of coverage and will happily hand you clause 137 twice and clause 138 never.

The reason that mistake is easy to make is that retrieval is the tool an engineer already owns for the sentence "get the relevant context into the call". That sentence is true of both jobs, and it is the only thing they share. A ranking answers "which of these is most relevant", and nothing in it answers "has every clause been sent exactly once", because coverage is not a quantity a similarity score has any means of expressing. A partition has to satisfy the second question, and a system that cannot represent the requirement cannot be tuned into meeting it.

Decomposition is also not summarisation, which topic 2.3 treats as a way to fit a conversation's history into a window. Summarisation reduces the token count by discarding information on purpose. Decomposition preserves everything and pays for it in calls, which is the right trade when the deliverable is the full text and the wrong one when the deliverable is a gist.

Jobs with genuinely global dependencies resist splitting altogether. "Rewrite these 214 clauses" decomposes because the dependency graph between clauses is sparse. "Find the two clauses anywhere in this contract that contradict each other" does not, because every clause depends on every other, and a nine-way split reduces the pairs examined from 22,791 to about 2,500. For that job the answer is a model that reads the whole thing and emits a short answer, which is precisely what Model A's shape is good at.

Both of those counts come from the same small formula. A pair is two distinct clauses, and the number of distinct pairs in a set is the size of the set multiplied by one less than itself, then halved — halved because choosing clause 12 and then clause 137 is the same pair as choosing them the other way round. For 214 clauses that is 214 multiplied by one less than itself and divided by two, which is the 22,791. A group of about twenty-four clauses has twenty-four multiplied by twenty-three over two, a few hundred, and nine such groups have about 2,500 between them.

What the comparison shows is the mechanism behind the whole section. Splitting did not make the contradiction job cheaper. It discarded roughly nine tenths of the comparisons, and the ones it discarded are every pair whose two clauses landed in different groups — which is exactly where a contradiction between distant parts of a contract would live. A rewrite survives that loss because clause 137 genuinely does not need clause 12. A contradiction search does not, because the pairs are the question.

Splitting an output-bound job is the only move available when the deliverable exceeds every writer you have, and it is a trade rather than an escape. Each piece gets its own output cap, which is the point; the shared context is paid for once per piece, which is the bill; and every property the finished document must hold — one word for one party, one numbering scheme, references that resolve — becomes something you either state in every call or verify after assembly, because no single generation can see the whole any more. Deciding which of those two treatments each property gets is where the engineering is.

Both this chapter and the last have assumed a job whose shape is known before it starts: a document with a known size, a deliverable with a known length, a budget computable in advance. An agent working through a task has none of that. Its transcript grows by amounts nobody planned, it accumulates tool results whose sizes are determined by whatever a command happened to print, and when it crosses the boundary the only thing available to cut is history that is already there. Cutting it turns out to have rules.