Parallelization — sectioning and voting

Read these first

• Choosing an architecture
• Self-consistency, majority voting over N samples, best-of-N with a judge

What you will know

• Split independent subtasks and run them concurrently

  A shared rate limit, mutable state, or dependency can erase concurrency gains or create races.

• Run the same task N times and aggregate by voting

  Agreement can amplify a shared prompt or model bias, so consensus is not independent evidence.

• Measure branch cost, latency, and concurrency before committing

  Include input and output tokens, orchestration overhead, retries, and peak concurrency rather than multiplying model price alone.

• Show accuracy improving with N and where it stops

  Use held-out cases and confidence intervals; correlated samples can make the apparent knee arrive earlier than expected.

• Compare sectioning and voting on a fixed workload and budget, measuring quality, latency, and cost

  Sectioning can lose cross-subtask context, while voting can spend extra calls without increasing coverage.

Chapters

1. 1

   Splitting the work and repeating the work are different bets

   Input size is a symptom, not the criterion: a short question can decompose into independent parts and a 60-page document can be one indivisible judgement. This chapter separates sectioning, which splits a task into parts that never need to read each other, from voting, which runs one task N times and aggregates. Only one of the two applies to a single hard question with one verifiable answer.

   Read chapter
2. 2

   Eight concurrent branches, 1.4x faster: the quota is queueing

   You fan out eight ways and wall-clock time falls by 1.4x, with no errors logged and all eight tasks created milliseconds apart. Tail domination cannot produce that arithmetic; a shared tokens-per-minute budget can, because eight simultaneous requests on one account queue rather than run. Measure the concurrency you achieved rather than the concurrency you requested, and check for mutable state shared across branches.

   Read chapter
3. 3

   Best-of-five costs more than five times one call

   Estimating a move to best-of-five as five times the per-request price misses four things: the prompt is re-sent on every branch so input tokens scale too, the aggregation or judge call is unaccounted, retries and timeouts are absent, and peak concurrency may move you into different rate-limit behaviour. Build the accounting that explains an 8.6x bill on an eight-way fan-out — a different cause from the missing speedup.

   Read chapter
4. 4

   Counting votes only works on outputs you can normalise

   Exact-match voting compares answers, so it works out of the box only where the answer is verifiable or can be normalised to a label; free-text drafts and extractions need an explicit equivalence rule, semantic clustering of the N samples, or a scorer that selects one sample rather than counting occurrences. This chapter builds the aggregation step itself and settles the cases that break naive counting — an exact tie, all N different, a plurality of two out of five — and names the aggregation call as a real cost line rather than a footnote. The point is to be able to implement best-of-N on your own task, not only to reason about one.

   Read chapter
5. 5

   Four samples agreeing is not four pieces of evidence

   Five samples at temperature 0.8 from one prompt on one model, four of them agreeing on a wrong answer, tell you very little: the errors are correlated by construction. This chapter explains what a majority actually measures, why raising the temperature buys variance rather than independence, and when consensus amplifies a shared prompt or model bias instead of cancelling noise.

   Read chapter
6. 6

   The accuracy curve flattens sooner than the sample count suggests

   On 200 held-out cases, voting scores 71, 78, 79 and 79 at N of 1, 3, 5 and 7. The last four points are two cases, which sits inside the interval, and the apparent monotonicity is an artefact of having taken four measurements. Plot accuracy against N with confidence intervals, find where correlated samples make the curve flatten, and report the interval next to whichever N you ship.

   Read chapter
7. 7

   Different models, prompts and evidence decorrelate samples; temperature does not

   Once the curve has flattened, the errors that remain are shared misunderstanding rather than sampling noise, and more draws from the same prompt on the same model cannot reach them. The moves that genuinely decorrelate are structural: a second model in the ensemble, materially different prompts or decompositions per branch, or branches given different tools and evidence — each of which costs more and changes how the aggregator from the earlier chapter must compare answers. When none of that is available, the budget belongs in the input — better retrieval, a sharper specification, a worked example — rather than in a larger N.

   Read chapter
8. 8

   A sectioned review cannot see what crosses its boundaries

   Split a 60-page contract into six concurrent sections and per-section quality rises while conflicts between a clause in section two and one in section five become structurally invisible — no prompt repairs what no branch can see. Add a cheap reconciliation pass over the structured outputs, understand why giving every branch the full document undoes the gain sectioning was adopted for, and fix the evaluation that never moved.

   Read chapter
9. 9

   Spending a fixed call budget on the shape the task actually has

   Six calls per request, one budget, two ways to spend it. Bring the previous chapters' measurements together on a fixed workload: quality, latency and true cost for the sectioned design against the voted one, with each design's blind spot named explicitly rather than discovered in production. A single question with one verifiable answer offers nothing to section, so the budget goes to samples and a vote you now know how to build.

   Read chapter