Counting votes only works on outputs you can normalise

The first version of the SQL voter was four lines and it never once returned a majority. Here are the five samples for "how many orders did customers in Germany place last quarter?", printed as they came back:

-- sample 1
SELECT COUNT(*) FROM orders o JOIN customers c ON c.id = o.customer_id
WHERE c.country = 'DE' AND o.placed_at >= '2026-01-01' AND o.placed_at < '2026-04-01';
-- sample 2
SELECT count(*) FROM customers c, orders o
WHERE o.customer_id = c.id AND c.country='DE'
  AND o.placed_at BETWEEN '2026-01-01' AND '2026-03-31';
-- sample 3
select count(1) as order_count
from orders join customers on customers.id = orders.customer_id
where customers.country = 'DE'
  and date_trunc('quarter', orders.placed_at) = '2026-01-01';

Samples four and five differ again in alias choice and in whether the count is aliased. Four of the five return the identical number against the fixture database, and collections.Counter sees five distinct strings, reports every one with a count of one, and the code falls through to candidates[0]. Five calls were spent, the answer selected was the first one to arrive, and the log line said consensus=1/5 in a field nobody had wired to an alert.

Counting requires an equivalence relation, which is a rule that decides when two differently worded outputs count as the same answer, and text does not come with one. That is the whole of the problem, stated plainly: a counter groups items that compare equal, string comparison is the only notion of equality a string carries, and string equality is a rule about characters rather than about meaning. Everything in this chapter is about supplying the rule that string equality is standing in for.

The ladder from string equality to executed results#

There are five practical ways to decide whether two samples are the same answer, and they differ in what they require of the task and what they cost.

Reading down that table in order is also the order to try things in, because each row is more expensive and more fragile than the one above it. The fragility is not incidental: each row further down replaces a rule you wrote with a rule some component infers, and an inferred rule has a failure mode you cannot enumerate in advance.

On a 400-question held-out set for the SQL feature — questions with known answers, kept out of prompt development so that a score on them is not a score on work already tuned against them — exact string match found a majority, meaning three or more identical samples out of five, in 41% of questions. Canonicalising through the SQL parser, so that case, whitespace, alias names, count(*) versus count(1), and join syntax all fold together, took it to 63%. Executing each candidate against a fixture database with a two-second statement timeout and hashing the sorted result set took it to 88%.

That third step is worth dwelling on because it is the one that generalises furthest and is the least often reached for. When the answer is a program, the equivalence relation you actually want is behavioural — two programs are the same answer when they do the same thing — and you can compute it without asking a model anything.

def result_key(sql: str, conn: Connection) -> str | None:
    """Canonical key = hash of the sorted result set, or None if it will not run."""
    try:
        with conn.cursor() as cur:
            cur.execute("SET LOCAL statement_timeout = '2s'")
            cur.execute(f"SELECT * FROM ({sql.rstrip(';')}) AS _v")
            rows = sorted(tuple(map(repr, r)) for r in cur.fetchall())
    except (DatabaseError, QueryCanceled):
        return None
    return hashlib.sha256(repr(rows).encode()).hexdigest()

Three choices in that function are doing the work. The rows are sorted before hashing, because two queries that return the same set in a different order are the same answer and a hash is sensitive to order. The statement timeout is set inside the transaction, because a candidate query can be arbitrarily expensive and an equivalence check that hangs is worse than one that abstains. And a failure returns None rather than a value, which the next section shows to be the load-bearing decision in the whole file.

The same move is available more widely than it looks. For generated code the key is the test-suite result vector. For a numeric extraction it is the parsed value: "$5,000,000", "5 million USD", "$5M" and "five million dollars" all normalise to Decimal("5000000"), which turns a five-way split into a four-to-one majority against the sample that said $500,000. For a classification it is the label after case-folding and synonym mapping. Free-text drafts are the case with no such move available, and for those the honest options are the bottom two rows: cluster the samples by embedding similarity, an embedding being a vector of numbers placed so that texts with close meanings sit close together, and take the largest cluster's medoid, the member nearest that cluster's centre, or stop counting altogether and have a judge select one, which is best-of-N rather than voting and was covered in 4.7.

Clustering deserves the numbers that make its fragility concrete, since it is the row people reach for the moment the answer is prose. Embedding five drafts with a small embedding model costs about $0.00002 and 40 milliseconds, and the clustering is agglomerative single-link over cosine similarity, which repeatedly joins the two nearest samples and measures nearness as the angle between their vectors, with a single threshold to choose.

Both ends of that threshold fail, and they fail differently. On 150 summary requests, a threshold of 0.94 left every sample in its own cluster on 61% of requests, which is the same fall-through the SQL voter started with — a threshold that strict is asking for near-identical wording, which is what string equality asked for. A threshold of 0.86 collapsed all five into one cluster on 44% of requests, including requests where two drafts carried opposite recommendations phrased in near-identical language. That last case is the one to hold onto: the sentences differed by the word "not", and cosine similarity measures overall closeness of meaning rather than agreement, so a single negation moves a vector very little while reversing what the draft says. There is a workable band between those two values, it is specific to the task, it has to be tuned against labelled pairs, and it moves when the prompt changes. Clustering buys an equivalence relation you could not derive; it does not avoid the work of defining one.

The aggregator, including the parts that are not the happy path#

With a key function in hand the voter is short, and almost all of its length is the cases that are not a clean majority.

@dataclass(frozen=True)
class Verdict:
    answer: str | None
    key: str | None
    support: int          # votes for the winner
    counted: int          # samples that produced a usable key
    outcome: Literal["majority", "plurality", "tie", "scattered", "no_answer"]

def aggregate(samples: list[str], key_of: Callable[[str], str | None]) -> Verdict:
    keyed = [(k, s) for s in samples if (k := key_of(s)) is not None]
    if not keyed:
        return Verdict(None, None, 0, 0, "no_answer")

    counts = Counter(k for k, _ in keyed)
    top = counts.most_common()
    best_key, best_n = top[0]
    tied = [k for k, n in top if n == best_n]
    first_of = {k: s for k, s in reversed(keyed)}

    if len(tied) > 1:
        return Verdict(first_of[best_key], best_key, best_n, len(keyed), "tie")
    if best_n > len(keyed) / 2:
        return Verdict(first_of[best_key], best_key, best_n, len(keyed), "majority")
    if best_n > 1:
        return Verdict(first_of[best_key], best_key, best_n, len(keyed), "plurality")
    return Verdict(None, None, 1, len(keyed), "scattered")

The first line carries the design decision that everything else depends on. A sample whose key function returns None is dropped from keyed entirely, so it abstains rather than voting. That is the correct behaviour and it has to be expressed in the type, because the alternative — a failure path that produces a value — puts that value into the counter alongside the real answers, where it competes on equal terms and can win.

The first_of line is a small trick worth reading rather than skipping. It builds a dictionary from the keyed samples in reverse order, and a dictionary comprehension keeps the last assignment for any repeated key. Walking the list backwards therefore means the last assignment made is the one from the earliest sample, so first_of[key] returns the first sample that produced that key. That matters because the aggregator returns an answer as well as a key, and when three samples agree on a key the three answer strings are still different text.

The second thing the code makes explicit is that "majority" is measured against the samples that produced a key rather than against N: the comparison is best_n > len(keyed) / 2. If two of five abstain, two matching answers out of three is a majority of what you have and a plurality of what you paid for, which is why the Verdict carries support and counted as separate fields. A caller that reads only support cannot tell three-of-three from three-of-five, and those warrant different responses.

What to do with the outcomes that are not a majority#

Across the 400-question set at N=5, after result-set normalisation, the five outcomes came out as follows, and the right response to each depends on whether the task has an answer that can be checked.

The plurality and tie rows are 13% of requests, which is the whole justification for handling them adaptively rather than raising N for everyone. Work the two costs against each other. Moving the whole workload to N=7 spends the two extra calls on every request. Drawing two more samples only when the first five fail to settle spends two extra calls on 13% of requests, which averages 0.26 extra calls per request, and it concentrates that spend exactly on the requests that were ambiguous. The implementation is a loop with a stopping rule rather than a fixed fan-out, at the price of a second round-trip on those requests — usually between 1.5 and 4 seconds of added tail latency, which is why the adaptive version belongs behind a flag you can turn off for latency-sensitive paths.

Ties on verifiable tasks have a resolution that costs nothing: run both candidates and prefer the one that executes without error, or the one that satisfies a cheap invariant such as returning a non-empty result set for a question that presupposes rows exist. Ties on free text have no such resolution, and the honest choices are a judge call, whose price the previous chapter put at 13% of the branch spend on the SQL feature, or a human. Picking the first sample because it arrived first is also a choice, and it is defensible, but it should be written down as a policy and counted, because a system silently taking candidates[0] on 4% of traffic is a system whose measured accuracy includes 4% of unvoted answers.

Voting per field builds records nobody wrote#

Structured extraction invites an aggregation that seems obviously better than voting on the whole record: vote on each field independently, so that a sample which got three fields right and one wrong still contributes its three. On a contract extraction with effective_date, term_months, notice_days and cap_amount, field-wise voting raised per-field accuracy from 0.86 to 0.91 and lowered whole-record accuracy from 0.71 to 0.68.

The reason is that the winning combination need not be a combination any sample produced. Each field's winner is decided by its own count over its own set of values, and nothing requires the same samples to be behind two different winners. term_months = 24 won its field from three samples that all also said notice_days = 90; notice_days = 30 won its field from two samples that said term_months = 12, plus one that abstained on the term. The assembled record says a 24-month term with 30 days' notice, which contradicts the agreement's clause tying notice to one-eighth of the term, and no sample ever asserted it.

Two properties of that failure make it worse than it first looks. The record is internally inconsistent in a way that no per-field check can catch, since every field is individually the most popular value. And the per-field number went up, so the metric the change was made against endorsed it — per-field accuracy is the quantity that stays reassuringly on trend while whole-record accuracy, which is what a downstream system consumes, moves the other way.

Fields that are coupled by the source document have to be voted as a unit, or the assembled record has to be run through the same consistency check you would apply to a single sample's output, which is a third call. Field-wise voting is right where the fields genuinely are independent — an address and an invoice number do not constrain each other — and the way to find out is to measure whole-record accuracy alongside per-field accuracy rather than reporting only the number that went up.

The voter returns support and counted, and the temptation is immediate: four out of five is 0.8, put it in the response, show it to the user, route anything under 0.6 to review. The routing part of that is defensible and the chapter after next gives it a measured basis. The number being a probability is not.

Four samples agreeing came out of one prompt on one model, and whatever made them agree is a property of that pair rather than evidence about the world. The next chapter takes the case where four of five agreed on an answer that was wrong, works out what a majority is actually estimating, and explains why the obvious fix — raise the temperature so the samples spread out — buys variance without buying independence.