Aggregating free-form answers

A vote is a count, and a count needs things that can be equal. Two paragraphs can express the same reason in different words and count as different; two paragraphs can differ by a single negation and be one edit apart. The space is effectively continuous, every draw is a fresh point in it, and every count is one. Run a majority vote over five generated paragraphs and it will return a winner with a vote share of 0.2, which is the arithmetic saying that nothing was aggregated.

That 0.2 is worth reading rather than glancing at, because it is not a low number on a scale — it is the lowest value the metric can take. A vote share is the winning count divided by N, and the winning count cannot be less than one, so at five samples the floor is one fifth. Seeing 0.2 means every count was one, which means no two answers were the same, which means the counting step had nothing to do. The reason this ships is that 0.2 reads as a confidence number and confidence numbers are expected to be low sometimes. It is not a confidence number here. It is the vote reporting that it was handed a set with no duplicates in it.

Equivalence is the property being assumed and the property that is missing. A count needs to be able to say that two answers are the same answer, and the relation that licenses that has to hold together in a particular way: every answer is the same as itself, sameness runs in both directions between two answers, and if two answers are each the same as a third then they are the same as one another. String equality has all three of those and is far too strict for prose, so any workable scheme has to supply a looser relation and then be checked against the same three properties. The third one is the one that gets quietly given up.

What replaces string equality is not one technique but four, and they are genuinely different techniques rather than variations on a theme: canonicalise the answers until they can be compared, vote only the discrete part of a mixed output, cluster by semantic similarity and vote on cluster membership, or hand all N candidates to a judge and ask it to select. The first two keep the properties that made voting attractive — deterministic, auditable, free. The last two do not, and the cost of finding out which one your output shape needs is the subject here.

Canonicalisation covers more cases than people expect#

Numeric answers arrive as 1,240.00, $1240, and 1240.0; dates as 2026-03-04 and 4 March 2026; entity names with and without Inc.. A normalisation function collapses those into one key, the vote works on the key, and the surviving raw string is returned to the caller. The work is unglamorous and worth doing carefully, because a normaliser that is too aggressive merges genuinely different answers and manufactures agreement, while one that is too timid leaves the vote fragmented and reports low confidence on cases where the model was in fact consistent.

The failure mode to watch for is asymmetric. Over-merging inflates the vote share and the accuracy at the same time, so it looks like the change worked; under-merging deflates the vote share while leaving accuracy alone, so it looks like the model is unreliable. Sampling twenty cases and reading the pre-normalised and post-normalised answers side by side catches both, and it is the only check that does.

The mechanism under that asymmetry is what decides which direction survives a review. Over-merging collapses answers that are genuinely different onto a single key, so the count on that key concentrates and the vote share rises; and where the key it merged onto is the one the scorer treats as correct, a case that was wrong is now scored right, so accuracy rises alongside it. Two numbers moving up together is what a successful change looks like from a dashboard. Under-merging leaves equivalent answers on separate keys, so the counts fragment and the vote share falls, while accuracy is untouched, because the answer that was going to win still wins — it just wins with a smaller share. The direction that looks like a broken model is the safe one. The direction that looks like a win is the one that has to be checked by reading, which is why the check is twenty cases read side by side rather than a threshold on either number.

Voting the discrete part of a mixed output#

Where the output has a discrete part and a free part, only the discrete part needs to be voted.

def hybrid_vote(samples: list[dict]) -> dict:
    """Vote the decision; take the prose from a sample that agrees with it."""
    decisions = [s["decision"] for s in samples]
    winner, count = Counter(decisions).most_common(1)[0]
    supporting = [s for s in samples if s["decision"] == winner]
    return {
        "decision": winner,
        "agreement": count / len(samples),
        "reason": max(supporting, key=lambda s: len(s["reason"]))["reason"],
        "n_supporting": len(supporting),
    }

That returns a paragraph written by a sample that reached the winning decision, which fixes a real defect in the naive version: with independent generation, the returned prose could argue for one decision while the voted decision was another. It does nothing to make the paragraphs better, because among the samples that voted the same way there is still no principled reason to prefer one paragraph over another, and picking the longest is a heuristic that rewards padding.

The defect it fixes is worth naming precisely, because the naive version looks correct until you ask where each piece of the record came from. The decision is a count over all five samples. The prose is a string lifted out of one of them. Nothing connects those two operations, so the sample that supplied the paragraph need not be among the samples that supplied the winning decision, and the record that ships can carry an argument for one outcome sitting above a different outcome. Filtering to supporting before choosing the prose restores the link by construction. The record is still not something any single sample said — it is a decision from the count and a paragraph from one member of the winning group — but its two halves can no longer contradict each other.

The point where it stops being voting#

To choose among paragraphs that all agree on the discrete part, you need something that can rank them, and the usual something is another model call with a rubric. That is a different method with a different name: best-of-N with a judge, sometimes called reranking or selection. The distinction is not pedantic, because the two have different requirements, different cost curves and different failure modes.

The last row is regularly underestimated. A judge call for five candidate paragraphs of 180 tokens each carries the original input, the rubric, and 900 tokens of candidates, so it is frequently the most expensive single call in the pipeline rather than a rounding error on top of the five.

The 900 is five times 180, and writing it that way shows which direction the judge prompt grows in. Every candidate has to be inside the prompt, because a judge cannot compare things it has not been shown, so the candidate block scales with N while the original input and the rubric are paid for once. Raising N therefore does not cost a little more at the judge, it costs proportionally more in the part of the prompt that is already the largest. It is also why the aggregation step stops being free in the sense counting is free: counting reads N strings that are already in memory, and judging sends those N strings back across the wire and pays for them again as input tokens.

The ceiling row is where best-of-N earns its place. Voting converges on the model's modal answer, so it can never return the unusual-but-correct sample that appeared once out of five. A judge can, in principle, since it is looking at all five and picking rather than counting. A temperature sweep with coverage at 92.6% against a vote accuracy of 88.7% has four points that a perfect selector would recover. Whether a real judge recovers any meaningful fraction of them depends heavily on whether the judge has any information the generator lacks.

That condition is the whole question, and it is worth having as a mechanism rather than as a caveat. A judge drawn from the same family, handed the same input and a rubric, computes a function of very nearly the information the generator already had. Where the generator's error came from missing information or from a misreading of the instructions, the judge is missing the same thing and reading it the same way, so it has no basis for preferring the odd correct sample over four fluent wrong ones — the property that made the wrong answer attractive to write also makes it attractive to pick. Where the error came from a slip that is easier to spot than to avoid, such as an arithmetic step that can be rechecked against numbers already on the page, the judge is doing genuinely new work and the coverage gap is partly recoverable. Which of those two your errors are is a property of the task, and it is settled before any judge is written.

Best-of-N does not strictly require a second model, either, which is worth knowing before you commit to the extra call. If the generator exposes token logprobs, the log-probability it assigned to each token at the moment it emitted that token, you already have a per-candidate score — mean logprob, or a sharper statistic over the answer tokens — and ranking candidates by it costs nothing beyond arithmetic. That family of methods is what Scalable Best-of-N Selection for LLMs via Self-Certainty formalises, and its appeal is that the selection signal comes free with the generation. Its limitation is that the score measures how confident the model is rather than how correct it is, so it inherits every miscalibration the generator has, and length normalisation choices change the ranking more than you would like.

Both halves of that limitation have a shape you can predict without running it. A logprob is the log of the probability the model assigned to a token, so it is high when the token was easy to predict, and a mean over the answer tokens is a score for how unsurprising the candidate was. Generic sentences are unsurprising. That is not a miscalibration to be tuned away; it is what the statistic measures. The length effect is arithmetic: every logprob is negative, since probabilities are below one, so a sum over more tokens is necessarily more negative and summing would rank short candidates above long ones for no reason other than their length. Dividing by the token count removes that bias and introduces a different one, because it lets a single very confident token pull up a candidate that was uncertain everywhere else.

Embedding similarity is a poor equivalence relation#

Between canonicalisation and a judge sits a third option: cluster the samples by semantic similarity and vote on cluster membership. Embed each answer — run it through an embedding model, which turns a piece of text into a fixed-length vector of numbers positioned so that texts about the same thing land near one another — group the vectors by cosine similarity above a threshold, which measures the angle between two of them on a scale where 1.0 is identical direction, and the largest cluster wins, with a representative from that cluster returned.

def cluster_vote(answers: list[str], threshold: float = 0.86) -> tuple[str, float]:
    vecs = embed(answers)                       # one embedding call, N texts
    clusters: list[list[int]] = []
    for i, v in enumerate(vecs):
        for c in clusters:
            if cosine(v, vecs[c[0]]) >= threshold:
                c.append(i)
                break
        else:
            clusters.append([i])
    best = max(clusters, key=len)
    return answers[best[0]], len(best) / len(answers)

Before any question of accuracy comes a structural point about that loop. Similarity above a threshold is not an equivalence relation, because it is not transitive: one answer can sit above the threshold from a second, the second above the threshold from a third, and the first and third below it from each other. Something has to break the tie, and the loop breaks it by comparing each new answer only against vecs[c[0]], the first member of each existing cluster, then joining the first cluster that accepts it. Which answer became c[0] is whichever one the list happened to start with, so the clusters — and with them the winner and the reported share — depend on the order the samples arrived in. With concurrent calls, that is the order the event loop finished them. The third property was lost when string equality was traded for a threshold, and the code had to substitute something arbitrary in its place.

Clustering works acceptably when the differences between correct and incorrect answers are topical, and it fails in a specific way worth knowing before you deploy it. Cosine similarity in a general-purpose embedding space is dominated by subject matter, so "declined because the applicant's stated income could not be verified against the submitted payslips" and "declined because the applicant's income was verified but falls below the threshold for this product" sit at cosine 0.94 — one clause apart in meaning and, for an adjudicator, opposite in what they imply about the file. Any threshold high enough to separate those will also split paraphrases of the same reason, so the cluster sizes stop tracking agreement at all. On the underwriting set, threshold sweeps from 0.80 to 0.95 produced either one cluster of five or five clusters of one on 78% of cases, with a narrow band in between where the clustering was doing something and it was not obviously correct.

The 0.94 is not an artefact of a weak embedding model, which is why raising the threshold does not rescue the design. An embedding summarises what a text is about, and both sentences are about a declined application on income grounds — same applicant, same product, same decision, nearly the same words in nearly the same order. The clause that reverses what the file means, could not be verified set against falls below the threshold, is a small share of what there is to summarise, so it moves the vector by a small amount. Two honest paraphrases of one reason, meanwhile, can differ in more words than that, and so can end up sitting further apart than the two opposed reasons do. Once that ordering is inverted there is no single threshold that separates the pairs you need separated without also splitting the pairs you need merged, and the sweep result is what that looks like from outside: on more than three cases in four, every threshold in the range either merged everything or merged nothing.

The failure has a shape you can predict from the task rather than discovering empirically: embedding similarity is a poor equivalence relation whenever the discriminating information is a small number of tokens against a large shared context. Negations, quantities, dates and named entities are the usual carriers of that information in business outputs, which covers most of what these services produce. Where the candidates differ substantially in content — five different proposed root causes for an incident, say — clustering is far more defensible.

Splitting a mixed record, and what splitting costs#

The concrete version of this decision arrives when the output is structured but mixed. Take an extraction that returns vendor, invoice_number, total, currency, payment_terms_summary and discrepancy_note. Four of those are discrete, one is a short paraphrase of a contractual clause, one is a free-text observation that may legitimately be empty.

Per-field voting is workable on the four and has nothing to count on the other two. Best-of-N with a judge is workable on all six and forces a single choice across the whole record, which means a candidate with the best discrepancy_note and a wrong total competes as one unit. The design that follows is to split the record: vote the four discrete fields independently, then handle the free-text pair with a judge restricted to candidates that agree with the voted fields, or with no aggregation at all if the free text is advisory. Splitting also removes an artefact of whole-record voting, where five candidates that each get five of six fields right can produce zero exact-record agreement while a per-field vote reconstructs a record better than any individual sample.

That artefact is worth walking, because it is the clearest case of an aggregation rule returning something no sample contained. Take five candidates on a six-field record, each correct on five fields and wrong on one, and let each be wrong on a different field. No two candidates are then identical, so whole-record voting sees five distinct records with a count of one apiece and returns an arbitrary one of them — a record with a known error in it. Now vote the fields separately. On any given field, at most one of the five candidates is the one that got that field wrong, so at least four of the five agree on the right value and the per-field mode is correct. Every field comes out right and the assembled record is better than all five of the records that were generated. Same samples, same votes; the only thing that changed is what the counting was applied to.

There is a caveat to per-field voting that shows up in production and not in evals. Fields voted independently can produce a record that no sample proposed and that is internally inconsistent: a total from sample 2 with a currency from sample 4, giving a euro amount labelled as dollars. If the fields have dependencies, the vote has to respect them, either by voting the dependent group as a tuple or by validating the assembled record afterwards.

The gain and the caveat come from the same fact, which is why no configuration separates them. The assembled record is not one of the samples. It is a column-by-column mode, and a column-by-column mode is under no obligation to be one of the rows it was computed from. In the six-field case that is precisely the point, because none of the rows was worth returning. Where total and currency have to be read together it is a defect, and the repair follows from the same sentence: stop treating them as two columns. Voting the pair as a single value forces the winning amount and the winning currency to have come from the same sample, at the cost of a vote that fragments more easily, since two fields now have to agree before two samples count as agreeing at all.

Neither method can select what was not generated#

Both approaches select from what was generated. If the correct answer is absent from all N samples, voting cannot count it and a judge cannot pick it, so coverage caps both. Both are also defeated by the same underlying condition: errors common to all samples. Voting confirms a shared error unanimously; a judge drawn from the same model family tends to endorse it, because the property that made the error attractive to the generator also makes it attractive to the selector.

So the question to settle first is not which aggregation rule to use but what shape the answer is, and that question has a cheap answer. If the outputs can be made equal by a normalisation function you would be willing to show someone, vote them. If the discrete part of the output carries the decision and the prose is commentary, vote the field and take supporting prose. If the whole output is prose and the distinguishing information is a clause rather than a topic, clustering will not save you and a judge is what remains — at N+1 calls, with a selector whose preferences you now have to measure, and with the aggregation step no longer deterministic or auditable.

That is a real change in what the design is, and the thing worth noticing is how quietly it happens. The proposal in the review said "self-consistency, five samples, majority wins". What ships, once the output turns out to be a paragraph, is a second model grading the first — a decision nobody made and nobody priced. The aggregation step changes what you need from the answer space and what you pay per request. It does not change the fact that N samples of one model on one prompt carry one model's beliefs.

That leaves an obvious question unanswered: given that N buys something real when the samples do decorrelate, how much it buys, and how fast it stops buying it.