A second model earns its cost only if its errors differ

The proposal arrived with a slide and a quote from a vendor's solutions engineer: use a different provider's model for the critic, because a second opinion from a different system is worth more than a second opinion from the same one. The engineer who ran it got a result that looked like support. Swapping the critic to provider B raised the catch rate on the constraint_violation class from 29 of 43 to 33 of 43 in a single run, at 2.2 times the critic's token cost and 3.4 seconds more at p50.

Four extra catches out of forty-three, on one run of a 43-item slice, is inside the noise of that slice — a fact the harness from chapter 5 makes checkable rather than arguable. Running it five times with different sampling seeds gave 33, 30, 31, 34 and 29, against the original critic's 29, 31, 28, 30 and 30. The medians are 31 and 30. Whatever provider B is doing on this task, catching a materially different number of constraint violations is not it.

The single run was compared against the wrong reference, and naming the right one is most of what this chapter teaches. A difference of four catches was being read against zero — against the implicit claim that a run of the same configuration would return the same number twice. The five seeds say otherwise. Provider B's own five runs span from twenty-nine to thirty-four, a range of five catches, and provider A's span from twenty-eight to thirty-one, a range of three. So the four-catch gap that arrived on the slide is smaller than the run-to-run range of either arm on its own. Any comparison of two configurations needs a reference for how much a single configuration moves when nothing changes, and that reference costs four more runs.

Count the items, not the totals#

Two critics with equal catch rates can be catching the same 30 items or two different sets of 30, and totals cannot tell you which. The measurement that can is per-item: for each item in the fixed set, record whether each model got it right, and then count the four cells.

def error_table(items, model_a, model_b):
    cells = {"both_wrong": 0, "a_only": 0, "b_only": 0, "both_right": 0}
    for it in items:
        wa = not caught_defect(model_a, it)
        wb = not caught_defect(model_b, it)
        key = ("both_right", "b_only", "a_only", "both_wrong")[wa * 2 + wb]
        cells[key] += 1
    return cells

def phi(c):                      # correlation of the two error indicators
    n11, n10, n01, n00 = (c["both_wrong"], c["a_only"],
                          c["b_only"], c["both_right"])
    num = n11 * n00 - n10 * n01
    den = math.sqrt((n11 + n10) * (n01 + n00) * (n11 + n01) * (n10 + n00))
    return num / den if den else 0.0

The indexing trick in error_table is worth reading once so the cells are not a mystery later. wa and wb are booleans, and Python counts True as one, so wa * 2 + wb turns the pair into a number from zero to three: both right is zero, only B wrong is one, only A wrong is two, both wrong is three. The tuple of names is written in that order, so the expression picks the right cell without any branching. The four counts it produces are exhaustive and mutually exclusive, which is why they must add to the number of items in the class.

Phi is the correlation coefficient for two yes/no variables, here the two models' error indicators, computed from the four cells of that table: it sits near 0 when the models are wrong on unrelated items, rises towards 1 as they are wrong on the same items, and would go negative in the unlikely event that one model's mistakes were where the other's strengths are. The number to read alongside phi is the pair of conditional error rates. If provider B is wrong on 26% of items overall but on 71% of the items where the writer's model is also wrong, its errors are concentrated exactly where you needed help, and the extra call is buying you very little. If B's error rate is roughly the same whether or not A erred, the two models are failing on different items and a critic on B will catch things a critic on A cannot.

Meridian ran this on two of its defect classes, over the same 120 items, with the writer fixed on model A throughout.

Every derived row in that table comes out of the four cells above it, and doing the arithmetic once is what makes the table readable rather than magical. B is wrong on an item whenever the cell counting it is both models wrong or B wrong only, so B's error count on the judgement class is eleven plus two, which is thirteen, and the four cells add to forty-three, giving the 30%. On the calculation class it is three plus eight, which is eleven, out of the sixty-one those cells add to, giving the 18%. The conditional rate restricts the denominator to the items A got wrong, which is both wrong plus A wrong only: fourteen items on the judgement class, of which eleven are also wrong for B, giving the 79%; and twelve items on the calculation class, of which three are also wrong for B, giving the 25%. The two rates are the same quantity measured on two different populations, and the gap between them is the whole finding.

Work phi by hand on the right-hand column, where the cells are small enough to carry. The numerator is both wrong times both right minus A wrong only times B wrong only: three times forty-one is a hundred and twenty-three, nine times eight is seventy-two, and the difference is fifty-one. The denominator multiplies the four margins — the two row totals and the two column totals of the two-by-two table, which here are twelve, forty-nine, eleven and fifty — and takes the square root of the product, which lands a little under five hundred and seventy. Fifty-one over that is the 0.09 the table reports. Reading the formula rather than the result: the numerator is a difference between the two diagonals, so it is zero exactly when knowing A's outcome tells you nothing about B's, and the denominator scales it so the answer lands between minus one and one whatever the class size.

The two columns describe two different worlds. On constraint violations the models are wrong together: when A misreads the relationship between a retention constraint and a consolidation recommendation, B misreads it 79% of the time, against a 30% base rate. Whatever produces that failure — a way of reading conditional obligations in business prose, probably learned from the same corpora and reinforced by similar preference training — is present in both systems. Paying provider B to look at those items buys three additional catches in exchange for eleven shared misses, and no amount of prompt work on B changes the shared part.

On arithmetic reconciliation the same two models are close to independent. A phi of 0.09 with a 41/61 both-right cell means the errors are essentially unrelated draws, so a critic on B genuinely catches most of what A's critic misses, and the union of the two catches 49 of the 61 rather than the 52-minus-overlap arithmetic you would get from correlated errors.

That is the answer to when a different model is worth paying for, stated as a property you can put a number on: when its errors are weakly correlated with the writer's on this particular task. Not when it scores higher on a leaderboard, not when it is newer, and not when it comes from a different company.

Two practical notes about running this. The per-item right/wrong judgement has to come from the labelled set rather than from either model, since asking one model whether the other was right reintroduces exactly the correlation you are trying to measure. And the sample size needed is smaller than for the quality comparison in chapter 5, because you are estimating a correlation between two binary indicators rather than a small difference in means — 40 to 60 items per defect class gives an interval on phi tight enough to separate 0.1 from 0.7, which is the only distinction the decision turns on.

That last point is worth one more sentence, because it looks like a contradiction of chapter 5's demand for hundreds of items. The difference is in what is being estimated. Chapter 5 was trying to resolve a gap of a fraction of a grade point against a per-item spread several times larger, which is a small effect swamped by noise. Here the two outcomes being distinguished are a correlation near a tenth and one near seven tenths, which is not a small effect at all — the two produce visibly different four-cell tables at forty items. A decision that turns on a large difference needs far less data than one that turns on a small one, whatever the topic.

The logo is a poor proxy for the weights#

Different provider is a convenient stand-in for different failure modes and a weak one. Frontier models are trained on heavily overlapping web corpora, tuned against overlapping public benchmarks, and shaped by preference data collected under similar guidelines, so they converge on similar readings of ambiguous instructions and similar gaps in specialised knowledge. Meridian's phi of 0.71 on the judgement class is two different companies' flagship models agreeing about how to misread a constraint.

Some pairings are near-useless by construction and worth ruling out before running anything. A smaller sibling of the writer's model shares the same base and usually the same tokeniser and post-training recipe, so its errors are close to a subset of the larger one's. A model that was distilled from the writer's model — trained to reproduce the larger model's outputs, which is how most small fast models are made — inherits its judgements almost by definition. Two checkpoints of the same family differing by a point release are the same system with a patch. In each of these the phi is high for a reason you could have predicted, and the run only confirms it.

Where the correlation genuinely tends to be lower is where the models differ in kind rather than in vendor. A critic that is not a language model at all — a script that recomputes the totals, a SQL query against the entitlement table, a schema validator — has an error profile with no relationship to the writer's, which is why chapter 4's arithmetic_error category is a strange thing to be paying two frontier models to argue about. Meridian eventually moved that class to eleven lines of Python that reconcile the section totals against the usage rollup, at a phi with the writer of approximately zero and a cost of nothing, and both models stopped being asked about it.

The reason a script's phi is approximately zero rather than merely low is worth stating, because it is the only place in this topic where a guarantee is available. Correlation between two error indicators requires a shared cause of error. A model and a script share nothing that could produce one: the script's failures come from bugs in eleven lines of arithmetic that a person wrote and can read, and the model's come from what its training distribution does or does not contain. Those two sources have no channel between them. So the independence is structural rather than measured, and it does not need re-checking when either provider ships a point release.

What you would want to see before agreeing#

The correlation is also not a fixed property of the two models, which is the limitation to keep in view. It is a property of the pair on this task, with these prompts, at this point in both providers' release cycles, and all three of those move. A point release on either side changes the failure modes it changes, and a phi measured in January is a claim about January. Re-running the error table is cheap once the harness exists — an hour of compute and no new labelling — so the honest cadence is to re-run it whenever either model version changes and whenever a defect class is added, and to treat a stale phi the way you would treat a stale benchmark result from a machine you no longer own.

When a teammate proposes moving the critic to another provider, the request is reasonable and the evidence for it is specific. Grounding should already be done, because a defect class that fails for missing knowledge will fail on any model and the second vendor will be blamed for not fixing it. The per-item error table should exist for the defect classes the loop is actually meant to catch, run on the fixed set inside the matched harness, with enough repeats to distinguish a real gap from sampling noise. And the latency and operational budget should be stated in advance, since a phi of 0.2 on a class you care about justifies quite a lot of spend, and a phi of 0.7 justifies none.

The ordering of those three requirements is not a preference. Grounding comes first because it changes what the second question means: a class failing for missing knowledge produces a high phi for a reason that has nothing to do with either model's judgement, so measuring the correlation before grounding measures the shared absence of a fact and calls it a shared failure mode. The error table comes second because it is what converts the proposal into a number. The budget comes last because it is the only one of the three that depends on the answer.

There is a case the error table does not resolve, and it is the one that turns up most often in practice. Two critics, or two arms, can come out with the same aggregate score and the same catch totals while failing on systematically different items, which the phi tells you but the score does not. When that happens the right conclusion is not that the arms are interchangeable, and the decision that follows is a different kind of decision from any made so far.