Paired Deltas: Compare Models on the Questions They Share

The Two-Point Gap That Wasn’t (Yet)
Model A scores 390/500 on your benchmark: 78%. Model B scores 400/500: 80%. A two-point gap, and everyone is ready to promote checkpoint B.
Run the arithmetic from the previous post and it looks hopeless. Treating the two scores as independent measurements:
The 2pp gap is 0.8 standard errors: indistinguishable from noise. That analysis, though, throws away the most important fact about the data. Both models answered the same 500 questions, and the score table decomposes:
- Both right: 375
- Both wrong: 85
- B right, A wrong: 25
- A right, B wrong: 15
The 375 questions both models solved tell you nothing about which is better. Neither do the 85 both flubbed. Only the 40 discordant questions carry any information about the difference, and there B won 25 to 15.
Analyze it paired and the picture sharpens: pp with
Same data, half the error bar, and nothing new to collect. The rest of this post is why that works and when it doesn’t.
The Whole Idea in One Formula
Score each question with a per-question difference:
A question both models get right (or both wrong) scores . Then the comparison is just a one-sample problem on the :
For binary outcomes this collapses to counts. With = questions only A got right and = questions only B got right:
The equivalent framing explains where the win comes from. The variance of a difference is
The unpaired analysis assumes the covariance is zero. But two models grading the same questions succeed and fail together: easy questions help both, cursed questions sink both. That shared fate is positive covariance, and pairing subtracts it. The size of the win tracks the covariance:
- Models that agree a lot → big covariance → big win from pairing. Checkpoints of the same run, prompt variants, models from the same family.
- Near-independent models → almost no covariance → pairing subtracts almost nothing, and the two bars come out nearly the same.
In principle two models could be negatively correlated (one fails exactly where the other succeeds), and then the paired bar is the wider one. It is still the correct one: with shared questions, the unpaired formula is simply the wrong model of the data, in whichever direction it errs. Real model pairs almost always share their easy and hard questions, so in practice pairing shrinks the bar.
A Worked Example
The hook’s numbers, walked at hand-calculator speed. These are the same four steps the practice exercise below will ask of you.
Step 1: Sort the table
Cross-tabulate the 500 questions by who got them right:
| B right | B wrong | Total | |
|---|---|---|---|
| A right | 375 | 15 | 390 |
| A wrong | 25 | 85 | 110 |
| Total | 400 | 100 | 500 |
So (both right), (A only), (B only), (both wrong). The marginals recover the headline scores: A right on , B right on . The 460 concordant questions just left the conversation; the effective sample is the discordant ones.
If this table looks familiar from Understanding Cohen’s Kappa, it is the same layout. Kappa reads the diagonal to measure agreement; the paired delta and McNemar’s test read the off-diagonal to measure who wins.
Step 2: The delta
Exactly 80% minus 78%, computed the long way. The point of the table isn’t the delta, it’s the error bar.
Step 3: The error bar, twice
Unpaired, from the hook: pp.
Paired, from the discordant counts:
Same table, and the error bar shrank by a factor of . To buy that shrink with sample size instead, you’d need roughly four times as many questions.
Step 4: The interval and the call
The uncomfortable conclusion: the interval still crosses zero. Pairing halved the bars and the verdict is still “promising, not proven.” A significance test agrees: under the null hypothesis that neither model is better, each discordant question is a fair coin flip, so B’s 25 wins out of 40 follow . The two-sided exact p-value is 0.154. That’s McNemar’s test, and is not a green light.
Pairing removes noise you never had to accept, and then the data says whatever it says.
Ways to Fool Yourself
1. Judging the comparison from the two separate intervals. The previous post warned that overlapping intervals don’t settle a comparison. With paired data the warning gets sharper. A’s CI is [74.4, 81.6] and B’s is [76.5, 83.5]; they overlap generously, and none of that matters. What decides the comparison is whether the CI on the delta excludes zero, and with positive covariance the delta’s CI can be far tighter than the overlap picture suggests.
2. Comparing scores from different question subsets or benchmark versions. If A ran on v1.2 of the benchmark and B on v1.3, there is no pairing to exploit, and it’s worse than that: the question-mix difference is a confound baked directly into the delta. Rerun on identical questions before comparing at all.
3. Keeping only the aggregate scores. If your eval harness logs “78.0%” and discards per-question outcomes, you paid for the pairing and then deleted it. The 2×2 table cannot be reconstructed from two percentages, and neither can the paired error bar.
4. Running the paired test on 20 slices and reporting the one that hit. Paired tests are still tests. Slice by topic, language, difficulty, and format, and at a 5% significance level one slice in twenty clears the bar by luck, even when the two models are identical everywhere. Decide the slices in advance, or correct for the shopping trip.
5. Expecting pairing to rescue a comparison between strangers. The win scales with agreement. Two models with near-independent errors have nothing for pairing to subtract, and the paired bar will sit almost directly on top of the unpaired one. The playground’s “Strangers” preset makes this painfully concrete.
Try It Yourself
Two simulated models answer the same questions. You control the true accuracy gap and how correlated the models’ per-question outcomes are; that correlation is called φ here, to keep it apart from the other ρs in these series. The grid shows every question once: gray where the two models gave the same verdict, amber where only one of them got it right. Both analyses below the grid are computed from that one table.
Interactive
Same questions, two analyses
Models that share most mistakes (φ = 0.8), B ahead by 2pp. Pairing roughly halves the bar.
One simulated run, as a 2×2 table
| B right | B wrong | A total | |
|---|---|---|---|
| A right | 373both right · a | 17only A · b | 78.0% |
| A wrong | 27only B · c | 83both wrong · d | |
| B total | 80.0% | n = 500 |
Only 44 of 500 questions can move the delta. The rest were answered the same way by both models.
- both right 373
- both wrong 83
- only A 17
- only B 27
Observed Δ = (c − b) / n = (27 − 17) / 500 = +2.0pp
Two 95% intervals from that same table
- Observed Δ
- +2.0
- pp · same for both
- SE unpaired
- 2.58
- pp · √(SE_A² + SE_B²)
- SE paired
- 1.32
- pp · from b and c
- Shrink
- ×1.95
- unpaired ÷ paired
Beyond the scenario buttons, three experiments:
- Hold everything else and sweep φ from 0 to 0.9. The amber squares thin out and only one of the two bars shrinks with them. The unpaired analysis cannot see the correlation at all.
- On Real jump, drag down and find the range where only the paired interval clears zero.
- On Dead heat, reshuffle a few times and watch wobble around zero. Any run where an interval clears zero is your one-in-twenty false alarm in the wild.
Your Turn
Each problem gives you sixteen items scored by two systems, in a new setting each time. The first four steps retrace the worked example. At sixteen items the normal-approximation interval is rough, so a fifth step asks for the count behind the exact McNemar test, the number to trust at this size. Hints are staged: ask once for a pointer, again for the setup with your numbers in it, and a third time for the full answer.
Practice problem
Two Checkpoints
| Coding task | Ckpt 2k | Ckpt 4k |
|---|---|---|
| #1 | Ckpt 2k wrong | Ckpt 4k wrong |
| #2 | Ckpt 2k correct | Ckpt 4k correct |
| #3 | Ckpt 2k correct | Ckpt 4k wrong |
| #4 | Ckpt 2k correct | Ckpt 4k correct |
| #5 | Ckpt 2k wrong | Ckpt 4k wrong |
| #6 | Ckpt 2k wrong | Ckpt 4k wrong |
| #7 | Ckpt 2k wrong | Ckpt 4k correct |
| #8 | Ckpt 2k correct | Ckpt 4k correct |
| #9 | Ckpt 2k correct | Ckpt 4k wrong |
| #10 | Ckpt 2k correct | Ckpt 4k correct |
| #11 | Ckpt 2k wrong | Ckpt 4k wrong |
| #12 | Ckpt 2k correct | Ckpt 4k correct |
| #13 | Ckpt 2k correct | Ckpt 4k correct |
| #14 | Ckpt 2k correct | Ckpt 4k correct |
| #15 | Ckpt 2k correct | Ckpt 4k wrong |
| #16 | Ckpt 2k correct | Ckpt 4k correct |
Guess first
Before you compute anything: will the 95% interval on Δ clear zero?
How Many Questions Do You Need?
Pairing also changes how big an eval has to be. To detect a true gap at the 5% level with 80% power (an 80% chance of the interval clearing zero when the gap is real), the paired design needs about
questions, where is the share of questions the two models split on. The unpaired design needs questions, for each model.
With the opening example’s numbers (A at 78%, 8% of questions discordant):
| True gap | Paired: shared questions | Unpaired: questions per model |
|---|---|---|
| 1pp | 6,272 | 26,490 |
| 2pp | 1,562 | 6,507 |
| 5pp | 244 | 982 |
A 2-point gap between two close checkpoints needs about 1,600 shared questions, four times fewer than the unpaired analysis would ask for. At 500 questions the paired design has only about a 35% chance of clearing zero when the true gap is 2 points, well short of the 80% target, so Step 4’s inconclusive result was the likely outcome. Estimate from any previous pair of runs; it is usually the number you know least about in advance.
Field Notes
Use a paired delta any time two systems answered the same items: two checkpoints, two prompts, two RAG configurations, two judge rubrics scored against the same key. Report three things: , its paired 95% CI, and the discordant counts and . The counts cost you one sentence and let any reader re-derive the rest.
Paired comparison is standard advice. Miller (2024, arXiv:2411.00640) makes paired differences the explicitly recommended analysis for model comparisons in “Adding Error Bars to Evals,” for the covariance reasons above. Dietterich (1998) reached the matching conclusion for classical classifiers: among the tests he benchmarked for comparing two learners on one test set, McNemar’s test was one of the few with acceptable Type I error. The test itself is from McNemar (1947) – only the eval-harness use case is new.
In practice:
- Persist per-item outcomes. A per-question correctness log is the difference between a paired analysis and an apology. You can always recompute aggregate scores from per-item logs later; you cannot recover per-item logs from the aggregates.
- Run both models on identical question versions. Same prompts, same few-shot examples, same scoring script. Any version skew turns into fake delta.
- Bootstrap paired, by resampling items. When the statistic is gnarlier than mean accuracy (pass@k, judge scores, length-normalized anything), resample items with replacement and keep each item’s pair of outcomes together. Resample the two models independently and you have rebuilt the unpaired analysis.
Paired bootstrap: resample items, keep pairs together
import numpy as np
def paired_bootstrap_ci(a, b, reps=10_000, seed=0):
"""a, b: per-item outcomes for models A and B, aligned by item."""
rng = np.random.default_rng(seed)
n = len(a)
deltas = np.empty(reps)
for r in range(reps):
idx = rng.integers(0, n, n) # resample ITEMS; pairs stay together
deltas[r] = b[idx].mean() - a[idx].mean()
return np.percentile(deltas, [2.5, 97.5])Where this sits in the series: the previous post put an error bar on a single score, this one compares two scores, and the next one handles questions that arrive in correlated clusters, which undermines the in everything above.
Code
Python: paired delta, CI, and McNemar exact
import numpy as np
from scipy import stats
def paired_delta(a: np.ndarray, b: np.ndarray):
"""Paired delta and 95% CI from two aligned boolean arrays."""
d = b.astype(float) - a.astype(float) # per-question difference in {-1, 0, +1}
n = len(d)
delta = d.mean()
se = d.std(ddof=1) / np.sqrt(n)
return delta, se, (delta - 1.96 * se, delta + 1.96 * se)
def mcnemar_exact(a: np.ndarray, b: np.ndarray) -> float:
"""Two-sided exact McNemar p-value: discordant wins vs Binomial(b+c, 0.5)."""
b_only = int((a & ~b).sum()) # A right, B wrong
c_only = int((~a & b).sum()) # B right, A wrong
return stats.binomtest(c_only, b_only + c_only, 0.5).pvalue
a = np.array([...], dtype=bool) # model A correct, one entry per question
b = np.array([...], dtype=bool) # model B correct, same question order
delta, se, ci = paired_delta(a, b)
print(f"delta = {delta:+.4f}, SE = {se:.4f}, 95% CI = [{ci[0]:.4f}, {ci[1]:.4f}]")
print(f"McNemar exact p = {mcnemar_exact(a, b):.4f}")d.std(ddof=1) uses the sample standard deviation; the count formula in the post is the ddof=0 version. At eval-sized n the difference is invisible. The paired bootstrap for messier statistics is in the field notes above.
Further Reading
- Miller, E. (2024). “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.” arXiv:2411.00640
- Dietterich, T. G. (1998). “Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms.” Neural Computation 10(7), 1895–1923.
- McNemar, Q. (1947). “Note on the sampling error of the difference between correlated proportions or percentages.” Psychometrika 12(2), 153–157.