Correlation Is Not Calibration

The Judge That Passed the Wrong Test
You want an LLM judge to replace your human graders, so you run the validation study everyone runs: 200 responses, each scored 1–10 by a human expert and by the judge. The result comes back Pearson r = 0.91, and the meeting is already moving on to rollout dates.
Look at the raw scores first. On those same 200 responses the humans averaged 5.8; the judge averaged 7.3. The judge scores everything about a point and a half higher, and r is structurally incapable of noticing. Pearson correlation is invariant to shifting and scaling: add 1.5 to every judge score, or double them all, and r does not move. A judge can rank responses perfectly and still grade on its own private curve.
A note on the word. “Calibration” here is the measurement-science sense: does a 7 from the judge mean what a 7 from a human means, in level and in spread. It is not the machine-learning sense of predicted probabilities matching observed frequencies, though the two share a spirit.
That matters the moment your pipeline consumes scores rather than orderings: an “auto-approve at 8” threshold, a dashboard comparing this month’s judge to last month’s, a comparison across two judges with different curves. For all of those, r = 0.91 is answering a different question than the one you asked.
The Whole Idea in One Formula
The numerator asks whether and move together around their own means; the denominator divides out each rater’s spread. An equivalent reading: standardize both raters (using the standard deviation with in the denominator) and take the average product,
- : the points sit on an upward-sloping line. Any upward slope, any intercept.
- : no linear relationship
- : on a downward-sloping line
The z-scoring is where both the feature and the bug live. Subtracting each rater’s mean and dividing by each rater’s spread makes r unit-free; you can correlate meters with dollars. It also means r deletes two facts before it looks at anything: where each rater’s scale sits, and how wide it is. Those two facts are calibration. Formally, for any constant and any ,
A Worked Example
Five responses, scored by a human () and a judge ():
| Response | Human () | Judge () |
|---|---|---|
| 1 | 2 | 4 |
| 2 | 4 | 5 |
| 3 | 5 | 6 |
| 4 | 6 | 7 |
| 5 | 8 | 9 |
Step 1: Means
, and .
Step 2: Deviations and their products
| Response | Product | ||
|---|---|---|---|
| 1 | |||
| 2 | |||
| 3 | |||
| 4 | |||
| 5 |
Sum of products: .
Squared deviations: and .
Step 3: Pearson r
Nearly perfect. Now the rank version.
Step 4: Spearman ρ
Both raters put the five responses in the same order, so the ranks are 1–5 against 1–5, every rank difference , and :
Exactly 1: the ordering is flawless.
Step 5: The number neither of them saw
ρ says the ordering is perfect, r says nearly-linear, and neither noticed that the judge adds 1.2 points to everything. Concretely: under an “auto-approve at 6” rule, response 3 gets approved by the judge and rejected by the human, on a dataset where and .
Spearman: Correlation on Ranks
Spearman’s ρ is nothing exotic: it is Pearson’s r computed on the ranks instead of the scores. When there are no ties, the arithmetic collapses to a shortcut:
where is the difference between the two ranks for item . For the denominator is , and runs from 0 (identical orderings, ) to 40 (a perfect reversal, ).
Ranks buy you two kinds of robustness. A wild outlier can only move to the end of the rank order; it cannot drag the whole line toward itself the way it drags r. And any relationship that is monotone but nonlinear (a judge that compresses the top of the scale, say) keeps high while r sags. The cost is that ρ is even more forgetful than r: it discards location and scale, like r does, and throws away the sizes of all the gaps as well. Two judges who order 100 responses identically get even if one scores everything 5.0–5.4 and the other 1–10.
A Number That Does See the Shift
If you want one coefficient that drops when the judge runs hot, there is one. Lin’s concordance correlation coefficient asks how close the points sit to the dashed line judge = human, rather than to the best-fit line:
where is the covariance and , the variances (all with in the denominator). The last term in the denominator is the squared bias, so a shift lowers CCC directly, and a slope away from 1 lowers it through the mismatch between and .
On the worked example: , , , and , so
against . CCC equals r only when the two raters share both a mean and a spread. It is a cousin of interval-level alpha from the first post in this series: both measure agreement with the diagonal rather than with a fitted line. They are not interchangeable, though, because they measure disagreement against different baselines, and on badly shifted data they can come out far apart.
CCC is a convenient headline, and it still folds two different failures into one number. When it comes in low, the bias and the slope tell you which failure it was and what to fix, so report them alongside.
Ways to Fool Yourself
1. Reading r as calibration. The theme of this post. Both r and ρ are pure association measures; a +1.5-point systematic shift is mathematically invisible to them. If any downstream decision consumes the scores themselves, you must also report the shift.
2. Range restriction. Validate the judge only on strong responses and r collapses, for a good judge. The playground’s “Faithful judge” (per-item noise ) scores r ≈ 0.96 across the full 1–10 range. Keep only the responses humans scored 7 or above (“Only the good ones”) and the same judge drops to r ≈ 0.71, with its bias still near zero. Nothing about the judge changed; you shrank the signal variance and kept the noise, which is what a curated “high-quality eval set” does to itself.
3. A single outlier can buy you a lot of Pearson r. The “One masterpiece” preset starts from twenty mid-scale pairs with r = 0.34 and ρ = 0.29. Add a single response that both raters score 10 and r jumps to 0.75, while ρ moves only to 0.39. If your validation set includes a couple of unambiguous disasters and masterpieces, they can carry the whole correlation while the mid-pack, where all the real grading decisions live, contributes nothing.
4. Likert ties and the shortcut formula.
Integer scales produce ties – lots of them on a 1–5 scale, plenty on 1–10. The shortcut assumes ties don’t exist and quietly returns the wrong number when they do. Use the tie-aware version: assign tied values the average of the rank positions they span (“midranks”), then compute Pearson on the ranks. That is what scipy.stats.spearmanr does; the shortcut is for hand arithmetic on tie-free data, like the exercise below.
5. Aggregate r can hide per-slice miscalibration. Suppose the judge runs +1.1 points hot on code answers and −0.9 cold on prose. Pooled, the mean bias is roughly zero and r looks healthy: the two slice biases offset each other in the average. Every per-slice decision (routing, thresholds, model comparisons within a category) is still wrong in both directions, so slice before you celebrate.
Try It Yourself
The playground simulates a judge against humans spread across the full 1–10 range, with three dials: a systematic shift (bias), a scale-use factor (slope), and per-item noise. The solid line is the best straight-line fit through the cloud and the dashed line is judge = human. Correlation only measures scatter around the solid line; calibration measures how far the solid line sits from the dashed one. The residual toggle draws both distances.
Interactive
Judge versus human
Tracks the humans with ordinary per-item noise. Now drag Bias to +2: bias and MAE jump, CCC drops, and r and ρ barely move.
- best fit: judge ≈ 0.95 × human + 0.3
- judge = human
Ordering · solid line
- Pearson r
- 0.96
- Spearman ρ
- 0.96
Calibration · dashed line
- Mean bias
- +0.01
- MAE
- 0.60
- Lin's CCC
- 0.96
Correlation only asks how tightly the points hug the solid line, wherever that line goes. Calibration asks whether the solid line is the dashed one.
Experiments to try:
- “Faithful judge”: the baseline, r ≈ 0.95 with ρ right behind it and bias ≈ 0. Now drag Bias to +2: mean bias and MAE jump while r and ρ barely move. They’d hold perfectly still if the scale didn’t stop at 10; the only dent comes from scores clipping at the ceiling.
- “Grade inflator” with residuals drawn to the diagonal: every gap points the same way. Switch the residuals to the fit line and they shrink back to plain noise, which is all r ever sees.
- “Compressor”: slope 0.4 maps the humans’ 1–10 onto roughly 4–7. r stays high because compression is linear, but the solid line has tipped well off the dashed one. An 8 from this judge means almost nothing.
- “Noise machine”: r collapses. No recalibration fixes a judge that can’t rank; the fix is a better judge, not a better offset.
- “Only the good ones” and “One masterpiece” set up pitfalls 2 and 3 above, so you can reproduce their numbers and then move the dials.
- Drop Items to 20 and press Reshuffle a few times. Correlations on small validation sets bounce hard. For scale: r = 0.91 on 200 items has a 95% interval of about [0.88, 0.93], while r = 0.5 on 20 items could be anything from about 0.07 to 0.77. The Error Bars for Evals series is about taking that seriously.
Your Turn
This time the scores are dealt to you: five fresh pairs from two raters, with the ranks already filled in. Before any arithmetic, guess how many points the judge adds on average. Then work through Σd², ρ, and the bias ρ never saw, and compare it with your guess. Three levels of hints wait behind the button if you want them.
Practice problem
LLM Judge Validation
| Response | Human score | Human rank | Judge score | Judge rank |
|---|---|---|---|---|
| #1 | 2 | 1 | 5 | 2 |
| #2 | 3 | 2 | 6 | 3 |
| #3 | 4 | 3 | 4 | 1 |
| #4 | 8 | 4 | 9 | 4 |
| #5 | 10 | 5 | 10 | 5 |
Guess first
Before any ranking: on average, how many points does the LLM judge add to the human expert's score? (Negative if it takes points away.)
Field Notes: What a Judge Validation Report Needs
A single correlation is not a validation. A defensible judge report has four parts:
1. Ranking power. Spearman’s ρ (or Kendall’s τ, which counts concordant vs. discordant pairs and behaves better on small tied samples). This answers: can the judge order responses the way humans do?
2. Calibration. Mean bias (judge − human, signed), MAE, Lin’s CCC if you want one summary number, and a look at how the differences behave across the score range – plot judge − human against the human score, Bland–Altman style. Bland & Altman (1986) made this same argument to medical researchers comparing measurement methods: correlation measures association, not agreement, and two instruments can correlate at 0.99 while disagreeing by a clinically fatal constant. Method-comparison studies dropped r for difference plots four decades ago; judge validation reports can too.
3. Chance-corrected categorical agreement. If the scores get bucketed downstream – accept/reject, severity bands – report agreement on the buckets corrected for chance: Krippendorff’s alpha or Cohen’s kappa. Raw bucket agreement inherits every base-rate illusion covered earlier in this series.
4. Slice-level checks. Repeat 1–3 per category (code vs. prose, short vs. long, topic areas). This is where the known LLM-judge failure modes surface. Zheng et al. (2023), the MT-Bench and Chatbot Arena paper, tested for three of them. Position bias appeared in all three judges they tried: asked to compare two similar answers in both orders, most judges favored the first slot, and even GPT-4, the most consistent, gave the same verdict both ways only 65% of the time. Verbosity bias: a “repetitive list” attack that padded an answer with rephrased copies of its own points fooled GPT-3.5 and Claude-v1 on 91% of test answers and GPT-4 on 9%. Self-enhancement bias (a judge favoring its own model’s outputs) showed up as small win-rate differences that the authors said their data could not settle either way. An aggregate r against humans on a balanced set will show none of these; they only surface in sliced and order-swapped checks.
One pragmatic dodge deserves a mention: pairwise-preference judging (“which response is better, A or B?”) sidesteps calibration entirely. With no score scale there is nothing to drift – part of why Chatbot Arena is built on it. But it does not sidestep the biases above: Zheng et al. measured position bias and verbosity bias in that pairwise setting. Preference judging trades the calibration problem for a counterbalancing problem (swap the order and judge twice).
Code
Python: the whole validation
import numpy as np
from scipy.stats import pearsonr, spearmanr
human = np.array([2, 4, 5, 6, 8])
judge = np.array([4, 5, 6, 7, 9])
r, _ = pearsonr(human, judge)
rho, _ = spearmanr(human, judge) # tie-aware: uses average ranks (midranks)
bias = float(np.mean(judge - human)) # the number r can't see
mae = float(np.mean(np.abs(judge - human)))
ccc = 2 * np.cov(human, judge, bias=True)[0, 1] / (
human.var() + judge.var() + (human.mean() - judge.mean()) ** 2)
print(f"r={r:.3f} rho={rho:.3f} bias={bias:+.2f} mae={mae:.2f} ccc={ccc:.3f}")
# r=0.988 rho=1.000 bias=+1.20 mae=1.20 ccc=0.810spearmanr handles ties with average ranks, so it is safe for Likert-style
integer scores. Do not use the shortcut on tied data.
R
cor(human, judge, method = "pearson")
cor(human, judge, method = "spearman") # midranks for ties
mean(judge - human) # calibration, one lineFurther Reading
- Zheng et al. (2023). “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” – the standard reference for LLM-judge agreement rates, with experiments on position, verbosity, and self-enhancement bias
- Spearman, C. (1904). “The proof and measurement of association between two things.” American Journal of Psychology – the original rank correlation paper
- Bland, J. M. & Altman, D. G. (1986). “Statistical methods for assessing agreement between two methods of clinical measurement.” The Lancet – why correlation is the wrong tool for agreement, and what to plot instead
- Kendall, M. G. (1938). “A new measure of rank correlation.” Biometrika – τ, the concordant-pairs alternative to ρ, with cleaner small-sample behavior under ties