Understanding Cohen’s Kappa

The Paradox of High Agreement
Your LLM judge and a human reviewer label the same 100 responses as Pass or Fail. They agree on 90. A 90% match rate sounds like the validation can end early.
Look at where the agreement came from. 85 of the responses were obviously fine, and both raters passed them. The other 15 were borderline, and on those the raters agreed only 5 times (both failed them), splitting the remaining 10 evenly: five the human passed and the judge failed, five the other way round.
Both raters said Pass 90 times out of 100. Two raters with that habit, labeling at random, would agree of the time. The 90% headline is eight points above a floor of 82, and the chance-corrected number this post builds toward comes out at κ = 0.44. Raw agreement mixed the judge’s skill with how easy the dataset was.
The Whole Idea in One Formula
Jacob Cohen’s question (1960) was how much better the observed agreement is than chance agreement:
- is the agreement coefficient, between −1 and 1
- is the observed agreement: the share of items where the two raters gave the same label
- is the expected agreement: the share they would match on by chance, given how often each says each label
- is the room left above chance
Read it as the share of that room the raters actually used:
- : perfect agreement
- : agreement no better than chance
- : less agreement than chance would produce
In the opening example, and , so : the raters used less than half the room above chance.
The 2×2 Table
Kappa is computed from a table counting agreements and disagreements:
| Rater B: Yes | Rater B: No | |
|---|---|---|
| Rater A: Yes | (both yes) | (A yes, B no) |
| Rater A: No | (A no, B yes) | (both no) |
Observed agreement:
Expected agreement, if the raters labeled independently at their own rates:
The two sums and are the marginals: how often each rater says yes, regardless of the other. If Rater A says yes 80% of the time and Rater B 70%, they would both say yes of the time by coincidence alone.
The same table turns up again in this site’s other series. Paired Deltas builds a 2×2 of “model A right / model B right” and reads only the off-diagonal cells and , because they decide which model is better. Kappa reads the diagonal, because that is where the agreement is.
A Worked Example
You’re validating an LLM judge against a human reviewer. Both label the same 50 responses as Pass or Fail:
| Judge: Pass | Judge: Fail | |
|---|---|---|
| Human: Pass | 18 | 4 |
| Human: Fail | 6 | 22 |
Step 1: observed agreement. They agree on responses: .
Step 2: the marginals. The human says Pass times; the judge, times.
Step 3: expected agreement.
Two raters with these base rates would agree half the time with zero skill.
Step 4: kappa.
The judge used about 60% of the room that chance left. Useful, and well short of the near-perfect proxy the 80% headline suggested. On the conventional scale in the field notes below, 0.60 sits on the line between “moderate” and “substantial”.
The table holds one more fact that κ doesn’t report. Of the judge’s ten disagreements with the human, six are false passes (the human failed it, the judge passed it) and four are false fails. Kappa is symmetric in the two raters and can’t tell those apart; if false passes are the expensive mistake, you need the counts as well as the coefficient.
Ways to Fool Yourself
1. Quoting raw agreement. The opening example: 90% agreement, κ = 0.44. Whenever one label dominates, raw agreement is mostly a measurement of the base rate.
2. Comparing kappas across datasets with different base rates. Kappa is a property of the raters and the data together. Take a judge that gets 90% of human passes right and 90% of human fails right. On a 50/50 dataset it scores κ = 0.80. Run the same judge on a dataset that is 95% passes and κ drops to 0.43, though the judge hasn’t changed: chance agreement climbed from 50% to 82% and squeezed the room above it. A κ that fell between two eval sets may say more about the sets than the judge. When one class dominates, report the two per-class rates next to κ.
3. Ignoring which way each rater leans. Two raters with identical accuracy get a higher κ when they lean the same way (both a little lenient) than when they lean opposite ways (one strict, one lenient), because opposite leans push the borderline items into the off-diagonal cells. In the playground below, the “Same lean” and “Opposite leans” presets give two pairs of raters the same noise; κ falls from 0.65 to 0.40 and raw agreement from 84% to 68%. Before blaming a rater’s judgment for a low κ, compare the two Pass rates.
4. Treating κ as accuracy when one rater is the reference. If the human is the gold standard, the question is how often the judge is right and in which direction it errs. κ answers a different question (how much the two agree beyond chance) and treats the human’s labels as just as noisy as the judge’s. Report false-pass and false-fail counts, or sensitivity and specificity, alongside it.
5. Trusting a κ from 50 items. Kappa from a small sample has wide error bars. Bootstrap it (code below) before reading the second decimal place.
Try It Yourself
Two simulated raters label the same items yes or no. Each has a lean (how readily they say yes) and their own independent noise. Every item appears as a dot in the quadrant for its pair of answers. On the diagonal, pale dots are agreement that chance would have produced anyway at these yes-rates, and solid dots are agreement beyond it; κ is the solid share of the room chance left over.
Interactive
Two raters, one yes-or-no question
Half the items are yeses. Chance agreement sits near 50%, so most of what the raters agree on is earned.
Rater A
Rater B
Every item, placed by what the two raters said
- agreement chance would give anyway
- agreement beyond chance
- disagreement
- what chance predicted but didn't happen
Agreement, with chance's share marked off
The raters agree on 87% of items. Chance alone would get them to 50%, leaving 50 points of headroom. They earned 36 of those points: κ = 0.73.
- Observed po
- 87%
- (a + d) / n
- Chance pe
- 50%
- from the margins
- Cohen's κ
- 0.73
- substantial
Start with Balanced classes, then switch to Prevalence paradox: raw agreement hardly moves, but the diagonal turns pale. Same lean and Opposite leans give the raters identical noise and differ only in which way they lean; that is pitfall 3 on screen.
Your Turn
Fresh counts each time, and the same four moves as the worked example: agreements, marginals, the chance floor, then κ. Some problems come from lopsided data, where the gap between raw agreement and κ is the point. The hints, if you need them, come in three sizes.
Practice problem
Radiology Reads
| Second radiologist: Normal | Second radiologist: Abnormal | |
|---|---|---|
| First radiologist: Normal | 20 | 4 |
| First radiologist: Abnormal | 5 | 21 |
n = 50 X-rays
Guess first
They agree on 41 of 50. What κ do you expect?
Field Notes: Reporting Agreement for a Judge
What to report. κ on its own is hard to read. Put it next to raw agreement and both raters’ base rates, so a reader can see how much of the agreement chance was handing out. If one rater is the reference, add the judge’s false-pass and false-fail counts (or per-class agreement): the worked example’s 6 against 4 is a fact about the judge that κ hides.
Interpretation scales. Landis & Koch (1977) proposed the labels most people quote:
| Label | |
|---|---|
| < 0.00 | Poor |
| 0.00–0.20 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.00 | Almost perfect |
Treat them as folklore. A κ of 0.60 might be fine for a subjective task like rating creativity and disqualifying for a safety classifier. They are also far more forgiving than the cutoffs Krippendorff recommends for alpha in the previous post, where anything under 0.67 is not ready to draw conclusions from. The same number can be “substantial” under one convention and “not yet usable” under the other, so name the scale when you quote a label.
Error bars. Resample items with replacement, recompute κ each time, and take percentiles. With 50 items, expect an interval about ±0.2 wide.
Ordered buckets. If the labels have an order (Fail / Borderline / Pass), plain κ treats Fail↔Pass and Fail↔Borderline as equally bad. Weighted κ charges less for near misses; it is the two-rater cousin of ordinal alpha.
Kappa or alpha?
| Feature | Cohen’s Kappa | Krippendorff’s Alpha |
|---|---|---|
| Number of raters | 2 only | Any number |
| Missing data | Not allowed | Handles it |
| Data types | Nominal (weighted κ for ordinal) | Nominal, ordinal, interval, ratio |
| Use case | Judge vs. one reference rater | Annotation projects with many raters |
Use kappa for quick two-rater checks, like a judge against one human. Use alpha once there are more raters or missing labels.
Code
Python: kappa, the per-class view, and a bootstrap interval
import numpy as np
from sklearn.metrics import cohen_kappa_score, confusion_matrix
human = np.array([1, 1, 0, 1, 0, 0, 1, 1, 0, 1]) # 1 = Pass
judge = np.array([1, 0, 0, 1, 0, 1, 1, 1, 0, 1])
kappa = cohen_kappa_score(human, judge)
(tn, fp), (fn, tp) = confusion_matrix(human, judge) # rows = human
print(f"kappa={kappa:.3f} false passes={fp} false fails={fn}")
def bootstrap_kappa(a, b, reps=2000, seed=0):
"""95% CI for kappa, resampling items so each pair of labels stays together."""
rng = np.random.default_rng(seed)
n = len(a)
ks = []
for _ in range(reps):
idx = rng.integers(0, n, n)
ks.append(cohen_kappa_score(a[idx], b[idx]))
return np.nanpercentile(ks, [2.5, 97.5])cohen_kappa_score(..., weights="linear") gives weighted κ for ordered labels.
R
library(irr)
kappa2(data.frame(human, judge)) # unweighted
kappa2(data.frame(human, judge), weight = "equal") # weighted, for ordered labelsFurther Reading
- Cohen, J. (1960). “A coefficient of agreement for nominal scales.” Educational and Psychological Measurement 20(1), 37–46.
- Landis, J. R. & Koch, G. G. (1977). “The measurement of observer agreement for categorical data.” Biometrics 33(1), 159–174.
- Feinstein, A. R. & Cicchetti, D. V. (1990). “High agreement but low kappa: I. The problems of two paradoxes.” Journal of Clinical Epidemiology 43(6), 543–549. The prevalence and bias effects, named.
- Byrt, T., Bishop, J. & Carlin, J. B. (1993). “Bias, prevalence and kappa.” Journal of Clinical Epidemiology 46(5), 423–429.