Understanding Cohen’s Kappa

Understanding Cohen’s Kappa

The Paradox of High Agreement

Your LLM judge and a human reviewer label the same 100 responses as Pass or Fail. They agree on 90. A 90% match rate sounds like the validation can end early.

Look at where the agreement came from. 85 of the responses were obviously fine, and both raters passed them. The other 15 were borderline, and on those the raters agreed only 5 times (both failed them), splitting the remaining 10 evenly: five the human passed and the judge failed, five the other way round.

Both raters said Pass 90 times out of 100. Two raters with that habit, labeling at random, would agree 0.9×0.9+0.1×0.1=82%0.9 \times 0.9 + 0.1 \times 0.1 = 82\% of the time. The 90% headline is eight points above a floor of 82, and the chance-corrected number this post builds toward comes out at κ = 0.44. Raw agreement mixed the judge’s skill with how easy the dataset was.

The Whole Idea in One Formula

Jacob Cohen’s question (1960) was how much better the observed agreement is than chance agreement:

κ=po−pe1−pe\kappa = \frac{p_o - p_e}{1 - p_e}
  • κ\kappa is the agreement coefficient, between −1 and 1
  • pop_o is the observed agreement: the share of items where the two raters gave the same label
  • pep_e is the expected agreement: the share they would match on by chance, given how often each says each label
  • 1−pe1 - p_e is the room left above chance

Read it as the share of that room the raters actually used:

  • κ=1\kappa = 1: perfect agreement
  • κ=0\kappa = 0: agreement no better than chance
  • κ<0\kappa < 0: less agreement than chance would produce

In the opening example, po=0.90p_o = 0.90 and pe=0.82p_e = 0.82, so κ=0.08/0.18=0.44\kappa = 0.08 / 0.18 = 0.44: the raters used less than half the room above chance.

The 2×2 Table

Kappa is computed from a table counting agreements and disagreements:

Rater B: YesRater B: No
Rater A: Yesaa (both yes)bb (A yes, B no)
Rater A: Nocc (A no, B yes)dd (both no)

Observed agreement: po=a+dnp_o = \frac{a + d}{n}

Expected agreement, if the raters labeled independently at their own rates:

pe=P(both yes)+P(both no)=(a+b)(a+c)n2+(c+d)(b+d)n2\begin{aligned} p_e &= P(\text{both yes}) + P(\text{both no}) \\ &= \frac{(a+b)(a+c)}{n^2} + \frac{(c+d)(b+d)}{n^2} \end{aligned}

The two sums a+ba+b and a+ca+c are the marginals: how often each rater says yes, regardless of the other. If Rater A says yes 80% of the time and Rater B 70%, they would both say yes 0.8×0.7=56%0.8 \times 0.7 = 56\% of the time by coincidence alone.

The same table turns up again in this site’s other series. Paired Deltas builds a 2×2 of “model A right / model B right” and reads only the off-diagonal cells bb and cc, because they decide which model is better. Kappa reads the diagonal, because that is where the agreement is.

A Worked Example

You’re validating an LLM judge against a human reviewer. Both label the same 50 responses as Pass or Fail:

Judge: PassJudge: Fail
Human: Pass184
Human: Fail622

Step 1: observed agreement. They agree on 18+22=4018 + 22 = 40 responses: po=4050=80%p_o = \frac{40}{50} = 80\%.

Step 2: the marginals. The human says Pass 18+4=2218+4 = 22 times; the judge, 18+6=2418+6 = 24 times.

Step 3: expected agreement.

pe=22⋅24+28⋅26502=528+7282500=12562500=50.2%\begin{aligned} p_e &= \frac{22 \cdot 24 + 28 \cdot 26}{50^2} \\ &= \frac{528 + 728}{2500} \\ &= \frac{1256}{2500} \\ &= 50.2\% \end{aligned}

Two raters with these base rates would agree half the time with zero skill.

Step 4: kappa.

κ=0.800−0.5021−0.502=0.2980.498≈0.60\kappa = \frac{0.800 - 0.502}{1 - 0.502} = \frac{0.298}{0.498} \approx 0.60

The judge used about 60% of the room that chance left. Useful, and well short of the near-perfect proxy the 80% headline suggested. On the conventional scale in the field notes below, 0.60 sits on the line between “moderate” and “substantial”.

The table holds one more fact that κ doesn’t report. Of the judge’s ten disagreements with the human, six are false passes (the human failed it, the judge passed it) and four are false fails. Kappa is symmetric in the two raters and can’t tell those apart; if false passes are the expensive mistake, you need the counts as well as the coefficient.

Ways to Fool Yourself

1. Quoting raw agreement. The opening example: 90% agreement, κ = 0.44. Whenever one label dominates, raw agreement is mostly a measurement of the base rate.

2. Comparing kappas across datasets with different base rates. Kappa is a property of the raters and the data together. Take a judge that gets 90% of human passes right and 90% of human fails right. On a 50/50 dataset it scores κ = 0.80. Run the same judge on a dataset that is 95% passes and κ drops to 0.43, though the judge hasn’t changed: chance agreement climbed from 50% to 82% and squeezed the room above it. A κ that fell between two eval sets may say more about the sets than the judge. When one class dominates, report the two per-class rates next to κ.

3. Ignoring which way each rater leans. Two raters with identical accuracy get a higher κ when they lean the same way (both a little lenient) than when they lean opposite ways (one strict, one lenient), because opposite leans push the borderline items into the off-diagonal cells. In the playground below, the “Same lean” and “Opposite leans” presets give two pairs of raters the same noise; κ falls from 0.65 to 0.40 and raw agreement from 84% to 68%. Before blaming a rater’s judgment for a low κ, compare the two Pass rates.

4. Treating κ as accuracy when one rater is the reference. If the human is the gold standard, the question is how often the judge is right and in which direction it errs. κ answers a different question (how much the two agree beyond chance) and treats the human’s labels as just as noisy as the judge’s. Report false-pass and false-fail counts, or sensitivity and specificity, alongside it.

5. Trusting a κ from 50 items. Kappa from a small sample has wide error bars. Bootstrap it (code below) before reading the second decimal place.

Try It Yourself

Two simulated raters label the same items yes or no. Each has a lean (how readily they say yes) and their own independent noise. Every item appears as a dot in the quadrant for its pair of answers. On the diagonal, pale dots are agreement that chance would have produced anyway at these yes-rates, and solid dots are agreement beyond it; κ is the solid share of the room chance left over.

Interactive

Two raters, one yes-or-no question

po 87%chance 50%κ 0.73

Half the items are yeses. Chance agreement sits near 50%, so most of what the raters agree on is earned.

Base rate50%
Items200

Rater A

Lean0.00
Noise0.3

Rater B

Lean0.00
Noise0.3

Every item, placed by what the two raters said

B says yesB says noA says yes
both yes91 · chance 55
A yes, B no12 · chance 48
A says no
A no, B yes15 · chance 51
both no82 · chance 46
  • agreement chance would give anyway
  • agreement beyond chance
  • disagreement
  • what chance predicted but didn't happen

Agreement, with chance's share marked off

0%chance 50%100%

The raters agree on 87% of items. Chance alone would get them to 50%, leaving 50 points of headroom. They earned 36 of those points: κ = 0.73.

Observed po
87%
(a + d) / n
Chance pe
50%
from the margins
Cohen's κ
0.73
substantial
κ = 0.73
0
0.20
0.40
0.60
0.80
1

Start with Balanced classes, then switch to Prevalence paradox: raw agreement hardly moves, but the diagonal turns pale. Same lean and Opposite leans give the raters identical noise and differ only in which way they lean; that is pitfall 3 on screen.

Your Turn

Fresh counts each time, and the same four moves as the worked example: agreements, marginals, the chance floor, then κ. Some problems come from lopsided data, where the gap between raw agreement and κ is the point. The hints, if you need them, come in three sizes.

Practice problem

Radiology Reads

A first radiologist and a second radiologist each labeled the same 50 X-rays as Normal or Abnormal. The counts are below. Work down to κ.
Second radiologist: NormalSecond radiologist: Abnormal
First radiologist: Normal204
First radiologist: Abnormal521

n = 50 X-rays

Guess first0/4

Guess first

They agree on 41 of 50. What κ do you expect?

Your guess0.50

Field Notes: Reporting Agreement for a Judge

What to report. κ on its own is hard to read. Put it next to raw agreement and both raters’ base rates, so a reader can see how much of the agreement chance was handing out. If one rater is the reference, add the judge’s false-pass and false-fail counts (or per-class agreement): the worked example’s 6 against 4 is a fact about the judge that κ hides.

Interpretation scales. Landis & Koch (1977) proposed the labels most people quote:

κ\kappaLabel
< 0.00Poor
0.00–0.20Slight
0.21–0.40Fair
0.41–0.60Moderate
0.61–0.80Substantial
0.81–1.00Almost perfect

Treat them as folklore. A κ of 0.60 might be fine for a subjective task like rating creativity and disqualifying for a safety classifier. They are also far more forgiving than the cutoffs Krippendorff recommends for alpha in the previous post, where anything under 0.67 is not ready to draw conclusions from. The same number can be “substantial” under one convention and “not yet usable” under the other, so name the scale when you quote a label.

Error bars. Resample items with replacement, recompute κ each time, and take percentiles. With 50 items, expect an interval about ±0.2 wide.

Ordered buckets. If the labels have an order (Fail / Borderline / Pass), plain κ treats Fail↔Pass and Fail↔Borderline as equally bad. Weighted κ charges less for near misses; it is the two-rater cousin of ordinal alpha.

Kappa or alpha?

FeatureCohen’s KappaKrippendorff’s Alpha
Number of raters2 onlyAny number
Missing dataNot allowedHandles it
Data typesNominal (weighted κ for ordinal)Nominal, ordinal, interval, ratio
Use caseJudge vs. one reference raterAnnotation projects with many raters

Use kappa for quick two-rater checks, like a judge against one human. Use alpha once there are more raters or missing labels.

Code

Python: kappa, the per-class view, and a bootstrap interval
import numpy as np
from sklearn.metrics import cohen_kappa_score, confusion_matrix
 
human = np.array([1, 1, 0, 1, 0, 0, 1, 1, 0, 1])   # 1 = Pass
judge = np.array([1, 0, 0, 1, 0, 1, 1, 1, 0, 1])
 
kappa = cohen_kappa_score(human, judge)
(tn, fp), (fn, tp) = confusion_matrix(human, judge)  # rows = human
print(f"kappa={kappa:.3f}  false passes={fp}  false fails={fn}")
 
def bootstrap_kappa(a, b, reps=2000, seed=0):
    """95% CI for kappa, resampling items so each pair of labels stays together."""
    rng = np.random.default_rng(seed)
    n = len(a)
    ks = []
    for _ in range(reps):
        idx = rng.integers(0, n, n)
        ks.append(cohen_kappa_score(a[idx], b[idx]))
    return np.nanpercentile(ks, [2.5, 97.5])

cohen_kappa_score(..., weights="linear") gives weighted κ for ordered labels.

R
library(irr)
kappa2(data.frame(human, judge))                   # unweighted
kappa2(data.frame(human, judge), weight = "equal") # weighted, for ordered labels

Further Reading

  • Cohen, J. (1960). “A coefficient of agreement for nominal scales.” Educational and Psychological Measurement 20(1), 37–46.
  • Landis, J. R. & Koch, G. G. (1977). “The measurement of observer agreement for categorical data.” Biometrics 33(1), 159–174.
  • Feinstein, A. R. & Cicchetti, D. V. (1990). “High agreement but low kappa: I. The problems of two paradoxes.” Journal of Clinical Epidemiology 43(6), 543–549. The prevalence and bias effects, named.
  • Byrt, T., Bishop, J. & Carlin, J. B. (1993). “Bias, prevalence and kappa.” Journal of Clinical Epidemiology 46(5), 423–429.