Understanding Cohen's Kappa

The Paradox of High Agreement

Two radiologists review 100 chest X-rays, classifying each as "Normal" or "Abnormal." They agree on 90 X-rays—90% agreement! Time to celebrate?

Not quite. Here's the catch: 85 of those X-rays were obviously normal, and both radiologists marked them as such. They're agreeing on the easy cases. On the 15 ambiguous cases, they only agreed on 5.

Raw agreement conflates skill with task difficulty. If 95% of your data falls into one category, two raters who match that base rate while guessing will "agree" about 90% of the time by accident.

Enter Cohen's Kappa

Jacob Cohen's insight (1960) was to ask: how much better is our agreement than random chance?

\kappa = \frac{p_o - p_e}{1 - p_e}

Symbol guide:

κ (kappa) — the agreement coefficient we're computing (range: -1 to 1)
p_o — observed agreement — actual proportion of items where both raters agreed
p_e — expected agreement — agreement we'd expect by pure chance
1 - p_e — maximum possible improvement over chance

Think of it as measuring the "improvement over guessing":

$\kappa = 1$ : Perfect agreement—raters always agree
$\kappa = 0$ : Agreement equals chance—no better than random
$\kappa < 0$ : Worse than chance—systematic disagreement

The 2×2 Contingency Table

Kappa is computed from a simple table counting agreements and disagreements:

	Rater B: Yes	Rater B: No
Rater A: Yes	$a$ (both yes)	$b$ (A yes, B no)
Rater A: No	$c$ (A no, B yes)	$d$ (both no)

Observed agreement: $p_o = \frac{a + d}{n}$

Expected agreement: If raters were independent:

p_e = P(\text{both yes}) + P(\text{both no}) = \frac{(a+b)(a+c)}{n^2} + \frac{(c+d)(b+d)}{n^2}

This accounts for the marginal distributions—if Rater A says "Yes" 80% of the time and Rater B says "Yes" 70% of the time, we'd expect them to both say "Yes" about 56% of the time by chance.

A Worked Example

You're validating an LLM judge against a human reviewer. Both label the same 50 responses as Pass or Fail:

	Judge: Pass	Judge: Fail
Human: Pass	18	4
Human: Fail	6	22

Step 1 — observed agreement. They agree on $18 + 22 = 40$ responses: $p_o = \frac{40}{50} = 80\%$ .

Step 2 — the marginals. The human says Pass $18+4 = 22$ times; the judge, $18+6 = 24$ times.

Step 3 — expected agreement.

p_e = \frac{22 \cdot 24 + 28 \cdot 26}{50^2} = \frac{528 + 728}{2500} = \frac{1256}{2500} = 50.2\%

Two raters with these base rates would agree half the time with zero skill.

Step 4 — kappa.

\kappa = \frac{0.800 - 0.502}{1 - 0.502} = \frac{0.298}{0.498} \approx 0.60

The 80% headline becomes "moderate, brushing substantial." The judge captured about 60% of the headroom that chance left available—useful, but not the near-perfect proxy the raw number implied.

Try It Yourself

Drag the raters to see how their positions affect Kappa. Notice how the class imbalance (controlled by the "Base rate" slider) changes what counts as "chance agreement."

Items

100

Base rate

50% pos

Rater A

Bias0.00

Noise0.5

Rater B

Bias0.00

Noise0.5

A:No, B:Yes

Both Yes

Both No

A:Yes, B:No

Rater A →

Rater B →

	B: Yes	B: No
A: Yes	53	0
A: No	0	47

Observed (p_o)

100%

Expected (p_e)

50%

Cohen's κ

1.00

Almost Perfect

AgreeDisagree

Experiments to try:

Set base rate to 50% (balanced classes). Even modest agreement yields decent Kappa.
Set base rate to 90%. Now watch Kappa plummet even with high raw agreement—the raters are mostly agreeing on obvious cases.
Make both raters biased in the same direction. High agreement, but is it meaningful?

Your Turn

Fresh counts each time, same four moves as the worked example: agreements, marginals, the chance floor, then κ. If you get stuck, help escalates gently—a nudge first, then the setup with your numbers plugged in, and only then the full answer.

Practice problem

Moderation Queue

A senior moderator and a trainee moderator each labeled the same 50 posts as Approve or Remove. The counts are below. Work down to κ.

	Trainee moderator: Approve	Trainee moderator: Remove
Senior moderator: Approve	21	7
Senior moderator: Remove	5	17

n = 50 posts

Step 1 of 40/4

Observed Agreement (pₒ)

Of the 50 posts, how many did the two raters call the same way—and what share is that?

Agreements

pₒ

Why Kappa Can Be Misleading

The Prevalence Problem

When one category dominates, Kappa can be counterintuitively low despite high agreement. With 95% base rate:

Raw agreement might be 94%
But chance agreement is ~90%
So $\kappa = \frac{0.94 - 0.90}{1 - 0.90} = 0.40$ — only "fair" agreement!

This isn't a bug—it's revealing that most of your "agreement" comes from the easy majority class.

The Bias Problem

If both raters have the same bias (e.g., both tend to say "Yes"), Kappa will be higher than if they have opposite biases—even if their accuracy is identical.

Interpretation Guidelines

Landis & Koch (1977) proposed these benchmarks:

$\kappa$	Interpretation
< 0.00	Poor (worse than chance)
0.00–0.20	Slight
0.21–0.40	Fair
0.41–0.60	Moderate
0.61–0.80	Substantial
0.81–1.00	Almost perfect

Caution: These thresholds are context-dependent. A Kappa of 0.60 might be excellent for a subjective task (rating "creativity") but unacceptable for a safety-critical classification.

Kappa vs Krippendorff's Alpha

Feature	Cohen's Kappa	Krippendorff's Alpha
Number of raters	2 only	Any number
Missing data	Not allowed	Handles gracefully
Data types	Nominal (extensions exist)	Nominal, ordinal, interval, ratio
Use case	Simple pairwise agreement	Complex annotation projects

Rule of thumb: Use Kappa for quick two-rater sanity checks. Use Alpha for serious annotation quality measurement.

Code

from sklearn.metrics import cohen_kappa_score
 
rater_a = [1, 1, 0, 1, 0, 0, 1, 1, 0, 1]
rater_b = [1, 0, 0, 1, 0, 1, 1, 1, 0, 1]
 
kappa = cohen_kappa_score(rater_a, rater_b)
print(f"Cohen's Kappa: {kappa:.3f}")

library(irr)
kappa2(data.frame(rater_a, rater_b))