Understanding Cohen's Kappa

The Paradox of High Agreement

Your LLM judge and a human reviewer label the same 100 responses as Pass or Fail. They agree on 90. A 90% match rate sounds like the validation can end early.

The catch: 85 of those responses were obviously fine, and both raters passed them. They're agreeing on the easy cases. On the 15 borderline responses, they only agreed on 5.

Raw agreement conflates skill with task difficulty. If 95% of your data falls into one category, two raters who match that base rate while guessing will "agree" about 90% of the time by accident.

Enter Cohen's Kappa

Jacob Cohen's insight (1960) was to ask: how much better is our agreement than random chance?

\kappa = \frac{p_o - p_e}{1 - p_e}

Symbol guide:

κ (kappa): the agreement coefficient we're computing (range: -1 to 1)
p_o: observed agreement, the actual proportion of items where both raters agreed
p_e: expected agreement, the agreement we'd expect by pure chance
1 - p_e: the maximum possible improvement over chance

Think of it as measuring the "improvement over guessing":

$\kappa = 1$ : Perfect agreement. The raters always agree.
$\kappa = 0$ : Agreement equals chance. No better than random.
$\kappa < 0$ : Worse than chance, which means systematic disagreement.

The 2×2 Contingency Table

Kappa is computed from a simple table counting agreements and disagreements:

	Rater B: Yes	Rater B: No
Rater A: Yes	$a$ (both yes)	$b$ (A yes, B no)
Rater A: No	$c$ (A no, B yes)	$d$ (both no)

Observed agreement: $p_o = \frac{a + d}{n}$

Expected agreement: If raters were independent:

p_e = P(\text{both yes}) + P(\text{both no}) = \frac{(a+b)(a+c)}{n^2} + \frac{(c+d)(b+d)}{n^2}

This accounts for the marginal distributions. If Rater A says "Yes" 80% of the time and Rater B says "Yes" 70% of the time, we'd expect them to both say "Yes" about 56% of the time by chance.

A Worked Example

You're validating an LLM judge against a human reviewer. Both label the same 50 responses as Pass or Fail:

	Judge: Pass	Judge: Fail
Human: Pass	18	4
Human: Fail	6	22

Step 1: observed agreement. They agree on $18 + 22 = 40$ responses: $p_o = \frac{40}{50} = 80\%$ .

Step 2: the marginals. The human says Pass $18+4 = 22$ times; the judge, $18+6 = 24$ times.

Step 3: expected agreement.

p_e = \frac{22 \cdot 24 + 28 \cdot 26}{50^2} = \frac{528 + 728}{2500} = \frac{1256}{2500} = 50.2\%

Two raters with these base rates would agree half the time with zero skill.

Step 4: kappa.

\kappa = \frac{0.800 - 0.502}{1 - 0.502} = \frac{0.298}{0.498} \approx 0.60

The 80% headline becomes "moderate, brushing substantial." The judge captured about 60% of the headroom that chance left available: useful, but not the near-perfect proxy the raw number implied.

Try It Yourself

Drag the raters to see how their positions affect Kappa. Notice how the class imbalance (controlled by the "Base rate" slider) changes what counts as "chance agreement."

Items

100

Base rate

50% pos

Rater A

Bias0.00

Noise0.5

Rater B

Bias0.00

Noise0.5

A:No, B:Yes

Both Yes

Both No

A:Yes, B:No

Rater A →

Rater B →

	B: Yes	B: No
A: Yes	53	0
A: No	0	47

Observed (p_o)

100%

Expected (p_e)

50%

Cohen's κ

1.00

Almost Perfect

AgreeDisagree

Experiments to try:

Set base rate to 50% (balanced classes). Even modest agreement yields decent Kappa.
Set base rate to 90%. Now watch Kappa plummet even with high raw agreement, because the raters are mostly agreeing on obvious cases.
Make both raters biased in the same direction. High agreement, but is it meaningful?

Your Turn

Fresh counts each time, same four moves as the worked example: agreements, marginals, the chance floor, then κ. The hints, if you need them, come in three sizes.

Practice problem

Moderation Queue

A senior moderator and a trainee moderator each labeled the same 50 posts as Approve or Remove. The counts are below. Work down to κ.

	Trainee moderator: Approve	Trainee moderator: Remove
Senior moderator: Approve	21	7
Senior moderator: Remove	5	17

n = 50 posts

Step 1 of 40/4

Observed Agreement (pₒ)

Of the 50 posts, how many did the two raters call the same way--and what share is that?

Agreements

pₒ

Why Kappa Can Be Misleading

The Prevalence Problem

When one category dominates, Kappa can be counterintuitively low despite high agreement. With 95% base rate:

Raw agreement might be 94%
But chance agreement is ~90%
So $\kappa = \frac{0.94 - 0.90}{1 - 0.90} = 0.40$ , which Landis and Koch would file under merely "fair."

Kappa is doing its job here: most of the "agreement" was coming from the easy majority class, and the low score says so.

The Bias Problem

If both raters have the same bias (e.g., both tend to say "Yes"), Kappa will be higher than if they have opposite biases, even when their accuracy is identical.

Interpretation Guidelines

Landis & Koch (1977) proposed these benchmarks:

$\kappa$	Interpretation
< 0.00	Poor (worse than chance)
0.00–0.20	Slight
0.21–0.40	Fair
0.41–0.60	Moderate
0.61–0.80	Substantial
0.81–1.00	Almost perfect

Treat the benchmarks as folklore rather than law. A Kappa of 0.60 might be fine for a subjective task like rating "creativity" and disqualifying for a safety-critical classifier.

Kappa vs Krippendorff's Alpha

Feature	Cohen's Kappa	Krippendorff's Alpha
Number of raters	2 only	Any number
Missing data	Not allowed	Handles gracefully
Data types	Nominal (extensions exist)	Nominal, ordinal, interval, ratio
Use case	Simple pairwise agreement	Complex annotation projects

Rule of thumb: Use Kappa for quick two-rater sanity checks. Use Alpha for serious annotation quality measurement.

Code

from sklearn.metrics import cohen_kappa_score
 
rater_a = [1, 1, 0, 1, 0, 0, 1, 1, 0, 1]
rater_b = [1, 0, 0, 1, 0, 1, 1, 1, 0, 1]
 
kappa = cohen_kappa_score(rater_a, rater_b)
print(f"Cohen's Kappa: {kappa:.3f}")

library(irr)
kappa2(data.frame(rater_a, rater_b))