Understanding Krippendorff’s Alpha

The Problem with Raw Agreement
You’re building an LLM evaluation pipeline. Four contractors rate 500 model responses as “Good,” “Okay,” or “Bad.” The dashboard shows 87% pairwise agreement, comfortably above the bar your team set for shipping the labels.
One rater noticed that 70% of responses are “Good” and started marking everything “Good” to speed through the batch. They still agree with the others most of the time – but they’re contributing zero signal.
This is the lazy rater problem, and raw agreement can’t detect it. If 70% of items are genuinely “Good,” two random guessers would agree about 54% of the time just by chance (). We need a number that discounts the agreement chance would have handed us anyway.
The Whole Idea in One Formula
Krippendorff’s Alpha measures the reduction in disagreement relative to random labeling:
where is the disagreement you actually observed, and is the disagreement you’d expect if the same labels had been dealt out at random.
- : Perfect agreement – no observed disagreement
- : Agreement equals chance – raters might as well flip coins
- : Less agreement than chance would produce
Everything else is careful bookkeeping: how to count observed disagreement fairly, and how to estimate what chance alone would produce. The fastest way to internalize both is to compute them by hand, once, on a dataset small enough to hold in your head.
A Worked Example
Three raters evaluate 5 items using categories A, B, and C:
| Item | Rater 1 | Rater 2 | Rater 3 |
|---|---|---|---|
| 1 | A | A | A |
| 2 | B | B | C |
| 3 | A | A | A |
| 4 | B | B | B |
| 5 | C | C | B |
We’ll follow the same five steps you’ll use on real data (and in the practice exercise below).
Step 1: Raw Agreement
First, the simple metric everyone knows – what percentage of rating pairs agree?
For each item with raters, there are unordered pairs. With 3 raters per item, that’s pairs per item. Across 5 items: pairs total.
| Item | Ratings | Pairs | Agreements |
|---|---|---|---|
| 1 | A, A, A | (A,A), (A,A), (A,A) | 3/3 |
| 2 | B, B, C | (B,B), (B,C), (B,C) | 1/3 |
| 3 | A, A, A | (A,A), (A,A), (A,A) | 3/3 |
| 4 | B, B, B | (B,B), (B,B), (B,B) | 3/3 |
| 5 | C, C, B | (C,C), (C,B), (C,B) | 1/3 |
Raw Agreement =
Not bad! But is 73% impressive, or would random labeling do nearly as well? Hold that thought.
Step 2: The Coincidence Matrix
The coincidence matrix counts how often each pair of categories co-occurs across all rater pairs. For each item rated by raters, we enumerate all ordered pairs and weight each by .
With 3 raters per item, each ordered pair gets weight .
Why the 1/(m-1) weighting?
An item with ratings produces ordered pairs. Weighted by , they add up to , so the item puts exactly one unit into the matrix per rating it received. Every rating counts once.
Unweighted, an item would add instead, and the count grows with the square of the number of raters: a five-rater item would outweigh a two-rater item 20 to 2 rather than 5 to 2.
Show item-by-item breakdown
Item 1 (A, A, A): 6 ordered pairs, all (A,A) → contributes to cell (A,A)
Item 2 (B, B, C):
- (B,B): 2 pairs →
- (B,C): 2 pairs →
- (C,B): 2 pairs →
Item 3 (A, A, A): Same as Item 1 → adds 3 to (A,A)
Item 4 (B, B, B): All pairs are (B,B) → adds 3 to (B,B)
Item 5 (C, C, B):
- (C,C): 2 pairs →
- (C,B): 2 pairs →
- (B,C): 2 pairs →
The resulting coincidence matrix :
| A | B | C | Row Total | |
|---|---|---|---|---|
| A | 6 | 0 | 0 | 6 |
| B | 0 | 4 | 2 | 6 |
| C | 0 | 2 | 1 | 3 |
Matrix total = 15 (equals total ratings across all items: 5 items × 3 raters). The diagonal is where agreement lives; everything off it is disagreement.
Marginal totals (row/column sums):
- , ,
- (total pairable values)
Step 3: Observed Disagreement ()
For nominal data (unordered categories), the difference function is:
Observed disagreement is the proportion of coincidences that are disagreements (off-diagonal entries):
From our matrix:
This equals . So far, nothing new – we’ve just reframed agreement as disagreement.
Step 4: Expected Disagreement ()
Here’s the question Krippendorff asked. If we shuffled all ratings randomly while keeping the same category frequencies, what disagreement would we expect by chance?
The probability that two random draws from our pool are different:
Where is the marginal frequency of category .
From our data: , , ,
Random raters with these category frequencies would disagree 68.6% of the time. Our actual raters disagreed only 26.7% of the time – much better than chance!
Step 5: Krippendorff’s Alpha
Our raters avoided 61% of the disagreement that randomly dealt labels would have produced. The raw 73% sounded good; after chance correction we land here:
Below Krippendorff’s 0.67 floor (more on thresholds in the field notes), so these labels aren’t ready to support conclusions yet. The raters clearly share something, and the rubric needs work before anyone builds on it.
Summary of the Example
| Metric | Value | Meaning |
|---|---|---|
| Raw Agreement | 73.3% | How often raters agreed |
| Observed Disagreement () | 26.7% | How often raters disagreed |
| Expected Disagreement () | 68.6% | Disagreement expected by chance |
| Krippendorff’s Alpha | 0.61 | Reliability beyond chance |
Choosing the Difference Function
Step 3 used the nominal difference function: any two different labels count as one full disagreement. That’s one of four choices, and the choice of encodes what “disagreement” means for your data:
| Level | Example | Intuition | |
|---|---|---|---|
| Nominal | if , else | Failure type (refusal / hallucination / formatting) | All disagreements equally bad |
| Ordinal | Good / Okay / Bad, 1–5 stars | Disagreeing Good↔Bad is worse than Good↔Okay | |
| Interval | Temperature, dates | Distance matters, ratios don’t | |
| Ratio | Counts, durations | 2↔4 same as 20↔40 |
Use nominal when the labels have no order. Good/Okay/Bad does have one, so it takes ordinal: a rater who says Okay where a colleague said Good has made a smaller mistake than one who says Bad, and the ordinal function charges less for it. The worked example above stays nominal to keep the arithmetic short.
Ways to Fool Yourself
Five mistakes account for most misreadings of Alpha:
1. Confusing the coincidence matrix with a confusion matrix. A confusion matrix compares predictions to ground truth. The coincidence matrix compares raters to each other – there’s no “correct” answer involved.
2. Forgetting the weighting. Without it, heavily rated items dominate: their pair count grows with the square of the number of raters. The weighting makes each rating count once.
3. Thinking high raw agreement means high Alpha. If categories are imbalanced (e.g., 90% of items are “Good”), random guessing produces high agreement. Alpha adjusts for this, so you can have 85% agreement but .
4. Treating “agreement” as “accuracy.” Alpha measures whether raters agree with each other, not whether they’re correct. Three raters trained on the same flawed rule (say, every confident-sounding answer is “Good”) will agree with each other on the very items where they are all wrong. High Alpha, poor accuracy.
5. Misinterpreting negative Alpha. Alpha a little below zero is ordinary sampling noise around “no agreement beyond chance”, especially on small samples. A clearly negative value means the raters disagree more than chance would, which usually points to a coding error or a rater using the scale upside down.
Try It Yourself
Four simulated raters label items whose true categories you control. Each rater reports the truth with the probability on their slider, and any of them can be told to answer 1 for everything, the shortcut from the opening example. The two bars under the ratings put the formula on screen: the top bar is the disagreement chance would produce, the bottom bar is what your raters actually produce, and α is the share of the top bar they managed to avoid.
Interactive
Four raters, one hidden truth
Four careful raters, balanced categories. Raw agreement and α tell the same story.
Rater skill, the chance of reporting the truth
Guessing at random scores 33% with 3 categories; 0% means always wrong.
The ratings
| Item | Rater 1 | Rater 2 | Rater 3 | Rater 4 |
|---|---|---|---|---|
| 1 | category 2 | category 2 | category 2 | category 2 |
| 2 | category 2 | category 2 | category 2 | category 2 |
| 3 | category 1 | category 1 | category 1 | category 1 |
| 4 | category 3 | category 3 | category 3 | category 3 |
| 5 | category 1 | category 1 | category 1 | category 1 |
| 6 | category 3 | category 3 | category 3 | category 3 |
| 7 | category 3 | category 3 | category 3 | category 3 |
| 8 | category 2 | category 2 | category 1 | category 2 |
| 9 | category 2 | category 3 | category 2 | category 2 |
| 10 | category 2 | category 2 | category 2 | category 3 |
| 11 | category 2 | category 2 | category 2 | category 1 |
| 12 | category 2 | category 1 | category 2 | category 2 |
| 13 | category 1 | category 1 | category 1 | category 1 |
| 14 | category 1 | category 1 | category 1 | category 2 |
| 15 | category 2 | category 2 | category 2 | category 2 |
| 16 | category 1 | category 1 | category 1 | category 1 |
| 17 | category 2 | category 2 | category 2 | category 2 |
| 18 | category 2 | category 2 | category 2 | category 2 |
| 19 | category 1 | category 1 | category 1 | category 1 |
| 20 | category 2 | category 2 | category 2 | category 2 |
- category 1
- category 2
- category 3
Disagreement, measured against chance
α = 1 − Do/De = 1 − 0.24 = 0.76. The raters avoided 76% of the disagreement that labels dealt out at random would have produced.
- Raw agreement
- 85%
- matching pairs
- Krippendorff's α
- 0.76
- nominal
The rubber stamp is the opening scenario: on its default draw, raw agreement reads 78% and α reads 0.34. Prevalence trap is its cousin, where the lopsided categories do the damage without anyone cheating. Near misses switches to five ordered categories where wrong answers land one step away from the truth; flip the difference function between nominal and ordinal and watch α move (0.36 nominal, 0.79 ordinal on the default draw). Rotating pool knocks out a share of the ratings at random, the way a contractor pool with uneven coverage does, and α is still defined (0.82 on the default draw). Kappa can’t do that. For one more, drag two raters to 0% (always wrong) and watch α go negative; one isn’t quite enough to outvote the other three.
Your Turn
Same drill, your numbers. Each problem generates fresh ratings in a new setting, and the five steps mirror the walkthrough. Ask for a hint if a step jams; ask again and it gets more specific.
Practice problem
News Desk
| Article | Editor 1 | Editor 2 | Editor 3 |
|---|---|---|---|
| #1 | B | S | S |
| #2 | S | S | S |
| #3 | P | S | S |
| #4 | B | B | B |
Guess first
Before computing anything: what α do you expect from this table?
Field Notes
How Good Is Good Enough?
Krippendorff (2004, p. 241) recommends:
| Interpretation | |
|---|---|
| Reliable for most purposes | |
| – | Acceptable for tentative conclusions |
| Don’t draw conclusions from the data yet |
Below 0.67 the raters still share something, but too much of what they produce is idiosyncratic to build on; fix the rubric and re-rate. For high-stakes LLM evaluations, aim for .
These cutoffs are stricter than the Landis and Koch labels used for Cohen’s kappa in the next post, where the worked example’s 0.61 would count as “substantial”. Same number, two verdicts. Neither convention is a law; this post uses Krippendorff’s because it was built for deciding whether a set of human labels can be trusted as data.
How Much Should You Trust That Number?
Alpha is a point estimate computed from a sample of items. A small or lopsided sample can produce an impressive Alpha by luck. The practical remedy is a bootstrap confidence interval: resample your items with replacement, compute Alpha for each sample, and take percentiles.
Show Python bootstrapping code
import numpy as np
import krippendorff
def bootstrap_alpha(ratings, n_bootstrap=1000, seed=0):
"""Alpha with a 95% bootstrap CI. ratings: one row per item, one column per rater."""
rng = np.random.default_rng(seed)
n_items = ratings.shape[0]
domain = np.unique(ratings[~np.isnan(ratings)]) # every category in the data
alpha = lambda r: krippendorff.alpha(reliability_data=r.T, # one row per rater
value_domain=domain,
level_of_measurement='nominal')
alphas = []
for _ in range(n_bootstrap):
sample = ratings[rng.integers(0, n_items, n_items)] # resample items
alphas.append(alpha(sample))
return {
'alpha': alpha(ratings),
'ci_low': np.nanpercentile(alphas, 2.5),
'ci_high': np.nanpercentile(alphas, 97.5),
}Passing value_domain keeps a resample that happens to contain one category from raising an error: α comes back as nan for it, and nanpercentile skips it.
| Scenario | Alpha | 95% CI | Interpretation |
|---|---|---|---|
| Tight CI | 0.75 | [0.72, 0.78] | Comfortably tentative; won’t reach 0.80 without changes |
| Wide CI | 0.75 | [0.55, 0.90] | Uncertain; need more data |
| CI crosses threshold | 0.75 | [0.62, 0.85] | Might not even clear 0.67 |
If your 95% CI sits entirely on one side of your decision threshold (0.67 or 0.80), you can make a confident claim; if it spans the threshold, get more data. Report CIs whenever you’re making go/no-go decisions, comparing reliability across conditions, or publishing results others will build on.
Beyond the overall number, examine disagreement item by item. High-disagreement items show you where your rubric is failing, and they make the best training examples for calibration sessions.
Show Python code for per-item analysis
def item_disagreement(ratings):
"""Share of disagreeing rater pairs for each item (rows = items)."""
disagreements = []
for item in ratings:
valid = item[~np.isnan(item)]
if len(valid) < 2:
disagreements.append(np.nan)
continue
pairs = [(valid[i], valid[j]) for i in range(len(valid))
for j in range(i+1, len(valid))]
disagree = sum(1 for a, b in pairs if a != b) / len(pairs)
disagreements.append(disagree)
return disagreementsWhen to Use Alpha (and When Not To)
Alpha earns its keep on annotation quality checks (are your labelers consistent enough to trust?), on codebook validation (can different people apply your categories reliably?), and on benchmarking raters to find who needs more training.
It’s the wrong tool when ground truth exists: with gold labels in hand, measure accuracy against them, because how well the raters agree with each other is beside the point. It’s also the wrong tool for comparing models, since it measures human consistency rather than model quality; use head-to-head preference rates or accuracy for that. And some tasks have legitimate ambiguity. When experts genuinely disagree, the disagreement is information, and driving Alpha up would just paper over it.
Improving Low Alpha
Low reliability is feedback you can act on. Start with the rubric, since ambiguous categories cause most of the disagreement: “Is this response helpful?” is vague; “Does this response directly answer the user’s question?” is testable. If raters keep splitting between “Good” and “Okay,” maybe the scale needs a middle step. Calibration helps too: give raters 20 pre-labeled examples to study before rating, and discuss the edge cases explicitly. Then look for structure in the disagreement itself. Compute Alpha with each rater removed (a rater whose removal sharply improves it needs retraining), and report “clear” and “borderline” cases separately so one blended number doesn’t hide the split.
Why Not Just Use Kappa?
Cohen’s Kappa addresses chance correction too, and when you have exactly two raters it’s less machinery. But Kappa handles exactly two raters, tolerates no missing data, and (without extensions) only speaks nominal. Alpha takes any number of raters, shrugs off missing ratings, and covers ordinal, interval, and ratio data through the difference function. For LLM evals with rotating contractor pools and incomplete coverage, Alpha is the practical choice; the Kappa post has a side-by-side table if you want the full comparison.
Code
Python example
import krippendorff
import numpy as np
# The worked example: one row per item, one column per rater.
# A=1, B=2, C=3; use np.nan for a missing rating.
ratings = np.array([
[1, 1, 1],
[2, 2, 3],
[1, 1, 1],
[2, 2, 2],
[3, 3, 2],
])
# The package expects the transpose: one row per rater.
alpha = krippendorff.alpha(reliability_data=ratings.T,
level_of_measurement='nominal')
print(f"Krippendorff's Alpha: {alpha:.3f}") # 0.611Passing ratings without the .T silently treats items as raters and returns a different number (−0.167 here).
R example
library(irr)
# kripp.alpha also wants one row per rater: transpose an items-by-raters matrix
kripp.alpha(t(ratings), method = "nominal")Further Reading
- Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology
- Hayes & Krippendorff (2007). “Answering the call for a standard reliability measure”
- krippendorff Python package