Understanding Krippendorff’s Alpha

Understanding Krippendorff’s Alpha

The Problem with Raw Agreement

You’re building an LLM evaluation pipeline. Four contractors rate 500 model responses as “Good,” “Okay,” or “Bad.” The dashboard shows 87% pairwise agreement, comfortably above the bar your team set for shipping the labels.

One rater noticed that 70% of responses are “Good” and started marking everything “Good” to speed through the batch. They still agree with the others most of the time – but they’re contributing zero signal.

This is the lazy rater problem, and raw agreement can’t detect it. If 70% of items are genuinely “Good,” two random guessers would agree about 54% of the time just by chance (0.72+0.152+0.152≈0.540.7^2 + 0.15^2 + 0.15^2 \approx 0.54). We need a number that discounts the agreement chance would have handed us anyway.

The Whole Idea in One Formula

Krippendorff’s Alpha measures the reduction in disagreement relative to random labeling:

α=1−DoDe\alpha = 1 - \frac{D_o}{D_e}

where DoD_o is the disagreement you actually observed, and DeD_e is the disagreement you’d expect if the same labels had been dealt out at random.

  • α=1\alpha = 1: Perfect agreement – no observed disagreement
  • α=0\alpha = 0: Agreement equals chance – raters might as well flip coins
  • α<0\alpha < 0: Less agreement than chance would produce

Everything else is careful bookkeeping: how to count observed disagreement fairly, and how to estimate what chance alone would produce. The fastest way to internalize both is to compute them by hand, once, on a dataset small enough to hold in your head.

A Worked Example

Three raters evaluate 5 items using categories A, B, and C:

ItemRater 1Rater 2Rater 3
1AAA
2BBC
3AAA
4BBB
5CCB

We’ll follow the same five steps you’ll use on real data (and in the practice exercise below).

Step 1: Raw Agreement

First, the simple metric everyone knows – what percentage of rating pairs agree?

For each item with mm raters, there are (m2)=m(m−1)2\binom{m}{2} = \frac{m(m-1)}{2} unordered pairs. With 3 raters per item, that’s (32)=3\binom{3}{2} = 3 pairs per item. Across 5 items: 5×3=155 \times 3 = 15 pairs total.

ItemRatingsPairsAgreements
1A, A, A(A,A), (A,A), (A,A)3/3
2B, B, C(B,B), (B,C), (B,C)1/3
3A, A, A(A,A), (A,A), (A,A)3/3
4B, B, B(B,B), (B,B), (B,B)3/3
5C, C, B(C,C), (C,B), (C,B)1/3

Raw Agreement = 3+1+3+3+115=1115=73.3%\frac{3+1+3+3+1}{15} = \frac{11}{15} = 73.3\%

Not bad! But is 73% impressive, or would random labeling do nearly as well? Hold that thought.

Step 2: The Coincidence Matrix

The coincidence matrix counts how often each pair of categories co-occurs across all rater pairs. For each item rated by mm raters, we enumerate all m(m−1)m(m-1) ordered pairs and weight each by 1m−1\frac{1}{m-1}.

With 3 raters per item, each ordered pair gets weight 13−1=0.5\frac{1}{3-1} = 0.5.

Why the 1/(m-1) weighting?

An item with mm ratings produces m(m−1)m(m-1) ordered pairs. Weighted by 1m−1\frac{1}{m-1}, they add up to m(m−1)m−1=m\frac{m(m-1)}{m-1} = m, so the item puts exactly one unit into the matrix per rating it received. Every rating counts once.

Unweighted, an item would add m(m−1)m(m-1) instead, and the count grows with the square of the number of raters: a five-rater item would outweigh a two-rater item 20 to 2 rather than 5 to 2.

Show item-by-item breakdown

Item 1 (A, A, A): 6 ordered pairs, all (A,A) → contributes 6×0.5=36 \times 0.5 = 3 to cell (A,A)

Item 2 (B, B, C):

  • (B,B): 2 pairs → 2×0.5=12 \times 0.5 = 1
  • (B,C): 2 pairs → 2×0.5=12 \times 0.5 = 1
  • (C,B): 2 pairs → 2×0.5=12 \times 0.5 = 1

Item 3 (A, A, A): Same as Item 1 → adds 3 to (A,A)

Item 4 (B, B, B): All pairs are (B,B) → adds 3 to (B,B)

Item 5 (C, C, B):

  • (C,C): 2 pairs → 2×0.5=12 \times 0.5 = 1
  • (C,B): 2 pairs → 2×0.5=12 \times 0.5 = 1
  • (B,C): 2 pairs → 2×0.5=12 \times 0.5 = 1

The resulting coincidence matrix OO:

ABCRow Total
A6006
B0426
C0213

Matrix total = 15 (equals total ratings across all items: 5 items × 3 raters). The diagonal is where agreement lives; everything off it is disagreement.

Marginal totals ncn_c (row/column sums):

  • nA=6n_A = 6, nB=6n_B = 6, nC=3n_C = 3
  • n=15n = 15 (total pairable values)

Step 3: Observed Disagreement (DoD_o)

For nominal data (unordered categories), the difference function is:

δ(c,k)={0if c=k1if c≠k\delta(c, k) = \begin{cases} 0 & \text{if } c = k \\ 1 & \text{if } c \neq k \end{cases}

Observed disagreement is the proportion of coincidences that are disagreements (off-diagonal entries):

Do=off-diagonal sumtotal=oAB+oAC+oBA+oBC+oCA+oCBn\begin{aligned} D_o &= \frac{\text{off-diagonal sum}}{\text{total}} \\ &= \frac{o_{AB} + o_{AC} + o_{BA} + o_{BC} + o_{CA} + o_{CB}}{n} \end{aligned}

From our matrix: Do=0+0+0+2+0+215=415=0.267D_o = \frac{0 + 0 + 0 + 2 + 0 + 2}{15} = \frac{4}{15} = 0.267

This equals 1−raw agreement=1−0.733=0.2671 - \text{raw agreement} = 1 - 0.733 = 0.267. So far, nothing new – we’ve just reframed agreement as disagreement.

Step 4: Expected Disagreement (DeD_e)

Here’s the question Krippendorff asked. If we shuffled all ratings randomly while keeping the same category frequencies, what disagreement would we expect by chance?

The probability that two random draws from our pool are different:

De=1−∑cnc(nc−1)n(n−1)D_e = 1 - \frac{\sum_c n_c(n_c - 1)}{n(n-1)}

Where ncn_c is the marginal frequency of category cc.

From our data: nA=6n_A = 6, nB=6n_B = 6, nC=3n_C = 3, n=15n = 15

De=1−6(5)+6(5)+3(2)15(14)=1−30+30+6210=1−66210=0.686\begin{aligned} D_e &= 1 - \frac{6(5) + 6(5) + 3(2)}{15(14)} \\ &= 1 - \frac{30 + 30 + 6}{210} \\ &= 1 - \frac{66}{210} \\ &= 0.686 \end{aligned}

Random raters with these category frequencies would disagree 68.6% of the time. Our actual raters disagreed only 26.7% of the time – much better than chance!

Step 5: Krippendorff’s Alpha

α=1−DoDe=1−0.2670.686=1−0.389=0.611\begin{aligned} \alpha &= 1 - \frac{D_o}{D_e} = 1 - \frac{0.267}{0.686} \\ &= 1 - 0.389 = 0.611 \end{aligned}

Our raters avoided 61% of the disagreement that randomly dealt labels would have produced. The raw 73% sounded good; after chance correction we land here:

Worked exampleα = 0.61
0
0.67
0.80
1

Below Krippendorff’s 0.67 floor (more on thresholds in the field notes), so these labels aren’t ready to support conclusions yet. The raters clearly share something, and the rubric needs work before anyone builds on it.

Summary of the Example

MetricValueMeaning
Raw Agreement73.3%How often raters agreed
Observed Disagreement (DoD_o)26.7%How often raters disagreed
Expected Disagreement (DeD_e)68.6%Disagreement expected by chance
Krippendorff’s Alpha0.61Reliability beyond chance

Choosing the Difference Function

Step 3 used the nominal difference function: any two different labels count as one full disagreement. That’s one of four choices, and the choice of δ\delta encodes what “disagreement” means for your data:

Levelδ(c,k)\delta(c,k)ExampleIntuition
Nominal00 if c=kc=k, else 11Failure type (refusal / hallucination / formatting)All disagreements equally bad
Ordinal(∑i=ckni−nc+nk2)2\left(\sum_{i=c}^{k} n_i - \frac{n_c + n_k}{2}\right)^2Good / Okay / Bad, 1–5 starsDisagreeing Good↔Bad is worse than Good↔Okay
Interval(c−k)2(c-k)^2Temperature, datesDistance matters, ratios don’t
Ratio(c−kc+k)2\left(\frac{c-k}{c+k}\right)^2Counts, durations2↔4 same as 20↔40

Use nominal when the labels have no order. Good/Okay/Bad does have one, so it takes ordinal: a rater who says Okay where a colleague said Good has made a smaller mistake than one who says Bad, and the ordinal function charges less for it. The worked example above stays nominal to keep the arithmetic short.

Ways to Fool Yourself

Five mistakes account for most misreadings of Alpha:

1. Confusing the coincidence matrix with a confusion matrix. A confusion matrix compares predictions to ground truth. The coincidence matrix compares raters to each other – there’s no “correct” answer involved.

2. Forgetting the 1m−1\frac{1}{m-1} weighting. Without it, heavily rated items dominate: their pair count grows with the square of the number of raters. The weighting makes each rating count once.

3. Thinking high raw agreement means high Alpha. If categories are imbalanced (e.g., 90% of items are “Good”), random guessing produces high agreement. Alpha adjusts for this, so you can have 85% agreement but α=0.3\alpha = 0.3.

4. Treating “agreement” as “accuracy.” Alpha measures whether raters agree with each other, not whether they’re correct. Three raters trained on the same flawed rule (say, every confident-sounding answer is “Good”) will agree with each other on the very items where they are all wrong. High Alpha, poor accuracy.

5. Misinterpreting negative Alpha. Alpha a little below zero is ordinary sampling noise around “no agreement beyond chance”, especially on small samples. A clearly negative value means the raters disagree more than chance would, which usually points to a coding error or a rater using the scale upside down.

Try It Yourself

Four simulated raters label items whose true categories you control. Each rater reports the truth with the probability on their slider, and any of them can be told to answer 1 for everything, the shortcut from the opening example. The two bars under the ratings put the formula on screen: the top bar is the disagreement chance would produce, the bottom bar is what your raters actually produce, and α is the share of the top bar they managed to avoid.

Interactive

Four raters, one hidden truth

raw agreement 85%α (nominal) 0.76

Four careful raters, balanced categories. Raw agreement and α tell the same story.

Items20
Share of category 133%
Categories
Difference function
When a rater is wrong, they pick
Missing ratings0%

Rater skill, the chance of reporting the truth

Rater 190%
Rater 288%
Rater 392%
Rater 487%

Guessing at random scores 33% with 3 categories; 0% means always wrong.

The ratings

Ratings by item. Rows are items, columns are raters.
ItemRater 1Rater 2Rater 3Rater 4
1category 2category 2category 2category 2
2category 2category 2category 2category 2
3category 1category 1category 1category 1
4category 3category 3category 3category 3
5category 1category 1category 1category 1
6category 3category 3category 3category 3
7category 3category 3category 3category 3
8category 2category 2category 1category 2
9category 2category 3category 2category 2
10category 2category 2category 2category 3
11category 2category 2category 2category 1
12category 2category 1category 2category 2
13category 1category 1category 1category 1
14category 1category 1category 1category 2
15category 2category 2category 2category 2
16category 1category 1category 1category 1
17category 2category 2category 2category 2
18category 2category 2category 2category 2
19category 1category 1category 1category 1
20category 2category 2category 2category 2
  • category 1
  • category 2
  • category 3

Disagreement, measured against chance

Chance would produce
0.622
Your raters produce
0.150

α = 1 − Do/De = 1 − 0.24 = 0.76. The raters avoided 76% of the disagreement that labels dealt out at random would have produced.

Raw agreement
85%
matching pairs
Krippendorff's α
0.76
nominal
α = 0.76
0
0.67
0.80
1

The rubber stamp is the opening scenario: on its default draw, raw agreement reads 78% and α reads 0.34. Prevalence trap is its cousin, where the lopsided categories do the damage without anyone cheating. Near misses switches to five ordered categories where wrong answers land one step away from the truth; flip the difference function between nominal and ordinal and watch α move (0.36 nominal, 0.79 ordinal on the default draw). Rotating pool knocks out a share of the ratings at random, the way a contractor pool with uneven coverage does, and α is still defined (0.82 on the default draw). Kappa can’t do that. For one more, drag two raters to 0% (always wrong) and watch α go negative; one isn’t quite enough to outvote the other three.

Your Turn

Same drill, your numbers. Each problem generates fresh ratings in a new setting, and the five steps mirror the walkthrough. Ask for a hint if a step jams; ask again and it gets more specific.

Practice problem

News Desk

3 editors rated 4 articles: P Politics, B Business, S Science. Work down to α, one step at a time.
ArticleEditor 1Editor 2Editor 3
#1BSS
#2SSS
#3PSS
#4BBB
Guess first0/5

Guess first

Before computing anything: what α do you expect from this table?

Your guess0.50

Field Notes

How Good Is Good Enough?

Krippendorff (2004, p. 241) recommends:

α\alphaInterpretation
≥0.80\geq 0.80Reliable for most purposes
0.670.67–0.800.80Acceptable for tentative conclusions
<0.67< 0.67Don’t draw conclusions from the data yet

Below 0.67 the raters still share something, but too much of what they produce is idiosyncratic to build on; fix the rubric and re-rate. For high-stakes LLM evaluations, aim for α≥0.80\alpha \geq 0.80.

These cutoffs are stricter than the Landis and Koch labels used for Cohen’s kappa in the next post, where the worked example’s 0.61 would count as “substantial”. Same number, two verdicts. Neither convention is a law; this post uses Krippendorff’s because it was built for deciding whether a set of human labels can be trusted as data.

How Much Should You Trust That Number?

Alpha is a point estimate computed from a sample of items. A small or lopsided sample can produce an impressive Alpha by luck. The practical remedy is a bootstrap confidence interval: resample your items with replacement, compute Alpha for each sample, and take percentiles.

Show Python bootstrapping code
import numpy as np
import krippendorff
 
def bootstrap_alpha(ratings, n_bootstrap=1000, seed=0):
    """Alpha with a 95% bootstrap CI. ratings: one row per item, one column per rater."""
    rng = np.random.default_rng(seed)
    n_items = ratings.shape[0]
    domain = np.unique(ratings[~np.isnan(ratings)])  # every category in the data
    alpha = lambda r: krippendorff.alpha(reliability_data=r.T,  # one row per rater
                                         value_domain=domain,
                                         level_of_measurement='nominal')
    alphas = []
    for _ in range(n_bootstrap):
        sample = ratings[rng.integers(0, n_items, n_items)]  # resample items
        alphas.append(alpha(sample))
    return {
        'alpha': alpha(ratings),
        'ci_low': np.nanpercentile(alphas, 2.5),
        'ci_high': np.nanpercentile(alphas, 97.5),
    }

Passing value_domain keeps a resample that happens to contain one category from raising an error: α comes back as nan for it, and nanpercentile skips it.

ScenarioAlpha95% CIInterpretation
Tight CI0.75[0.72, 0.78]Comfortably tentative; won’t reach 0.80 without changes
Wide CI0.75[0.55, 0.90]Uncertain; need more data
CI crosses threshold0.75[0.62, 0.85]Might not even clear 0.67

If your 95% CI sits entirely on one side of your decision threshold (0.67 or 0.80), you can make a confident claim; if it spans the threshold, get more data. Report CIs whenever you’re making go/no-go decisions, comparing reliability across conditions, or publishing results others will build on.

Beyond the overall number, examine disagreement item by item. High-disagreement items show you where your rubric is failing, and they make the best training examples for calibration sessions.

Show Python code for per-item analysis
def item_disagreement(ratings):
    """Share of disagreeing rater pairs for each item (rows = items)."""
    disagreements = []
    for item in ratings:
        valid = item[~np.isnan(item)]
        if len(valid) < 2:
            disagreements.append(np.nan)
            continue
        pairs = [(valid[i], valid[j]) for i in range(len(valid))
                 for j in range(i+1, len(valid))]
        disagree = sum(1 for a, b in pairs if a != b) / len(pairs)
        disagreements.append(disagree)
    return disagreements

When to Use Alpha (and When Not To)

Alpha earns its keep on annotation quality checks (are your labelers consistent enough to trust?), on codebook validation (can different people apply your categories reliably?), and on benchmarking raters to find who needs more training.

It’s the wrong tool when ground truth exists: with gold labels in hand, measure accuracy against them, because how well the raters agree with each other is beside the point. It’s also the wrong tool for comparing models, since it measures human consistency rather than model quality; use head-to-head preference rates or accuracy for that. And some tasks have legitimate ambiguity. When experts genuinely disagree, the disagreement is information, and driving Alpha up would just paper over it.

Improving Low Alpha

Low reliability is feedback you can act on. Start with the rubric, since ambiguous categories cause most of the disagreement: “Is this response helpful?” is vague; “Does this response directly answer the user’s question?” is testable. If raters keep splitting between “Good” and “Okay,” maybe the scale needs a middle step. Calibration helps too: give raters 20 pre-labeled examples to study before rating, and discuss the edge cases explicitly. Then look for structure in the disagreement itself. Compute Alpha with each rater removed (a rater whose removal sharply improves it needs retraining), and report “clear” and “borderline” cases separately so one blended number doesn’t hide the split.

Why Not Just Use Kappa?

Cohen’s Kappa addresses chance correction too, and when you have exactly two raters it’s less machinery. But Kappa handles exactly two raters, tolerates no missing data, and (without extensions) only speaks nominal. Alpha takes any number of raters, shrugs off missing ratings, and covers ordinal, interval, and ratio data through the difference function. For LLM evals with rotating contractor pools and incomplete coverage, Alpha is the practical choice; the Kappa post has a side-by-side table if you want the full comparison.

Code

Python example
import krippendorff
import numpy as np
 
# The worked example: one row per item, one column per rater.
# A=1, B=2, C=3; use np.nan for a missing rating.
ratings = np.array([
    [1, 1, 1],
    [2, 2, 3],
    [1, 1, 1],
    [2, 2, 2],
    [3, 3, 2],
])
 
# The package expects the transpose: one row per rater.
alpha = krippendorff.alpha(reliability_data=ratings.T,
                           level_of_measurement='nominal')
print(f"Krippendorff's Alpha: {alpha:.3f}")   # 0.611

Passing ratings without the .T silently treats items as raters and returns a different number (−0.167 here).

R example
library(irr)
# kripp.alpha also wants one row per rater: transpose an items-by-raters matrix
kripp.alpha(t(ratings), method = "nominal")

Further Reading

  • Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology
  • Hayes & Krippendorff (2007). “Answering the call for a standard reliability measure”
  • krippendorff Python package