Understanding Krippendorff's Alpha

The Problem with Raw Agreement

You're building an LLM evaluation pipeline. Four contractors rate 500 model responses as "Good," "Okay," or "Bad." The dashboard shows 87% pairwise agreement. Ship it?

Not so fast. One rater noticed that 70% of responses are "Good" and started marking everything "Good" to speed through the batch. They still agree with the others most of the time—but they're contributing zero signal.

This is the lazy rater problem, and raw agreement can't detect it. If 70% of items are genuinely "Good," two random guessers would agree about 54% of the time just by chance ($0.7^2 + 0.15^2 + 0.15^2 \approx 0.54$). We need a metric that asks: how much better is our agreement than random chance?

The Whole Idea in One Formula

Krippendorff's Alpha measures the reduction in disagreement relative to random labeling:

where $D_o$ is the disagreement you actually observed, and $D_e$ is the disagreement you'd expect if the same labels had been dealt out at random.

• $\alpha = 1$: Perfect agreement—no observed disagreement
• $\alpha = 0$: Agreement equals chance—raters might as well flip coins
• $\alpha < 0$: Systematic disagreement—raters actively contradict each other

Everything else is careful bookkeeping: how to count observed disagreement fairly, and how to estimate what chance alone would produce. The fastest way to internalize both is to compute them by hand, once, on a dataset small enough to hold in your head.

A Worked Example

Three raters evaluate 5 items using categories A, B, and C:

We'll follow the same five steps you'll use on real data (and in the practice exercise below).

Step 1: Raw Agreement

First, the simple metric everyone knows—what percentage of rating pairs agree?

For each item with $m$ raters, there are $\binom{m}{2} = \frac{m(m-1)}{2}$ unordered pairs. With 3 raters per item, that's $\binom{3}{2} = 3$ pairs per item. Across 5 items: $5 \times 3 = 15$ pairs total.

Raw Agreement = $\frac{3+1+3+3+1}{15} = \frac{11}{15} = 73.3\%$

Not bad! But is 73% impressive, or would random labeling do nearly as well? Hold that thought.

Step 2: The Coincidence Matrix

The coincidence matrix counts how often each pair of categories co-occurs across all rater pairs. For each item rated by $m$ raters, we enumerate all $m(m-1)$ ordered pairs and weight each by $\frac{1}{m-1}$.

With 3 raters per item, each ordered pair gets weight $\frac{1}{3-1} = 0.5$.

The resulting coincidence matrix $O$:

Matrix total = 15 (equals total ratings across all items: 5 items × 3 raters). The diagonal is where agreement lives; everything off it is disagreement.

Marginal totals $n_c$ (row/column sums):

• $n_A = 6$, $n_B = 6$, $n_C = 3$
• $n = 15$ (total pairable values)

Step 3: Observed Disagreement ($D_o$)

For nominal data (unordered categories), the difference function is:

Observed disagreement is the proportion of coincidences that are disagreements (off-diagonal entries):

From our matrix: $D_o = \frac{0 + 0 + 0 + 2 + 0 + 2}{15} = \frac{4}{15} = 0.267$

This equals $1 - \text{raw agreement} = 1 - 0.733 = 0.267$. So far, nothing new—we've just reframed agreement as disagreement.

Step 4: Expected Disagreement ($D_e$)

Here's Krippendorff's key insight. If we shuffled all ratings randomly while keeping the same category frequencies, what disagreement would we expect by chance?

The probability that two random draws from our pool are different:

Where $n_c$ is the marginal frequency of category $c$.

From our data: $n_A = 6$, $n_B = 6$, $n_C = 3$, $n = 15$

Key insight: Random raters with these category frequencies would disagree 68.6% of the time. Our actual raters disagreed only 26.7% of the time—much better than chance!

Step 5: Krippendorff's Alpha

Interpretation: Our raters captured 61% of the maximum possible agreement beyond chance. The raw 73% sounded good, but after chance correction we land here:

Moderate reliability—acceptable for exploratory work, but not for high-stakes decisions.

Summary of the Example

Ways to Fool Yourself

Five mistakes account for most misreadings of Alpha:

1. Confusing the coincidence matrix with a confusion matrix. A confusion matrix compares predictions to ground truth. The coincidence matrix compares raters to each other—there's no "correct" answer involved.

2. Forgetting the $\frac{1}{m-1}$ weighting. Without it, items with more raters dominate the calculation. The weighting ensures each item contributes equally, not each pair.

3. Thinking high raw agreement means high Alpha. If categories are imbalanced (e.g., 90% of items are "Good"), random guessing produces high agreement. Alpha adjusts for this, so you can have 85% agreement but $\alpha = 0.3$.

4. Confusing "agreement" with "accuracy." Alpha measures whether raters agree with each other, not whether they're correct. Three raters who all wrongly label "Bad" responses as "Good" will have perfect Alpha and zero accuracy.

5. Misinterpreting negative Alpha. Negative Alpha doesn't mean "bad raters"—it means raters systematically disagree more than chance. This is rare and usually indicates a coding error, inverted scales, or genuinely adversarial labeling.

Try It Yourself

The playground below simulates raters with controllable accuracy. Each row has a hidden ground truth; drag the sliders to see how rater quality affects Alpha.

Experiments to try:

• "LLM Eval: The Lazy Rater" — One rater at 33% (random guessing for 3 categories). Watch Alpha drop despite decent raw agreement.
• "Pure Random" — All raters at 33%. They agree by chance about 1/3 of the time, but $\alpha \approx 0$.
• "Expert vs Novice" — Mixed skill levels. Alpha reflects systematic disagreement.
• Drag one slider to 0% (always wrong). How does systematic anti-correlation affect Alpha?

Your Turn

The worked example above was mine; this one is yours. Each problem generates fresh ratings in a new setting, and the five steps mirror the walkthrough exactly. If you get stuck, help escalates gently—a nudge first, then the setup with your numbers plugged in, and only then the full answer.

Choosing the Difference Function

The choice of $\delta$ encodes what "disagreement" means for your data type:

For most LLM evaluation tasks with categorical labels (Good/Okay/Bad), use nominal. For Likert scales or ranked preferences, use ordinal. You can flip between them in the playground above and watch Alpha move.

How Good Is Good Enough?

Krippendorff (2004, p. 241) recommends:

For high-stakes LLM evaluations, aim for $\alpha \geq 0.80$. Below 0.67, your "signal" is mostly noise.

How Much Should You Trust That Number?

Alpha is a point estimate computed from a sample of items. A small or lopsided sample can produce an impressive Alpha by luck. The practical remedy is a bootstrap confidence interval: resample your items with replacement, compute Alpha for each sample, and take percentiles.

import numpy as np
import krippendorff
 
def bootstrap_alpha(ratings, n_bootstrap=1000):
    """Compute Alpha with 95% confidence interval."""
    n_items = ratings.shape[0]
    alphas = []
 
    for _ in range(n_bootstrap):
        # Resample items with replacement
        indices = np.random.choice(n_items, n_items, replace=True)
        sample = ratings[indices]
        alphas.append(krippendorff.alpha(sample, level_of_measurement='nominal'))
 
    return {
        'alpha': krippendorff.alpha(ratings, level_of_measurement='nominal'),
        'ci_low': np.percentile(alphas, 2.5),
        'ci_high': np.percentile(alphas, 97.5),
        'std': np.std(alphas)
    }

A practical rule: if your 95% CI doesn't include your decision threshold (0.67 or 0.80), you can make a confident claim. If it spans the threshold, get more data. Always report CIs when making go/no-go decisions, comparing reliability across conditions, or publishing results others will build on.

Beyond the overall number, examine disagreement item by item. High-disagreement items are diagnostic gold—they show you exactly where your rubric is failing, and they make the best training examples for calibration sessions.

def item_disagreement(ratings):
    """Compute disagreement for each item."""
    disagreements = []
    for item in ratings:
        valid = item[~np.isnan(item)]
        if len(valid) < 2:
            disagreements.append(np.nan)
            continue
        # Proportion of pairs that disagree
        pairs = [(valid[i], valid[j]) for i in range(len(valid))
                 for j in range(i+1, len(valid))]
        disagree = sum(1 for a, b in pairs if a != b) / len(pairs)
        disagreements.append(disagree)
    return disagreements

When to Use Alpha (and When Not To)

Alpha Is Right For:

• Annotation quality checks — Are your labelers consistent enough to trust?
• Codebook validation — Can different people apply your categories reliably?
• Benchmarking raters — Identifying who needs more training

Alpha Is Wrong For:

1. When ground truth exists. If you have gold labels, measure accuracy, not agreement. Alpha tells you if raters agree with each other, not if they're correct.

2. When you're comparing models. Alpha measures human consistency, not model quality. Use head-to-head preference rates or accuracy instead.

3. When disagreement is the signal. Some tasks have legitimate ambiguity. If experts genuinely disagree, that's valuable information—not measurement failure.

Why Not Just Use Kappa?

Cohen's Kappa addresses chance correction too, but Krippendorff's Alpha is more general:

For LLM evals with rotating contractor pools and incomplete coverage, Alpha is the practical choice.

Improving Low Alpha

Low reliability isn't failure—it's actionable feedback:

1. Refine your rubric. Ambiguous categories cause disagreement. "Is this response helpful?" is vague; "Does this response directly answer the user's question?" is testable.

2. Calibrate with examples. Give raters 20 pre-labeled examples to study before rating. Discuss edge cases explicitly.

3. Identify problem raters. Calculate Alpha with each rater removed. If one rater's removal dramatically improves Alpha, they need retraining.

4. Stratify by difficulty. Separate "clear" from "borderline" cases and report Alpha for each.

5. Add categories. If raters keep disagreeing between "Good" and "Okay," maybe you need a "Good-ish" middle ground.

Code

import krippendorff
import numpy as np
 
# Ratings matrix: rows = items, columns = raters
# Use np.nan for missing values
ratings = np.array([
    [1, 1, 2, 1],
    [2, 2, 2, 2],
    [1, 2, 1, 1],
    # ...
])
 
alpha = krippendorff.alpha(ratings, level_of_measurement='nominal')
print(f"Krippendorff's Alpha: {alpha:.3f}")

library(irr)
kripp.alpha(data_matrix, method="nominal")

Further Reading

• Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology
• Hayes & Krippendorff (2007). "Answering the call for a standard reliability measure"
• krippendorff Python package