Understanding Krippendorff's Alpha

The Problem with Raw Agreement

You're building an LLM evaluation pipeline. Four contractors rate 500 model responses as "Good," "Okay," or "Bad." The dashboard shows 87% pairwise agreement. Ship it?

Not so fast. One rater noticed that 70% of responses are "Good" and started marking everything "Good" to speed through the batch. They still agree with the others most of the time--but they're contributing zero signal.

This is the lazy rater problem, and raw agreement can't detect it. If 70% of items are genuinely "Good," two random guessers would agree about 54% of the time just by chance ($0.7^2 + 0.15^2 + 0.15^2 \approx 0.54$). We need a number that discounts the agreement chance would have handed us anyway.

The Whole Idea in One Formula

Krippendorff's Alpha measures the reduction in disagreement relative to random labeling:

where $D_o$ is the disagreement you actually observed, and $D_e$ is the disagreement you'd expect if the same labels had been dealt out at random.

• $\alpha = 1$: Perfect agreement--no observed disagreement
• $\alpha = 0$: Agreement equals chance--raters might as well flip coins
• $\alpha < 0$: Systematic disagreement--raters actively contradict each other

Everything else is careful bookkeeping: how to count observed disagreement fairly, and how to estimate what chance alone would produce. The fastest way to internalize both is to compute them by hand, once, on a dataset small enough to hold in your head.

A Worked Example

Three raters evaluate 5 items using categories A, B, and C:

We'll follow the same five steps you'll use on real data (and in the practice exercise below).

Step 1: Raw Agreement

First, the simple metric everyone knows--what percentage of rating pairs agree?

For each item with $m$ raters, there are $\binom{m}{2} = \frac{m(m-1)}{2}$ unordered pairs. With 3 raters per item, that's $\binom{3}{2} = 3$ pairs per item. Across 5 items: $5 \times 3 = 15$ pairs total.

Raw Agreement = $\frac{3+1+3+3+1}{15} = \frac{11}{15} = 73.3\%$

Not bad! But is 73% impressive, or would random labeling do nearly as well? Hold that thought.

Step 2: The Coincidence Matrix

The coincidence matrix counts how often each pair of categories co-occurs across all rater pairs. For each item rated by $m$ raters, we enumerate all $m(m-1)$ ordered pairs and weight each by $\frac{1}{m-1}$.

With 3 raters per item, each ordered pair gets weight $\frac{1}{3-1} = 0.5$.

The resulting coincidence matrix $O$:

Matrix total = 15 (equals total ratings across all items: 5 items × 3 raters). The diagonal is where agreement lives; everything off it is disagreement.

Marginal totals $n_c$ (row/column sums):

• $n_A = 6$, $n_B = 6$, $n_C = 3$
• $n = 15$ (total pairable values)

Step 3: Observed Disagreement ($D_o$)

For nominal data (unordered categories), the difference function is:

Observed disagreement is the proportion of coincidences that are disagreements (off-diagonal entries):

From our matrix: $D_o = \frac{0 + 0 + 0 + 2 + 0 + 2}{15} = \frac{4}{15} = 0.267$

This equals $1 - \text{raw agreement} = 1 - 0.733 = 0.267$. So far, nothing new--we've just reframed agreement as disagreement.

Step 4: Expected Disagreement ($D_e$)

Here's the question Krippendorff asked. If we shuffled all ratings randomly while keeping the same category frequencies, what disagreement would we expect by chance?

The probability that two random draws from our pool are different:

Where $n_c$ is the marginal frequency of category $c$.

From our data: $n_A = 6$, $n_B = 6$, $n_C = 3$, $n = 15$

Random raters with these category frequencies would disagree 68.6% of the time. Our actual raters disagreed only 26.7% of the time--much better than chance!

Step 5: Krippendorff's Alpha

Of the agreement chance couldn't explain, our raters delivered 61%. The raw 73% sounded good, but after chance correction we land here:

Moderate reliability--acceptable for exploratory work, but not for high-stakes decisions.

Summary of the Example

Ways to Fool Yourself

Five mistakes account for most misreadings of Alpha:

1. Confusing the coincidence matrix with a confusion matrix. A confusion matrix compares predictions to ground truth. The coincidence matrix compares raters to each other--there's no "correct" answer involved.

2. Forgetting the $\frac{1}{m-1}$ weighting. Without it, items with more raters dominate the calculation. The weighting ensures each item contributes equally, not each pair.

3. Thinking high raw agreement means high Alpha. If categories are imbalanced (e.g., 90% of items are "Good"), random guessing produces high agreement. Alpha adjusts for this, so you can have 85% agreement but $\alpha = 0.3$.

4. Treating "agreement" as "accuracy." Alpha measures whether raters agree with each other, not whether they're correct. Three raters who all wrongly label "Bad" responses as "Good" will have perfect Alpha and zero accuracy.

5. Misinterpreting negative Alpha. Negative Alpha means the raters systematically disagree more than chance. That's rare, and usually indicates a coding error, inverted scales, or genuinely adversarial labeling.

Try It Yourself

The playground below simulates raters with controllable accuracy. Each row has a hidden ground truth; drag the sliders to see how rater quality affects Alpha.

Experiments to try:

• "LLM Eval: The Lazy Rater" -- One rater at 33% (random guessing for 3 categories). Watch Alpha drop despite decent raw agreement.
• "Pure Random" -- All raters at 33%. They agree by chance about 1/3 of the time, but $\alpha \approx 0$.
• "Expert vs Novice" -- Mixed skill levels. Alpha reflects systematic disagreement.
• Drag one slider to 0% (always wrong). How does systematic anti-correlation affect Alpha?

Your Turn

Same drill, your numbers. Each problem generates fresh ratings in a new setting, and the five steps mirror the walkthrough exactly. Ask for a hint if a step jams; ask again and it gets more specific.

Choosing the Difference Function

The choice of $\delta$ encodes what "disagreement" means for your data type:

For most LLM evaluation tasks with categorical labels (Good/Okay/Bad), use nominal. For Likert scales or ranked preferences, use ordinal. You can flip between them in the playground above and watch Alpha move.

How Good Is Good Enough?

Krippendorff (2004, p. 241) recommends:

For high-stakes LLM evaluations, aim for $\alpha \geq 0.80$. Below 0.67, your "signal" is mostly noise.

How Much Should You Trust That Number?

Alpha is a point estimate computed from a sample of items. A small or lopsided sample can produce an impressive Alpha by luck. The practical remedy is a bootstrap confidence interval: resample your items with replacement, compute Alpha for each sample, and take percentiles.

import numpy as np
import krippendorff
 
def bootstrap_alpha(ratings, n_bootstrap=1000):
    """Compute Alpha with 95% confidence interval."""
    n_items = ratings.shape[0]
    alphas = []
 
    for _ in range(n_bootstrap):
        # Resample items with replacement
        indices = np.random.choice(n_items, n_items, replace=True)
        sample = ratings[indices]
        alphas.append(krippendorff.alpha(sample, level_of_measurement='nominal'))
 
    return {
        'alpha': krippendorff.alpha(ratings, level_of_measurement='nominal'),
        'ci_low': np.percentile(alphas, 2.5),
        'ci_high': np.percentile(alphas, 97.5),
        'std': np.std(alphas)
    }

If your 95% CI doesn't include your decision threshold (0.67 or 0.80), you can make a confident claim; if it spans the threshold, get more data. Report CIs whenever you're making go/no-go decisions, comparing reliability across conditions, or publishing results others will build on.

Beyond the overall number, examine disagreement item by item. High-disagreement items are diagnostic gold: they show you exactly where your rubric is failing, and they make the best training examples for calibration sessions.

def item_disagreement(ratings):
    """Compute disagreement for each item."""
    disagreements = []
    for item in ratings:
        valid = item[~np.isnan(item)]
        if len(valid) < 2:
            disagreements.append(np.nan)
            continue
        # Proportion of pairs that disagree
        pairs = [(valid[i], valid[j]) for i in range(len(valid))
                 for j in range(i+1, len(valid))]
        disagree = sum(1 for a, b in pairs if a != b) / len(pairs)
        disagreements.append(disagree)
    return disagreements

When to Use Alpha (and When Not To)

Alpha earns its keep on annotation quality checks (are your labelers consistent enough to trust?), on codebook validation (can different people apply your categories reliably?), and on benchmarking raters to find who needs more training.

It's the wrong tool when ground truth exists: with gold labels in hand, measure accuracy against them, because how well the raters agree with each other is beside the point. It's also the wrong tool for comparing models, since it measures human consistency rather than model quality -- use head-to-head preference rates or accuracy for that. And some tasks have legitimate ambiguity. When experts genuinely disagree, the disagreement is information, and driving Alpha up would just paper over it.

Why Not Just Use Kappa?

Cohen's Kappa addresses chance correction too, and when you have exactly two raters it's less machinery. But Kappa handles exactly two raters, tolerates no missing data, and (without extensions) only speaks nominal. Alpha takes any number of raters, shrugs off missing ratings, and covers ordinal, interval, and ratio data through the difference function above. For LLM evals with rotating contractor pools and incomplete coverage, Alpha is the practical choice; the Kappa post has a side-by-side table if you want the full comparison.

Improving Low Alpha

Low reliability is feedback you can act on. Start with the rubric, since ambiguous categories cause most of the disagreement: "Is this response helpful?" is vague; "Does this response directly answer the user's question?" is testable. If raters keep splitting between "Good" and "Okay," maybe the scale needs a "Good-ish" middle ground. Calibration helps too -- give raters 20 pre-labeled examples to study before rating, and discuss the edge cases explicitly. Then look for structure in the disagreement itself: compute Alpha with each rater removed (a rater whose removal dramatically improves it needs retraining), and report "clear" and "borderline" cases separately so one blended number doesn't hide the split.

Code

import krippendorff
import numpy as np
 
# Ratings matrix: rows = items, columns = raters
# Use np.nan for missing values
ratings = np.array([
    [1, 1, 2, 1],
    [2, 2, 2, 2],
    [1, 2, 1, 1],
    # ...
])
 
alpha = krippendorff.alpha(ratings, level_of_measurement='nominal')
print(f"Krippendorff's Alpha: {alpha:.3f}")

library(irr)
kripp.alpha(data_matrix, method="nominal")

Further Reading

• Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology
• Hayes & Krippendorff (2007). "Answering the call for a standard reliability measure"
• krippendorff Python package