Clustered Data and the Sampling Budget

One Bad Tool Call, Nine Bad Rows
Your agent eval grades 100 conversations, 10 turns each: 1,000 graded rows. The pass rate comes in at 70%, and the naive 95% confidence interval is
That interval is too narrow, because the thousand rows aren’t a thousand independent observations.
Turns within a conversation sink or swim together. One bad tool call on turn 2 poisons that row and the eight after it; a conversation that starts clean tends to stay clean. If the intraclass correlation is (half of the outcome variance lives between conversations), the design effect is
The effective sample size is , and the clustered interval is pp. Your error bars were 2.35× too tight, and nobody lied. The rows just weren’t independent.
This is the third post in the Error Bars for Evals series. The first put a confidence interval on a single score; the second compared two models on the same questions. This one asks what happens when your rows arrive in clumps, and where the next thousand dollars of eval budget should go.
The Whole Idea in One Formula
Everything runs through the intraclass correlation : the share of outcome variance that lives between clusters rather than within them. means knowing one turn’s outcome tells you nothing about its siblings. means a conversation passes or fails as a unit and the turns are photocopies.
Given and a common cluster size , Kish’s design effect says how much clustering inflates the variance of your pass rate:
Two corollaries do all the practical work:
Anchor the formula at its edges:
- → no penalty, ever. One row per cluster is just an independent sample, whatever is.
- → no penalty, ever. Independent rows are independent rows, however you bundle them.
- The penalty is depth × stickiness. It grows with how deep each cluster goes () multiplied by how much the rows inside it move together ().
The Budget Identity
The second big idea decides how to spend. Fix a budget of rows – clusters, rows each – and expand the variance of the pooled pass rate:
Read it out loud. The first term has the full budget in its denominator: any spending shrinks it, depth or breadth alike. The second term has only . There is no in it anywhere; no amount of extra depth touches it. Once is meaningful, that second term becomes the floor under your precision, and past that point extra depth stops paying.
The chart below draws the identity. Hold the budget fixed and slide the depth: each point on the curve is a way to spend the same money, and its height is the 95% half-width that design would give you.
Interactive · the budget identity
Where the next budget should go
No setup cost: a new conversation costs only its turns. At ρ = 0.5 the shallowest design wins, as in the worked example.
Drag along the curve to try other depths. At a fixed spend each step right buys deeper conversations and fewer of them; the diamond marks the depth with the narrowest interval, √(setup × (1 − ρ)/ρ).
- Your design
- ±6.7
- pp · 100 × 10
- Best whole depth
- ±2.8
- pp · 1000 × 1
- Formula m*
- 1.0
- √(setup·(1−ρ)/ρ)
Where the two-term split comes from
Start from the design-effect form and split :
The between-cluster term is the variance of the cluster-quality lottery itself: with only conversations, you only get draws of “was this a good conversation,” no matter how carefully you grade each one.
A Worked Example
Take the eval from the hook: conversations, turns, and a pilot estimate of . The same five steps appear in the practice exercise below.
Step 1: count the rows. . This is the the report will quote.
Step 2: the design effect. . Each conversation’s ten turns carry the information of roughly two independent rows.
Step 3: effective sample size. . A thousand rows doing the work of 182.
Step 4: the bar-widening factor. . Every naive standard error in the report needs to stretch by 2.35× before you believe it.
Step 5: spend the next budget. Leadership doubles your spend to 2,000 rows. Two ways to use it:
| Design | Rows | DEFF | |
|---|---|---|---|
| Today: 100 conversations × 10 turns | 1,000 | ||
| (a) 200 conversations × 10 turns | 2,000 | ||
| (b) 100 conversations × 20 turns | 2,000 |
Same money. Doubling depth bought almost nothing (182 → 190 effective samples); doubling clusters doubled the work done (182 → 364). The budget identity said this would happen: option (b) spent its whole raise on the within-cluster term, and the floor never moved.
When a New Cluster Costs More Than a New Row
The worked example priced every row the same, and so conversations always won. Real evals rarely work that way. Writing a new prompt, or standing up a new conversation with its scenario, tools, and seeded environment, usually costs far more than grading one more turn of a conversation you already have. Kish had the same problem: one more interview in a village you’ve already traveled to is cheap, one more village is not.
Price everything in rows. Say each cluster carries a setup cost worth rows on top of its own rows, so a budget buys clusters. The depth that buys the most precision for the money is
rounded to a whole number and never below 1. Two cases, each with a budget worth 1,200 rows and a setup cost worth nine rows ():
| Depth | Clusters | ||
|---|---|---|---|
| 0.5 | 1 | 120 | 120 |
| 0.5 | 3 | 100 | 150 |
| 0.5 | 9 | 67 | 120 |
| 0.1 | 1 | 120 | 120 |
| 0.1 | 3 | 100 | 250 |
| 0.1 | 9 | 67 | 333 |
| 0.1 | 20 | 41 | 285 |
At , ; at , . With no setup cost at all (), falls to the floor of 1 for any positive , which is the worked example’s conclusion. The rule underneath: sticky clusters push you toward breadth, and expensive clusters push you toward depth, at the square root of each. The cost dial on the budget chart above marks for whatever you set.
Where Clusters Hide in LLM Evals
Conversations are the most familiar case, but the same structure shows up all over LLM evals:
| Cluster | Rows inside it | How it bites |
|---|---|---|
| Conversation | Graded turns | One early tool-call failure cascades through every later turn |
| Prompt | resamples (pass@k, best-of-, temperature sampling) | Ten generations of the same prompt mostly re-measure that prompt |
| Template | Instantiations with different entities/values | A broken template fails identically a hundred times |
| Document | Chunks, or questions about the same passage | A hard document drags down every question written against it |
| User | Sessions, requests | Power users overweight their own quirks in the average |
| Grader | Every item they grade | A rubric-happy grader clusters everything they touch |
Miller’s “Adding Error Bars to Evals” treats this exact structure (question groups drawn from a shared context) and recommends clustered standard errors whenever eval questions arrive in related bundles. The design effect itself is much older: it comes from Kish (1965), built for household surveys where interviewing one more person in the same village was cheap and one more village was expensive. Swap “village” for “conversation” and the arithmetic transfers unchanged.
Ways to Fool Yourself
1. Counting turns (or resamples) as independent . The dashboard says ; the statistics say . Any calculation that feeds the raw row count into (power analyses, CIs, significance tests) inherits the lie. The same applies to generations per prompt: pass@k evaluated on 100 prompts is 100 clusters, not observations.
2. Bootstrapping rows instead of clusters. The bootstrap is not automatically correct. Resample individual rows and every resampled dataset scrambles conversations together, preserving none of the within-cluster dependence. The interval that comes out is the naive one with extra steps. Resample whole clusters (keep every turn of each drawn conversation together) and the dependence survives into the replicates.
3. Assuming is small without measuring it. Intuition about is poor and the penalty is linear in it. A pilot run and a one-way ANOVA give you an estimate in five lines: with the between-cluster mean square and the within, . Code in the field notes. In multi-turn agent evals, where one early failure shapes every later turn, there is no reason to expect it to be small; “it’s probably about 0.1” is a guess until a pilot measures it.
4. Buying ten generations per prompt when new prompts cost about the same. Resampling feels rigorous: look how we averaged out the temperature noise. But if prompts differ more than resamples do (they usually do), the tenth resample of prompt 17 carries little new information. When you’re drawing prompts from a pool that already exists, a new prompt costs no more than a resample, and the eleventh prompt is the better buy. When new prompts have to be written, the cost section above tells you how deep to go instead.
5. Mixing cluster sizes and averaging carelessly. With unequal cluster sizes there are two different answers: pool all rows (big conversations count more) or average the cluster means (every conversation counts once). These are different estimands, and they genuinely differ when cluster size correlates with difficulty – long conversations are often long because they’re going badly. Pick the one that matches the question you’re asking, and state which one you picked.
Try It Yourself
The playground simulates the whole pipeline: each conversation draws a quality , each turn is a coin flip around it, and both intervals are computed from the same rows. The dot field is where the intuition comes from: as rises, conversations go solid or hollow, and the sentence under the dots says what the rows are worth.
Interactive
Conversations, not rows
The opening example: 100 conversations × 10 turns at ρ = 0.5. Reshuffle and watch n_eff orbit the 182 computed by hand.
The sample: 100 conversations × 10 turns
Each cluster of dots is one conversation; each dot is one graded turn, filled for pass, hollow for fail. Showing 64 of 100 conversations.
1,000 graded rows carry about as much information as 216 independent ones.
Two 95% intervals from the same rows
- Pass rate p̂
- 72.6%
- 1,000 rows
- Naive 95%
- ±2.8
- pp · every row independent
- Clustered 95%
- ±6.3
- pp · cluster bootstrap
- Ratio
- ×2.26
- formula √DEFF = 2.35
Two comparisons are worth making on purpose:
- “Wide and shallow” against “Deep and sticky”: same budget and roughly the same , very different clustered intervals. The budget chart in “The Budget Identity” shows where each design sits on the curve.
- “Bootstrap on fumes” keeps only 20 conversations. Press Reshuffle a few times: the clustered interval itself jumps around, because a bootstrap over 20 clusters has only 20 things to resample.
Your Turn
You’ve seen my numbers; now run yours. Each problem generates a fresh eval design with its row count, pass rate, and naive interval already worked out. The four steps take you from the design effect to an interval you can believe, then to the next budget. Hints escalate if you need them, from a pointer all the way to the worked answer.
Practice problem
Prompt Resampling
Guess first
Before computing: how many independent rows are these 1200 graded rows worth?
Field Notes
Estimating from a pilot. Run a slice of the eval (even 30 clusters will do) and fit the one-way ANOVA estimator. With equal cluster sizes , grand mean , and cluster means :
Compact Python: ICC from a pilot
import numpy as np
def icc_pilot(cluster_ids, outcomes):
"""One-way ANOVA intraclass correlation from (cluster_id, outcome) pairs."""
ids, y = np.asarray(cluster_ids), np.asarray(outcomes, dtype=float)
groups = [y[ids == c] for c in np.unique(ids)]
K, n = len(groups), len(y)
m_bar = n / K # average cluster size
grand = y.mean()
ms_b = sum(len(g) * (g.mean() - grand) ** 2 for g in groups) / (K - 1)
ms_w = sum(((g - g.mean()) ** 2).sum() for g in groups) / (n - K)
return (ms_b - ms_w) / (ms_b + (m_bar - 1) * ms_w)A small negative just means “indistinguishable from zero at this pilot size” – treat it as , not as an anti-correlation discovery.
When depth is the point. Everything above assumes the estimand is a population average over clusters. Sometimes it isn’t. If you’re measuring within-conversation degradation (does turn 15 fail more than turn 3?), then long conversations are the only place that signal lives, and cutting depth destroys the experiment. If pass@k is your metric, the resamples aren’t a cluster inflating your variance; they’re the estimand itself, and you need all of them per prompt (see Chen et al. 2021 for the unbiased estimator). The design effect punishes depth only when depth is redundant measurement of the same thing – so decide what the thing is before you cut.
What to report. , (or its distribution), from a pilot, and clustered standard errors – via the cluster bootstrap below or the analytic clustered SEs in Miller (2024). If your CI would embarrass you at instead of , the number to fix is .
Earlier in this series. The mechanics of the basic CI live in Your Benchmark Score Has Error Bars; comparing two models on the same clustered questions is the subject of Paired Deltas. Clustering composes with both: cluster the bootstrap there the same way you cluster it here.
Code
Python: cluster bootstrap CI
import numpy as np
def cluster_bootstrap_ci(cluster_ids, outcomes, n_boot=2000, seed=0):
"""95% CI for the pooled mean, resampling whole clusters with replacement."""
rng = np.random.default_rng(seed)
ids, y = np.asarray(cluster_ids), np.asarray(outcomes, dtype=float)
groups = [y[ids == c] for c in np.unique(ids)]
K = len(groups)
stats = []
for _ in range(n_boot):
draw = rng.integers(0, K, size=K) # sample K clusters
sample = np.concatenate([groups[j] for j in draw])
stats.append(sample.mean()) # pool the rows
return np.percentile(stats, [2.5, 97.5])
# cluster_ids = conversation IDs, one per graded turn
# lo, hi = cluster_bootstrap_ci(df.conversation_id, df.passed)Two implementation notes. First, the resampling unit is the cluster: a drawn conversation brings all of its turns. Second, sample.mean() pools rows, so bigger clusters weigh more – swap in np.mean([g.mean() for ...]) if your estimand is the mean of cluster means (see fooling-yourself #5).
Python: design-effect arithmetic
import numpy as np
def design_effect(m, rho):
return 1 + (m - 1) * rho
def effective_n(K, m, rho):
return K * m / design_effect(m, rho)
# The worked example
print(design_effect(10, 0.5)) # 5.5
print(effective_n(100, 10, 0.5)) # 181.8
print(effective_n(200, 10, 0.5)) # 363.6 <- double the clusters
print(effective_n(100, 20, 0.5)) # 190.5 <- double the depth
def optimal_m(setup, rho, max_m=20):
"""Variance-minimizing depth when each cluster costs `setup` rows on top of its m rows."""
if rho <= 0:
return max_m if setup > 0 else 1 # no correlation: depth is free information
return min(max_m, max(1.0, np.sqrt(setup * (1 - rho) / rho)))
print(optimal_m(9, 0.5)) # 3.0
print(optimal_m(9, 0.1)) # 9.0Further Reading
- Miller, E. (2024). “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.” arXiv:2411.00640. Section on clustered standard errors for question groups.
- Kish, L. (1965). Survey Sampling. Wiley. The origin of the design effect, built for the same trade-off.
- Cameron, A. C. & Miller, D. L. (2015). “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources 50(2). What to do when cluster counts are small or sizes are wild.