When Should You Adjust Standard Errors for Clustering?

Alberto Abadie, Susan Athey, Guido W. Imbens, Jeffrey M. WooldridgeThe Quarterly Journal of Economics 2017

TL;DR

This paper introduces a new framework for calculating standard errors when data is clustered, moving beyond the traditional "correlation within clusters" explanation. The authors argue that clustering decisions should be based on the design of the data collection (sampling or treatment assignment) rather than just observed correlations, offering more accurate and less conservative standard errors, especially in complex settings.

What Problem Does This Paper Address?

In many fields, especially economics, researchers often adjust standard errors to account for "clustering" – when observations within certain groups (like states, schools, or individuals in the same family) are not independent. The standard justification is that unobserved factors shared within these clusters lead to correlated outcomes. However, this explanation has limitations: it doesn't clearly explain why we cluster by some groups (e.g., states) but not others (e.g., gender), nor why clustering is sometimes needed in observational studies but not in randomized experiments. This ambiguity can lead to standard errors that are either too large (overly conservative) or too small (leading to incorrect conclusions). The paper aims to provide a more robust and principled framework for deciding when and how to cluster standard errors.

Methods

The authors propose a novel framework that views clustering as a design problem, focusing on two key aspects:

  1. Sampling Process: How the data was sampled from a larger population. This includes scenarios where only a fraction of clusters are observed (two-stage sampling).
  2. Assignment Process: How the treatment or exposure was assigned to units. This is particularly relevant when treatment assignment is correlated within clusters (e.g., all individuals in a village receive the same treatment).

Their framework incorporates both sampling and assignment variations, allowing for "partially clustered" assignment where treatment varies within clusters but is still influenced by cluster-level factors. They derive new variance estimators, termed Causal Cluster Variance (CCV) and a Two-Stage-Cluster-Bootstrap (TSCB), which aim to correct biases found in traditional robust and cluster-robust standard errors.

Datasets

The authors use a sample from the 2000 U.S. Decennial Census data, containing log earnings, college attendance indicators, and state of residence for 2,632,838 individuals. This dataset is used to illustrate the empirical relevance of their proposed methods. No new benchmarks are introduced; rather, the paper focuses on developing and evaluating new variance estimators.

Key Insights and Ideas

  • Clustering is a Design Problem: The decision to cluster standard errors should stem from the data's sampling or treatment assignment design, not just from observed correlations in residuals.
  • Beyond "All-or-Nothing": Traditional clustering is often an "all-or-nothing" approach. The new framework allows for more nuanced situations, like partially clustered assignment, where treatment varies within clusters but is still influenced by cluster-level factors.
  • Finite Population Focus: Shifting from infinite superpopulation models to finite population average treatment effects simplifies the justification for clustering.
  • Misconceptions Debunked: The paper addresses common misunderstandings, such as the idea that clustering is only needed when residuals are correlated within clusters, or that there's no harm in clustering if it makes a difference.
  • New Estimators Outperform: The proposed CCV and TSCB estimators offer substantial improvements over traditional robust and cluster standard errors, providing more accurate and less conservative estimates.

Key Results

  • In an empirical example using U.S. Census data, cluster standard errors were found to be 26 times larger than robust standard errors for one specification and 23 times larger for another, highlighting the potential for over-conservatism.
  • The proposed CCV and TSCB standard errors for OLS estimates were substantially smaller than conventional cluster standard errors but larger than robust standard errors, falling in a more accurate intermediate range.
  • Simulation results showed that robust standard errors were often too small, leading to low coverage rates for confidence intervals, while cluster standard errors were too large.
  • The CCV and TSCB estimators consistently outperformed robust and cluster standard errors across various simulation designs, offering more accurate standard errors and better coverage rates.

Limitations and Future Work

  • Assumptions: The derivations rely on certain assumptions about sampling probabilities, cluster sizes, and the distribution of assignment probabilities.
  • Further Refinements: The authors mention that their proposed variance estimators might benefit from further refinements, and their formal properties are left for future work.
  • Non-linear Estimators: The current framework is primarily applied to linear estimators (OLS and fixed effects). Extending the ideas to non-linear estimators is an avenue for future research.
  • Other Assignment/Sampling Processes: Deriving standard error formulas for sampling and assignment processes not covered in this article is an important area for future research.

Summary updated

Abstract

Abstract Clustered standard errors, with clusters defined by factors such as geography, are widespread in empirical research in economics and many other disciplines. Formally, clustered standard errors adjust for the correlations induced by sampling the outcome variable from a data-generating process with unobserved cluster-level components. However, the standard econometric framework for clustering leaves important questions unanswered: (i) Why do we adjust standard errors for clustering in some ways but not others, for example, by state but not by gender, and in observational studies but not in completely randomized experiments? (ii) Is the clustered variance estimator valid if we observe a large fraction of the clusters in the population? (iii) In what settings does the choice of whether and how to cluster make a difference? We address these and other questions using a novel framework for clustered inference on average treatment effects. In addition to the common sampling component, the new framework incorporates a design component that accounts for the variability induced on the estimator by the treatment assignment mechanism. We show that, when the number of clusters in the sample is a nonnegligible fraction of the number of clusters in the population, conventional clustered standard errors can be severely inflated, and propose new variance estimators that correct for this bias.
BibTeX
@article{abadie2017when,
  author = {Alberto Abadie and Susan Athey and Guido W. Imbens and Jeffrey M. Wooldridge},
  title = {{When Should You Adjust Standard Errors for Clustering?}},
  journal = {The Quarterly Journal of Economics},
  year = {2017},
}
Figure