Abstract
Most Difference-in-Difference (DD) papers rely on many years of data and focus on serially correlated outcomes. Yet almost all these papers ignore the bias in the estimated standard errors that serial correlation introduces. This is especially troubling because the independent variable of interest in DD estimation (e.g., the passage of a law) is itself very serially correlated, which exacerbates the bias in standard errors. To illustrate the severity of this issue, we randomly generate placebo laws in state-level data on female wages from the Current Population Survey. For each law, we use OLS to compute the DD estimate of its "effect" as well as the standard error for this estimate. The standard errors are severely biased: with about 20 years of data, DD estimation finds an "effect" significant at the 5% level for up to 45% of the placebo laws.
Introduction: The Serial Correlation Blind Spot
Difference-in-Differences (DD) estimation is a cornerstone of causal inference, yet it has a massive "blind spot": serial correlation. This research demonstrates that ignoring this correlation leads to a staggering number of false positives—finding "statistically significant" policy effects where none exist. By diagnosing the root of this bias, the paper provides a toolkit of simple, robust fixes to restore the integrity of these estimates.
The Problem: A Triple Threat to Validity
The DD method estimates the causal effect of an intervention by comparing changes in outcomes for a "treated" group against a "control" group. While researchers often worry about whether groups are truly comparable, they frequently ignore a technical threat: serial correlation.
Serial correlation is uniquely dangerous in DD because of three factors:
- Long Time Series: DD often relies on many years of data.
- Sticky Variables: Dependent variables (like wages or employment) are highly persistent; if they are high this year, they are likely to be high next year.
- Persistent Treatments: The "treatment" variable (the law) changes very little over time within a state once passed.
Standard Ordinary Least Squares (OLS) assumes that errors in each year are independent. When they aren't, OLS drastically underestimates standard errors, leading researchers to claim results are "significant" when they are actually just observing patterns of random, persistent noise.
Methodology: The Placebo Test
To diagnose the severity of the problem, the authors used a "placebo" approach. They took real-world wage data from the Current Population Survey (CPS)—specifically log weekly earnings for women aged 25–50 from 1979 to 1999—and "passed" fake laws at random times in random states.
Since these laws are fictitious, a valid statistical test should only find a "significant" effect 5% of the time (at the 5% significance level). However, when using standard OLS on these placebo laws, the null hypothesis was rejected up to 45% of the time. In other words, OLS "found" an effect in nearly half of the cases where there was absolutely no effect.
Evaluating the Toolkit for Fixes
The authors evaluated several alternative techniques to correct this bias:
- Aggregation: Collapsing the time-series data into just two periods—"before" and "after." This effectively kills the time-series dimension and the serial correlation problem with it.
- Clustering: Using the White/Huber estimator to cluster standard errors at the state level. This works well when the number of states is large (e.g., 50), but fails as the number of states decreases.
- Randomization Inference: Comparing the actual estimate to a distribution of estimates generated from thousands of placebo laws. This produced the correct 5% rejection rate regardless of sample size.
- Parametric Corrections: Attempting to model the error as an AR(1) process, which proved less robust than simpler methods.
- Block Bootstrap: Resampling entire states to preserve internal correlation structures.
Key Insights and Recommendations
One of the most effective solutions is the simplest: aggregation. By averaging data into two points per state, researchers can bypass the complexities of serial correlation without sophisticated modeling. For researchers working with small or idiosyncratic datasets, Randomization Inference remains the gold standard for robustness.
Common diagnostic checks, such as testing for "pre-existing trends," do not reliably solve the serial correlation problem. They may reduce the false positive rate slightly, but not enough to ensure validity. Given that many published DD papers report t-statistics near the 5% significance threshold, a significant portion of existing literature may be based on spurious results, necessitating a careful re-evaluation of high-profile studies using these more robust methods.
Summary updated