TL;DR
Difference-in-Differences (DD) estimation is a cornerstone of causal inference, yet it has a massive "blind spot": serial correlation. This paper demonstrates that ignoring this correlation leads to a staggering number of false positives—finding "statistically significant" policy effects where none exist—and provides a toolkit of simple, robust fixes to restore the integrity of these estimates.
What Problem Does This Paper Address?
The Difference-in-Differences (DD) method is used to estimate the causal effect of an intervention (like a new law) by comparing the change in outcomes for a "treated" group against a "control" group. While researchers often worry about whether the groups are truly comparable, they frequently ignore a more technical threat: serial correlation.
In most DD studies, researchers use many years of data. However, economic variables like wages or employment are "sticky"—if they are high this year, they are likely to be high next year. Furthermore, the "treatment" (the law) is also highly correlated with itself; once a law is passed, it usually stays in effect. Standard Ordinary Least Squares (OLS) assumes that the "shocks" or errors in each year are independent. When they aren't, OLS drastically underestimates the standard errors, leading researchers to claim their results are "significant" when they are actually just seeing the patterns of random, persistent noise.
Methods
The authors use a "placebo" approach to diagnose the severity of the problem. They take real-world wage data and "pass" fake laws at random times in random states. Since these laws are fictitious, a valid statistical test should only find a "significant" effect 5% of the time (at the 5% significance level).
To find a solution, they evaluate several alternative techniques:
- Parametric Corrections: Attempting to model the error as an AR(1) process.
- Block Bootstrap: Resampling entire states to preserve the internal correlation structure.
- Clustering: Using an arbitrary covariance structure (clustering at the state level).
- Aggregation: Collapsing the time-series data into just two periods: "before" and "after."
- Randomization Inference: Comparing the actual estimate to a distribution of estimates generated from thousands of placebo laws.
Datasets
- Current Population Survey (CPS): Specifically, the authors use log weekly earnings for women aged 25–50 from the Merged Outgoing Rotation Groups (1979–1999).
- Manufactured Data: The authors also create synthetic datasets where they can precisely control the level of serial correlation (using AR(1) processes) to see exactly how different levels of "stickiness" break the standard DD model.
Key Insights and Ideas
- The "Triple Threat": Serial correlation is uniquely dangerous in DD because of three factors: (1) DD often relies on long time series, (2) the dependent variables are highly persistent, and (3) the treatment variable itself changes very little over time within a state.
- Standard Fixes are Insufficient: Common diagnostic checks, such as testing for "pre-existing trends" or looking for "persistence of effects," do not reliably solve the serial correlation problem. They may reduce the false positive rate slightly, but not nearly enough.
- The Power of Simplicity: One of the most effective solutions is the simplest: aggregation. By averaging all data before the law and all data after the law into just two data points per state, the researcher effectively kills the time-series dimension and the serial correlation problem along with it.
Key Results
- The False Positive Explosion: When using standard OLS on placebo laws with 21 years of CPS data, the null hypothesis was rejected up to 45% of the time. In other words, OLS "found" an effect in nearly half of the cases where there was absolutely no effect.
- Clustering Works (With Enough Groups): Clustering standard errors at the state level (using the White/Huber estimator) works well when the number of states is large (e.g., 50), but it begins to fail and over-reject as the number of states decreases.
- Randomization Inference is the Gold Standard: This method produced the correct 5% rejection rate regardless of the sample size or the number of states, making it the most robust option for researchers working with small or idiosyncratic datasets.
Limitations and Future Work
- The Power Trade-off: While techniques like aggregation and randomization inference solve the "size" problem (false positives), they can reduce the "power" of the test (the ability to detect a real effect when it exists).
- Small Sample Constraints: Randomization inference requires the researcher to make an assumption about the "law-making process" (e.g., that any state could have passed the law at any time). If the true process is very specific, the placebo distribution might be misleading.
- Re-evaluating the Literature: The authors suggest that because many published DD papers report t-statistics around 2 (the threshold for 5% significance), a significant portion of the existing literature may be based on spurious results. A natural next step is a systematic re-analysis of high-profile DD studies using these more robust methods.
Summary updated