TL;DR
This paper introduces a robust framework for calculating p-values and confidence intervals in policy evaluation, specifically for methods like Synthetic Control and Difference-in-Differences. By recasting causal inference as a "structural break" prediction problem, the authors provide a method that is mathematically exact in small samples and remains valid even if the underlying statistical model is slightly misspecified.
What Problem Does This Paper Address?
In social sciences and economics, researchers often want to know the effect of a specific policy (e.g., a new tax law in one state). The gold standard is the "Synthetic Control" (SC) method, which creates a "pseudo-unit" from a weighted average of untreated units to act as a counterfactual.
However, performing statistical inference—deciding if the observed change is a real effect or just a fluke—is notoriously difficult in these settings. Traditional methods often rely on "asymptotic" logic (assuming you have infinite data) or "randomization inference" (assuming the policy was assigned randomly like a coin flip). In reality, researchers often have very few time periods, and policies are rarely assigned at random. This paper tackles the challenge of providing reliable, "model-free" inference that works even when data is scarce and models are imperfect.
Methods
The authors bridge two distinct fields: Conformal Prediction (a machine learning framework for uncertainty quantification) and Structural Break Testing (a time-series technique to detect changes in data patterns).
Key components of their approach include:
- Recasting the Problem: They treat the policy intervention as a potential "break" in a time series. If the policy had no effect (the null hypothesis), the errors (residuals) of a counterfactual prediction model should look the same before and after the policy started.
- Permutation of Residuals: Instead of permuting which units were treated, they permute the residuals over time. To account for the fact that time-series data is often correlated with itself, they use "moving block permutations."
- Estimation Under the Null: Unlike standard approaches that train models only on pre-treatment data, this method estimates model parameters using the entire dataset while imposing the null hypothesis. This "trick" significantly improves accuracy in small samples.
- The Constrained Lasso: The authors introduce the $\ell_1$-constrained least squares estimator as a "tuning-free" alternative that nests both Synthetic Control and Difference-in-Differences, providing a unified way to build counterfactuals.
Datasets
The authors evaluate their method using two primary sources:
- Extensive Simulations: They test the method against various Data Generating Processes (DGPs) involving different levels of serial correlation and non-stationarity to prove its robustness.
- Rhode Island Prostitution Data: They re-analyze a famous study (Cunningham and Shah, 2018) on the decriminalization of indoor prostitution in Rhode Island in 2003. The goal was to determine the effect on female gonorrhea incidence using 50 other U.S. states as a control group ($T_0=19$ years of pre-data, $T_*=6$ years of post-data).
Key Insights and Ideas
- Robustness to Misspecification: One of the biggest "aha" moments is that the method doesn't require the estimator to be "consistent" (perfectly accurate). As long as the estimator is "stable"—meaning its predictions don't swing wildly if you change a few data points—the resulting p-values remain valid.
- Small Sample Superiority: Traditional methods often produce "size distortions" (reporting a 5% significance level when the real error rate is much higher) when the pre-treatment period is short. By estimating under the null, this method maintains near-perfect size control even with very limited data.
- Causal Inference as Prediction: The paper reinforces the idea that causal inference is essentially a prediction problem: if we can predict the "normal" state of the world accurately, any deviation from that prediction after a policy change is our causal effect.
Key Results
- Simulation Performance: In tests with a nominal 10% significance level, the proposed method achieved an actual rejection rate of almost exactly 0.10, whereas methods that didn't estimate under the null showed significant distortions (rejecting the null up to 30-40% of the time when they shouldn't).
- Rhode Island Case Study: The method confirmed that decriminalizing indoor prostitution led to a significant decrease in female gonorrhea. Using the new conformal method, the null hypothesis of "zero effect" was rejected at the 10% level across all variations (Synthetic Control, DiD, and Constrained Lasso), with p-values as low as 0.01.
- Leave-one-out Robustness: The results held up even when individual control states were removed, proving the findings weren't driven by a single outlier state.
Limitations and Future Work
- Stationarity Assumption: The method assumes that the "shocks" (the noise in the data) are stationary over time. If the underlying volatility of the data changes for reasons unrelated to the policy, the test might become less reliable.
- Feed-forward Effects: The current model assumes the policy effect is a simple shift. The authors note that future work could extend this to models where the policy effect "feeds back" into future lags of the outcome variable.
- Computational Cost: While moving block permutations are efficient, calculating full confidence intervals via "test inversion" (running the test hundreds of times for different hypothesized effects) can be computationally intensive for very large datasets.
Summary updated