Abstract
We introduce new inference procedures for counterfactual and synthetic control methods for policy evaluation. We recast the causal inference problem as a counterfactual prediction and a structural breaks testing problem. This allows us to exploit insights from conformal prediction and structural breaks testing to develop permutation inference procedures that accommodate modern high-dimensional estimators, are valid under weak and easy-to-verify conditions, and are provably robust against misspecification. Our methods work in conjunction with many different approaches for predicting counterfactual mean outcomes in the absence of the policy intervention. Examples include synthetic controls, difference-in-differences, factor and matrix completion models, and (fused) time series panel data models. Our approach demonstrates an excellent small-sample performance in simulations and is taken to a data application where we re-evaluate the consequences of decriminalizing indoor prostitution. Open-source software for implementing our conformal inference methods is available.
Introduction
In the social sciences and economics, evaluating the impact of a specific policy—such as a tax law change or a public health intervention—often relies on comparative case studies. The Synthetic Control (SC) method has emerged as a gold standard for this purpose, constructing a "pseudo-unit" from a weighted average of untreated units to serve as a counterfactual. However, performing rigorous statistical inference in these settings remains a significant challenge.
Traditional inference methods typically rely on asymptotic logic, which assumes an infinite number of time periods, or randomization inference, which assumes policies are assigned randomly. In practice, researchers often face small sample sizes with few pre-treatment periods, and policy assignments are rarely random. This paper addresses these gaps by introducing a robust framework for calculating p-values and confidence intervals that is mathematically exact in small samples and remains valid even under model misspecification.
Methodology: Bridging Conformal Prediction and Structural Breaks
The proposed method recasts causal inference as a "structural break" prediction problem. By integrating Conformal Prediction—a machine learning framework for uncertainty quantification—with time-series structural break testing, the authors develop a more flexible inference procedure.
Recasting the Problem
The core idea is to treat a policy intervention as a potential break in a time series. Under the null hypothesis that the policy has no effect, the residuals (the difference between observed and predicted outcomes) should exhibit consistent patterns before and after the intervention. If the policy did have an effect, the post-intervention residuals will deviate significantly from the pre-intervention distribution.
Permutation of Residuals
Unlike traditional methods that permute which units are treated, this approach permutes the residuals over time. To account for temporal dependencies and serial correlation common in economic data, the authors utilize "moving block permutations." This ensures that the time-series structure of the noise is preserved during the inference process.
Estimation Under the Null
A critical innovation in this framework is the practice of estimating model parameters using the entire dataset while imposing the null hypothesis. Standard approaches often train models only on pre-treatment data, which can lead to overfitting or poor generalization in small samples. By utilizing the full dataset under the null, the method significantly improves the accuracy and "size control" of the statistical tests.
The Constrained Lasso
To provide a unified approach to counterfactual construction, the authors introduce an $\ell_1$-constrained least squares estimator. This "tuning-free" alternative nests both Synthetic Control and Difference-in-Differences (DiD), allowing researchers to move seamlessly between different modeling assumptions within a single framework.
Empirical Evaluation
Simulation Performance
The authors conducted extensive simulations across various Data Generating Processes (DGPs), including scenarios with high serial correlation and non-stationarity. The results demonstrated that the proposed method maintains near-perfect size control. For instance, at a nominal 10% significance level, the method achieved an actual rejection rate of approximately 0.10. In contrast, methods that did not estimate under the null showed significant distortions, erroneously rejecting the null hypothesis up to 30-40% of the time.
Case Study: Decriminalization of Prostitution in Rhode Island
The method was applied to a re-evaluation of the decriminalization of indoor prostitution in Rhode Island in 2003. Using 50 other U.S. states as a control group ($T_0=19$ years of pre-data, $T_*=6$ years of post-data), the authors tested the effect on female gonorrhea incidence.
The conformal inference method confirmed a significant decrease in gonorrhea rates following decriminalization. Across all variations—including SC, DiD, and the Constrained Lasso—the null hypothesis of "zero effect" was rejected at the 10% level, with p-values as low as 0.01. Robustness checks, including leave-one-out analyses, confirmed that these results were not driven by any single outlier state.
Key Insights and Robustness
One of the most significant findings is the method's robustness to misspecification. The procedure does not require the underlying estimator to be perfectly consistent. As long as the estimator is "stable"—meaning its predictions are not overly sensitive to small changes in the data—the resulting p-values remain valid. This reinforces the perspective that causal inference is essentially a prediction problem: if we can accurately predict the "normal" state of the world, any significant deviation following a policy change can be interpreted as a causal effect.
Limitations and Future Work
While the method is robust, it does rely on the assumption of stationarity in the data's "shocks" or noise. If the underlying volatility of the data changes for reasons unrelated to the policy, the test's reliability may be affected. Additionally, the current model assumes a simple shift effect; future research could extend this to accommodate "feed-forward" effects where the policy impact influences future lags of the outcome variable. Finally, while efficient, the computational cost of calculating full confidence intervals via test inversion can be high for exceptionally large datasets.
Conclusion
This paper provides a powerful new tool for policy evaluation, offering a bridge between machine learning and traditional econometrics. By providing exact inference that is robust to model choice and sample size, the conformal inference framework ensures that researchers can draw more reliable conclusions from comparative case studies.
Summary updated