On the Misspecification of Linear Assumptions in Synthetic Controls

Achille O. R. Nazaret, Claudia Shi, David M. Blei

Abstract

The Synthetic Control (SC) method is a cornerstone of policy evaluation, yet it relies on the often-fragile assumption that a treated unit can be perfectly represented as a linear combination of untreated donor units. This paper addresses the risks of model misspecification when this assumption fails. By introducing theoretical bounds—the M-bound and James-bound—and leveraging external demographic data, the authors demonstrate how to quantify estimation error and maintain robust causal inference even when the underlying linear model is imperfect.

The Problem: Fragile Linear Assumptions

In typical policy evaluation, researchers use the Synthetic Control method to estimate counterfactuals. For example, to study the impact of a tobacco tax in California, one might create a "Synthetic California" by weighting other states to match California's pre-intervention smoking rates. However, this method assumes that California is a simple linear combination of other states. If California’s population possesses unique characteristics that a weighted average cannot capture, the resulting estimates can be biased and misleading. This paper investigates how to make SC robust when these mathematical assumptions do not perfectly align with reality.

Methodology: Matching Populations, Not Just Outcomes

To address misspecification, the authors shift the focus from matching observed outcome curves to matching underlying population distributions. They propose a model where unit-level outcomes are viewed as averages of individual behaviors driven by specific demographic traits.

Figure 1: The proposed framework combines traditional panel data with external demographic data (e.g., from the U.S. Census) to create more robust synthetic controls.

Two primary theoretical bounds are introduced to quantify mismatch:

  1. The M-bound: Utilizes the Wasserstein distance to measure the difference between the target population and the synthetic one, proving that small misspecifications lead to only small errors.
  2. The James-bound: Combines the Wasserstein distance of observed traits (like age and sex) with pre-intervention outcome fit to account for unobserved factors.

Unlike standard SC, which optimizes weights based solely on past outcomes, these new estimators prioritize weights that ensure the synthetic population is demographically similar to the target unit.

Key Results and the "Escape" Principle

The authors re-evaluated the classic 1988 California tobacco tax (Proposition 99) using their new framework. While they confirmed that California is not a perfect linear mix of other states, the calculated bounds showed that the observed drop in smoking was significantly larger than any potential error introduced by misspecification.

Figure 2: Application of M-bound and James-bound to the California tobacco study. The observed outcomes clearly "escape" the misspecification intervals, providing rigorous evidence of the policy's effect.

This illustrates the "Escape" principle: if the post-treatment data falls outside the calculated misspecification interval, it provides strong evidence that the policy had a genuine effect that cannot be attributed to model error. In synthetic experiments where the linear assumption was intentionally broken, the M-bound and James-bound estimators significantly outperformed standard SC and other common alternatives like ElasticNet.

Conclusion

By incorporating external demographic data as proxies for latent causes, this research provides a more rigorous foundation for the Synthetic Control method. While the approach requires high-quality external data, it offers practitioners a powerful tool to validate their findings against model misspecification, ensuring that policy evaluations remain reliable even in complex, non-linear real-world settings.

Summary updated

Abstract

The synthetic control (SC) method is a popular approach for estimating treatment effects from observational panel data. It rests on a crucial assumption that we can write the treated unit as a linear combination of the untreated units. This linearity assumption, however, can be unlikely to hold in practice and, when violated, the resulting SC estimates are incorrect. In this paper we examine two questions: (1) How large can the misspecification error be? (2) How can we limit it? First, we provide theoretical bounds to quantify the misspecification error. The bounds are comforting: small misspecifications induce small errors. With these bounds in hand, we then develop new SC estimators that are specially designed to minimize misspecification error. The estimators are based on additional data about each unit, which is used to produce the SC weights. (For example, if the units are countries then the additional data might be demographic information about each.) We study our estimators on synthetic data; we find they produce more accurate causal estimates than standard synthetic controls. We then re-analyze the California tobacco-program data of the original SC paper, now including additional data from the US census about per-state demographics. Our estimators show that the observations in the pre-treatment period lie within the bounds of misspecification error, and that the observations post-treatment lie outside of those bounds. This is evidence that our SC methods have uncovered a true effect.
BibTeX
@inproceedings{nazaret2024misspecification,
  author = {Achille O. R. Nazaret and Claudia Shi and David M. Blei},
  title = {{On the Misspecification of Linear Assumptions in Synthetic Controls}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
}
Figure