On the Misspecification of Linear Assumptions in Synthetic Control

Achille Nazaret, Claudia Shi, David M. Blei

Abstract

The synthetic control (SC) method is a popular approach for estimating treatment effects from observational panel data. It rests on a crucial assumption that we can write the treated unit as a linear combination of the untreated units. This linearity assumption, however, can be unlikely to hold in practice and, when violated, the resulting SC estimates are incorrect. In this paper we examine two questions: (1) How large can the misspecification error be? (2) How can we limit it? First, we provide theoretical bounds to quantify the misspecification error. The bounds are comforting: small misspecifications induce small errors. With these bounds in hand, we then develop new SC estimators that are specially designed to minimize misspecification error. The estimators are based on additional data about each unit, which is used to produce the SC weights. (For example, if the units are countries then the additional data might be demographic information about each.) We study our estimators on synthetic data; we find they produce more accurate causal estimates than standard synthetic controls. We then re-analyze the California tobacco-program data of the original SC paper, now including additional data from the US census about per-state demographics. Our estimators show that the observations in the pre-treatment period lie within the bounds of misspecification error, and that the observations post-treatment lie outside of those bounds. This is evidence that our SC methods have uncovered a true effect.

Introduction: The Linearity Assumption

The Synthetic Control (SC) method is a cornerstone of policy evaluation. To estimate the effect of a policy change in a specific region (the treated unit), researchers typically construct a "Synthetic" version of that unit by weighting other "donor" units so that their combined pre-treatment outcomes match the target. However, SC relies on a significant "Linearity Assumption": it assumes the treated unit is exactly a weighted average of donor units. In practice, relationships are often non-linear, or donor units may be fundamentally too different to replicate the target perfectly. When this assumption is violated—a state of "misspecification"—the resulting causal estimates can be significantly biased. This research addresses how to quantify this error and minimize it using external data.

Methodology: Bounding Misspecification

The authors move away from the standard "black box" linear model and adopt a "fine-grained" view, where a unit’s outcome is the average of individuals within that unit, driven by invariant causes such as age, sex, or education. To handle misspecification, they introduce two theoretical bounds:

  1. The M-bound: This uses the 1-Wasserstein distance to measure the distributional difference of causes between the "Synthetic" and "Target" units.
  2. The James Bound: This combines the Wasserstein distance for observed causes with pre-intervention fit to account for unobserved causes.
The proposed framework integrates traditional panel data with external demographic information (e.g., from the U.S. Census) to refine synthetic control weights.

New estimators were developed using projected gradient descent to find weights that explicitly minimize these error bounds. Unlike standard SC, which relies solely on past outcomes, these estimators leverage external demographic data to ensure donor selection is based on fundamental similarity rather than just coincidental trend matching. This approach prioritizes "similarity" over mere "fit," leading to more stable and accurate long-term predictions.

Results and Case Study

The methods were evaluated using synthetic data and the classic California "Prop 99" tobacco-program dataset.

Synthetic Data Performance

On synthetic data where the ground truth is known, the M-bound and James-bound estimators produced significantly more accurate counterfactuals than standard SC, particularly when the underlying relationships were non-linear. A major finding is that the error is Lipschitz continuous: small errors in matching the underlying distributions of causes lead to only small errors in the final causal estimate.

California Tobacco Data

Re-analyzing cigarette sales from 1970–2019 with per-state demographics (age, race, sex) from the U.S. Census, the authors found that the M-bound estimator selected geographically and demographically similar states like New Mexico and Nevada.

Application of M-bound and James bound to California cigarette sales. Post-intervention outcomes fall outside the misspecification bounds, providing strong evidence of a true treatment effect.

Crucially, while some misspecification existed (the pre-tax fit was not perfect), the smoking rates after the tax fell far outside the misspecification bounds. This provides robust evidence that the tobacco program had a real effect, rather than the result being an artifact of model misspecification.

Conclusion and Limitations

The research demonstrates that leveraging external data to bound uncertainty provides a more rigorous foundation for synthetic control. However, these methods are data-dependent, requiring high-quality external data on the "causes" of the outcome. Future work may focus on better estimation of the Lipschitz constant ($\ell$) and exploring scenarios where observed and unobserved causes are not independent.

Summary updated

Abstract

The synthetic control (SC) method is a popular approach for estimating treatment effects from observational panel data. It rests on a crucial assumption that we can write the treated unit as a linear combination of the untreated units. This linearity assumption, however, can be unlikely to hold in practice and, when violated, the resulting SC estimates are incorrect. In this paper we examine two questions: (1) How large can the misspecification error be? (2) How can we limit it? First, we provide theoretical bounds to quantify the misspecification error. The bounds are comforting: small misspecifications induce small errors. With these bounds in hand, we then develop new SC estimators that are specially designed to minimize misspecification error. The estimators are based on additional data about each unit, which is used to produce the SC weights. (For example, if the units are countries then the additional data might be demographic information about each.) We study our estimators on synthetic data; we find they produce more accurate causal estimates than standard synthetic controls. We then re-analyze the California tobacco-program data of the original SC paper, now including additional data from the US census about per-state demographics. Our estimators show that the observations in the pre-treatment period lie within the bounds of misspecification error, and that the observations post-treatment lie outside of those bounds. This is evidence that our SC methods have uncovered a true effect.
BibTeX
@inproceedings{nazaret2024misspecification,
  author = {Achille Nazaret and Claudia Shi and David M. Blei},
  title = {{On the Misspecification of Linear Assumptions in Synthetic Control}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
}
Figure