Moving the Goalposts: Addressing Limited Overlap in Estimation of Average Treatment Effects by Changing the Estimand

Richard K. Crump, V. Joseph Hotz, Guido W. Imbens, Oscar A. MitnikSSRN 2006 · DOI

Introduction

In causal inference, the gold standard is often the Average Treatment Effect (ATE), which represents the expected impact of an intervention if applied to an entire population. However, a fundamental requirement for identifying the ATE is the "overlap" or "common support" assumption: for every set of observed characteristics, there must be a non-zero probability of individuals being in both the treatment and control groups. When this assumption is violated—a condition known as limited overlap—standard estimators become highly sensitive to model specification and suffer from extreme imprecision.

In their seminal paper, "Moving the Goalposts: Addressing Limited Overlap in the Estimation of Average Treatment Effects by Changing the Estimand," Richard Crump, V. Joseph Hotz, Guido Imbens, and Oscar Mitnik propose a principled solution. Rather than relying on ad-hoc trimming or potentially biased extrapolations, the authors suggest changing the target of inference (the estimand) to a subpopulation where the treatment effect can be estimated with maximum precision.

The Problem of Limited Overlap

Limited overlap occurs when certain subpopulations are almost exclusively treated or exclusively assigned to the control group. For example, if a job training program only enrolls low-income individuals while the control group consists primarily of high-income individuals, comparing the two groups directly becomes problematic.

In such cases, the propensity score—the probability of receiving treatment given a set of covariates—approaches 0 or 1. Mathematically, the variance of the ATE estimator is inversely proportional to $e(x)(1-e(x))$, where $e(x)$ is the propensity score. As the score nears the boundaries, the variance explodes, leading to wide confidence intervals and unreliable results. While researchers often "trim" the sample to remove these problematic observations, this process is frequently arbitrary and lacks a formal theoretical justification.

Methodology: Redefining the Estimand

Crump et al. formalize the trimming process by introducing two new estimands designed to optimize the trade-off between sample representation and statistical precision:

  1. Optimal Subpopulation Average Treatment Effect (OSATE): This approach identifies a specific subset of the population—defined by a range of propensity scores—that minimizes the variance of the treatment effect estimate. The authors demonstrate that under the assumption of homoskedasticity, the optimal selection rule is to include only those observations where the propensity score $e(x)$ falls within the interval $[\alpha, 1-\alpha]$.
  2. Optimally Weighted Average Treatment Effect (OWATE): Instead of a binary inclusion/exclusion rule, this method assigns weights to each observation to minimize the overall variance of the estimate.

A key contribution of the paper is the derivation of a practical "rule of thumb." After analyzing various distributions, the authors find that a propensity score range of $[0.1, 0.9]$ consistently serves as an excellent approximation for the optimal subpopulation in most empirical settings.

Empirical Evidence: The LaLonde Dataset

The authors apply their methodology to the LaLonde (1986) dataset, a classic benchmark in econometrics that combines experimental data from the National Supported Work (NSW) program with non-experimental control data from the Panel Study of Income Dynamics (PSID). The PSID control group is significantly different from the NSW participants in terms of pre-program earnings and demographics, creating a severe overlap problem.

The results of the OSATE approach are transformative:

  • Full Sample ATE: The initial estimate yielded a value of -14.75 with a massive standard error of 637.90, rendering the result statistically meaningless.
  • OSATE (Optimal Trimming): By restricting the sample to the optimal subpopulation (dropping approximately 88% of the observations, mostly from the PSID control group), the estimate shifted to -1.17 with a standard error of only 1.69.

This represents a variance reduction by a factor of over 100,000. Even when compared to the Average Treatment Effect on the Treated (ATT), the OSATE provided a 60% gain in precision.

Discussion: Internal vs. External Validity

The authors acknowledge that "moving the goalposts" involves a significant trade-off. By narrowing the study to a subpopulation, researchers sacrifice external validity; the findings can no longer be generalized to the entire population. However, the authors argue that a precise and reliable estimate for a well-defined subpopulation is far more valuable than an imprecise, model-dependent estimate for a population where the data provides little information.

Crucially, because the optimal subpopulation is determined solely by the propensity score (which does not involve the outcome variable), the method prevents "p-hacking" or cherry-picking data to achieve a desired result.

Conclusion

Crump et al. provide a rigorous framework for handling limited overlap in causal inference. By shifting the focus from the population ATE to the OSATE, researchers can produce more robust and transparent results. The $[0.1, 0.9]$ propensity score rule remains a vital tool for practitioners seeking to balance the demands of internal validity and statistical precision in observational studies.

Summary updated

BibTeX
@misc{crump2006moving,
  author = {Richard K. Crump and V. Joseph Hotz and Guido W. Imbens and Oscar A. Mitnik},
  title = {{Moving the Goalposts: Addressing Limited Overlap in Estimation of Average Treatment Effects by Changing the Estimand}},
  year = {2006},
  doi = {10.2139/ssrn.937912},
  howpublished = {SSRN},
}
Figure