Overlap in Observational Studies with High-Dimensional Covariates

Alexander D’Amour, Peng Ding, Avi Feller, Lihua Lei, Jasjeet S. Sekhon

Abstract

Estimating causal effects under exogeneity hinges on two key assumptions: unconfoundedness and overlap. Researchers often argue that unconfoundedness is more plausible when more covariates are included in the analysis. Less discussed is the fact that covariate overlap is more difficult to satisfy in this setting. In this paper, we explore the implications of overlap in observational studies with high-dimensional covariates and formalize a curse-of-dimensionality argument, suggesting that these assumptions are stronger than investigators likely realize. Our key innovation is to explore how strict overlap restricts global discrepancies between the covariate distributions in the treated and control populations. Exploiting results from information theory, we derive explicit bounds on the average imbalance in covariate means under strict overlap and show that these bounds become more restrictive as the dimension grows large. We discuss how these implications interact with assumptions and procedures commonly deployed in observational causal inference, including sparsity and trimming.

Introduction: The Causal Inference Paradox

In the quest to estimate causal effects from observational data, researchers face a fundamental tension. To satisfy the assumption of unconfoundedness—the idea that we have measured all variables influencing both treatment and outcome—the standard practice is to include as many covariates as possible. The intuition is straightforward: the more we control for, the less likely we are to suffer from omitted variable bias.

However, this paper, published in the Journal of the Royal Statistical Society, demonstrates that this strategy creates a "double-edged sword." While adding variables may help satisfy unconfoundedness, it simultaneously makes the overlap assumption—the requirement that individuals with the same characteristics have a non-zero probability of being in either the treated or control group—mathematically much harder to maintain. As the number of variables increases, the treated and control groups must become increasingly similar for overlap to hold, eventually reaching a point where they must be nearly identical.

The Information-Theoretic Framework

The authors move beyond viewing overlap as a local property of individual data points. Instead, they treat it as a global restriction on the probability distributions of the treated and control populations. By re-framing the strict overlap assumption as a bound on the likelihood ratio between these two distributions, the authors apply tools from information theory to quantify the discrepancy between groups.

Using $f$-divergences (mathematical measures of distributional distance) and adapting Rukhin’s Theorem, the study proves that if strict overlap holds, the statistical "distance" between the treated and control distributions must be small. This allows the authors to model covariates as a sequence and analyze the behavior of the system as the number of variables ($p$) approaches infinity.

The Curse of Dimensionality: Vanishing Mean Differences

The most significant finding is the formalization of a "curse of dimensionality" for causal inference. The authors derive an explicit upper bound on the Euclidean distance between the mean vectors of the treated and control groups.

Key insights from this analysis include:

  • The $1/\sqrt{p}$ Scaling: Under strict overlap, the average difference between covariate means must shrink as the number of variables increases. Specifically, the average imbalance scales at a rate of $1/\sqrt{p}$. In high dimensions, even small differences in many variables can lead to a total loss of overlap.
  • The Correlation Trade-off: For overlap to persist in high-dimensional settings, the variables must either be highly redundant (correlated) or the groups must be almost perfectly balanced from the start. One cannot have many independent, informative variables and still maintain the ability to compare treated and control units.
  • The Classifier Bound: The authors prove that if strict overlap holds, it should be impossible to build a statistical classifier that accurately distinguishes between the treated and control groups. If a model can predict treatment assignment with high accuracy, the overlap assumption has likely been violated.

Practical Implications: Trimming and Sparsity

The paper also challenges common methodological "fixes" used in observational studies:

The Trimming Trap

When overlap is poor, researchers often use "trimming"—discarding units with extreme propensity scores (probabilities of treatment). The authors demonstrate that in high dimensions, this is often a non-viable strategy. As the dimension $p$ increases, the proportion of the population satisfying strict overlap drops toward zero. Consequently, achieving a desired level of overlap might require throwing away nearly the entire sample.

The Role of Sparsity

The authors note that these results are most punishing in "dense" settings, where treatment assignment is influenced by many small factors (common in social science and medicine). If treatment assignment actually depends on only a small subset of variables (sparsity), the curse of dimensionality is mitigated. However, assuming sparsity simply to save the overlap assumption may undermine the original goal of including many covariates to ensure unconfoundedness.

Conclusion

This research serves as a cautionary note for practitioners of causal inference. The assumption of strict overlap is a much more restrictive constraint than is typically recognized in high-dimensional settings. As we move toward increasingly data-rich environments, researchers must balance the desire to eliminate confounding bias with the mathematical reality that, in high dimensions, comparability between groups is a rare and fragile property.

Summary updated

Abstract

Estimating causal effects under exogeneity hinges on two key assumptions: unconfoundedness and overlap. Researchers often argue that unconfoundedness is more plausible when more covariates are included in the analysis. Less discussed is the fact that covariate overlap is more difficult to satisfy in this setting. In this paper, we explore the implications of overlap in observational studies with high-dimensional covariates and formalize curse-of-dimensionality argument, suggesting that these assumptions are stronger than investigators likely realize. Our key innovation is to explore how strict overlap restricts global discrepancies between the covariate distributions in the treated and control populations. Exploiting results from information theory, we derive explicit bounds on the average imbalance in covariate means under strict overlap and show that these bounds become more restrictive as the dimension grows large. We discuss how these implications interact with assumptions and procedures commonly deployed in observational causal inference, including sparsity and trimming.
BibTeX
@article{damour2020overlap,
  author = {Alexander D’Amour and Peng Ding and Avi Feller and Lihua Lei and Jasjeet S. Sekhon},
  title = {{Overlap in Observational Studies with High-Dimensional Covariates}},
  journal = {Journal of Econometrics},
  year = {2020},
  doi = {10.1016/j.jeconom.2019.10.014},
}

Loading related content...

Figure