Bounding Treatment Effects by Pooling Limited Information Across Observations

Sokbae Lee, Martin WeidnerSSRN 2024 · DOI

TL;DR

This paper introduces a new method for estimating treatment effects in observational studies, particularly when standard assumptions like "overlap" (where treatment and control groups have similar characteristics) are violated. Their approach offers a flexible middle ground between overly simplistic bounds and complex methods that require extensive data pooling, providing more robust and informative estimates in challenging real-world scenarios.

What Problem Does This Paper Address?

In causal inference, estimating the effect of a "treatment" (like a new drug, policy, or intervention) from observational data is tricky. We want to know what would have happened if a person received the treatment versus if they didn't. The gold standard is a randomized controlled trial (RCT), but these are often infeasible or unethical. In observational studies, we rely on assumptions like "unconfoundedness" (all relevant factors influencing both treatment and outcome are observed) and "overlap" (the groups being compared are similar enough).

The "overlap" assumption is often violated in practice. This means there are situations where we have very few or no observations with similar characteristics but different treatment statuses. This "limited overlap" makes it hard to estimate treatment effects reliably. Existing methods either:

  • Ignore the problem: leading to biased estimates.
  • Use Manski bounds: which are very robust but often too wide to be useful.
  • Use inverse propensity score weighting (IPW): which requires extensive data pooling to estimate treatment probabilities and can be sensitive to misspecification or the "curse of dimensionality" with many covariates.

This paper addresses the challenge of estimating treatment effects when overlap is limited, and the propensity score (the probability of receiving treatment) is unknown or hard to estimate reliably. They aim to provide bounds that are more informative than Manski bounds but more robust than IPW.

Methods

The core innovation of this paper lies in developing "second-order" and "higher-order" bounds. Instead of pooling all information (like IPW) or no information (like Manski bounds), they propose a strategy that pools information across small groups of observations with similar covariate values.

Here's a breakdown:

  • Limited Pooling: The bounds are constructed by averaging functions of observed outcomes. The key is that each outcome's contribution depends only on the treatment status of a limited number of other observations with similar covariates. This is a middle ground between no pooling (Manski) and unlimited pooling (IPW).
  • Second-Order Bounds: The simplest form of this idea involves considering pairs of observations with identical covariate values. By leveraging unconfoundedness, they can derive tighter bounds than Manski bounds.
  • Higher-Order Bounds: This concept is generalized to pooling information from larger groups of observations with similar covariates. The bounds are expressed as polynomials in the propensity score, allowing for flexibility and incorporating prior knowledge about the propensity score.
  • Reference Propensity Score: A crucial element is the researcher's choice of a "reference propensity score" (p*(x)). This score doesn't need to be perfectly accurate for the bounds to remain valid, but a good guess can lead to tighter bounds. This allows for incorporating prior knowledge in a robust way.
  • Inference Methods: The paper also develops estimation and inference procedures for these bounds, including asymptotically valid confidence intervals. For continuous covariates, they propose a clustering approach to group similar covariate values, effectively creating discrete "clusters" to apply their bounds.

Datasets

The paper evaluates their methods using two well-known datasets:

  • Connors et al. (1996) Right Heart Catheterization (RHC) dataset: This dataset has been widely analyzed in the context of limited overlap.
  • Dehejia and Wahba (1999) LaLonde dataset (NSW experimental data): Specifically, they use a subsample and control groups from this dataset.

No new datasets were introduced.

Key Insights and Ideas

  • The "Middle Ground" is Powerful: The paper demonstrates that a strategic, limited pooling of information across observations with similar characteristics can yield significantly more informative bounds than extreme approaches (no pooling or full pooling).
  • Robustness to Propensity Score Misspecification: The proposed bounds remain valid even if the researcher's chosen reference propensity score is not perfectly accurate. This is a major advantage over methods like IPW that are highly sensitive to propensity score estimation.
  • Flexibility via Higher-Order Polynomials: By expressing bounds as polynomials in the propensity score, the authors can control the trade-off between tightness and robustness, allowing for a more nuanced approach.
  • Clustering for Continuous Covariates: The proposed clustering method effectively transforms continuous covariate spaces into discrete "clusters," making the bounds applicable even when no two observations have identical covariate values.
  • Practical Implications for Policy and Medicine: The methods are directly applicable to real-world scenarios where overlap is an issue, such as evaluating the impact of medical treatments or social programs.

Key Results

  • Improved Bounds: The proposed second-order and higher-order bounds are shown to be tighter (more informative) than Manski bounds in simulations and empirical examples, especially when the reference propensity score is reasonably chosen.
  • Robust Confidence Intervals: The paper provides asymptotically valid confidence intervals that are robust to limited overlap and propensity score misspecification.
  • Empirical Validation: The methods are applied to the RHC and LaLonde datasets, demonstrating their practical relevance and ability to produce meaningful bounds in challenging settings. For instance, in the RHC study, their bounds exclude large beneficial effects with confidence, based solely on unconfoundedness. In the LaLonde dataset, their bounds suggest that limited overlap is a concern, leading to wider intervals compared to a scenario with better overlap.
  • Monte Carlo Simulations: Experiments show that the proposed inference bounds provide good coverage and non-empty intervals, outperforming sample analog bounds in certain scenarios, especially when dealing with discrete covariates. However, for continuous covariates and certain DGPs, issues with clustering bias can arise.

Limitations and Future Work

  • Discretization Bias: For continuous covariates, the paper acknowledges that discretizing them (e.g., via binning or clustering) introduces approximation errors for which formal theory is not provided. Controlling this error would require additional smoothness assumptions.
  • Choice of Reference Propensity Score: While the bounds are valid regardless of the reference propensity score, their tightness depends on how close it is to the true score. Choosing an optimal reference score is a practical challenge. The paper suggests defaults (like the global treatment share) and discusses more refined choices, but a formal data-driven strategy is left for future work.
  • Empty Confidence Intervals: While the proposed inference method aims to avoid empty intervals, the paper notes that in finite samples, sample analog bounds can still result in empty intervals, which is undesirable. They propose a method to ensure non-empty intervals, but acknowledge it might lead to conservative results.
  • Intersection Bounds: The paper mentions that forming intersections of different bounds could lead to even tighter results but does not explore this further, leaving it for future research.
  • Smoothness Assumptions for Clustering: The clustering procedure introduces dependence among labeled covariates, requiring more rigorous treatment with additional smoothness assumptions or a formal model for the clustering structure.

Summary updated

BibTeX
@misc{lee2024bounding,
  author = {Sokbae Lee and Martin Weidner},
  title = {{Bounding Treatment Effects by Pooling Limited Information Across Observations}},
  year = {2024},
  doi = {10.2139/ssrn.4758812},
  howpublished = {SSRN},
}
Figure