Variance estimation for the average treatment effects on the treated and on the controls

Roland Matsouaka, Yi Liu, Yunji ZhouStatistical 2022 · DOI

TL;DR

This paper introduces and evaluates new methods for estimating the uncertainty (variance) of treatment effect estimates in observational studies, specifically for the "treated" and "control" groups. The authors propose and compare wild bootstrap methods against traditional sandwich variance estimation, finding that wild bootstrap offers more robust and reliable variance estimates, especially in challenging scenarios like small sample sizes or when models are misspecified.

What Problem Does This Paper Address?

In observational studies, we often want to understand the effect of a treatment or intervention. However, unlike randomized controlled trials, we can't randomly assign participants, leading to potential biases. Propensity score methods are a common way to adjust for these differences. A key challenge is not just estimating the average treatment effect (ATE), but also the effect specifically for those who received the treatment (ATT) and those who didn't (ATC). Estimating the uncertainty around these ATT and ATC estimates is crucial for drawing valid conclusions. Traditional methods for variance estimation can be unreliable, especially when dealing with complex models, limited data, or when assumptions are violated. This paper aims to provide and validate more robust methods for this crucial step.

Methods

The core of this paper is about variance estimation for doubly robust estimators of ATT and ATC. Doubly robust estimators are desirable because they remain consistent even if one of the underlying models (propensity score or outcome model) is misspecified.

The authors propose and investigate three main approaches:

  • Asymptotic Sandwich Variance Estimation: This is a standard, large-sample method that relies on estimating equations. It's theoretically sound but can be sensitive to model misspecification and small sample sizes.
  • Standard Bootstrap: This involves resampling the data with replacement. While generally useful, it can exacerbate issues with extreme weights or small sample sizes in propensity score methods.
  • Wild Bootstrap: This is the novel contribution. Instead of resampling the data, it perturbs the influence functions of the estimators using independent random variables. This approach is designed to be more robust to issues like heteroscedasticity and model misspecification, and it doesn't rely on resampling the original data, which can be problematic.

They also explore the performance of these methods under various simulation scenarios, including different treatment effect heterogeneities, proportions of treated participants, and model misspecifications.

Datasets

The paper uses a real-world dataset from the Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments (SUPPORT). This dataset concerns the use of right heart catheterization (RHC) in critically ill patients and focuses on its effect on patients' length of stay in the ICU. The data is publicly accessible.

Key Insights and Ideas

  • Wild bootstrap is a powerful alternative: The wild bootstrap methods consistently outperform traditional sandwich variance estimation and standard bootstrap, particularly in challenging scenarios. This is a significant finding for researchers working with observational data.
  • Model misspecification is a major hurdle: The performance of all variance estimation methods degrades when either the propensity score or outcome regression models are misspecified. However, wild bootstrap shows greater resilience.
  • ATT vs. ATC estimation has different sensitivities: The paper highlights that estimating ATC can be more sensitive to issues like small sample sizes and extreme propensity scores compared to ATT.
  • The importance of influence functions: The wild bootstrap's reliance on perturbing influence functions is key to its robustness, as influence functions capture the core mathematical structure of the estimator.

Key Results

  • In extensive simulation studies, the two wild bootstrap methods (Rademacher and exponential distributions) generally provided the best variance estimations, showing good coverage probabilities (close to 0.95) and robust performance across various scenarios, including model misspecification.
  • The sandwich variance estimation performed reasonably well but was less robust to model misspecification and could fail entirely in small sample sizes with extreme weights (as seen in Model 5a/5b).
  • The standard bootstrap also performed well but was sometimes less robust to model misspecification than the wild bootstrap.
  • The TMLE (Targeted Maximum Likelihood Estimation) method sometimes showed larger biases and was more sensitive to model misspecifications, potentially due to its specific implementation in the study.
  • In the SUPPORT data analysis, the sandwich variance estimation was found to be the most efficient for ATT, but wild bootstrap methods showed competitive performance and better robustness in other scenarios.

Limitations and Future Work

  • Small sample size challenges: While wild bootstrap is more robust, small sample sizes still pose challenges, and results from such scenarios should be interpreted with caution. The paper notes that standard bootstrap can be useful in very small samples, but its results are often unstable.
  • Model selection: The paper emphasizes that careful, domain-knowledge-driven model specification is crucial. Blindly using automated variable selection methods (like "kitchen-sink" approaches or stepwise selection) is discouraged as it can lead to biased or inefficient estimators.
  • Future research: The authors suggest further investigations are warranted to assess the final-sample performance of these methods for various estimands, especially in comparison to other weighting schemes (overlap weights, matching weights, etc.) and for classic ATE estimation. They also highlight the need for threshold-independent estimators for extreme weights.

Summary updated

Abstract

Common causal estimands include the average treatment effect, the average treatment effect of the treated, and the average treatment effect on the controls. Using augmented inverse probability weighting methods, parametric models are judiciously leveraged to yield doubly robust estimators, that is, estimators that are consistent when at least one the parametric models is correctly specified. Three sources of uncertainty are associated when we evaluate these estimators and their variances, that is, when we estimate the treatment and outcome regression models as well as the desired treatment effect. In this article, we propose methods to calculate the variance of the normalized, doubly robust average treatment effect of the treated and average treatment effect on the controls estimators and investigate their finite sample properties. We consider both the asymptotic sandwich variance estimation, the standard bootstrap as well as two wild bootstrap methods. For the asymptotic approximations, we incorporate the aforementioned uncertainties via estimating equations. Moreover, unlike the standard bootstrap procedures, the proposed wild bootstrap methods use perturbations of the influence functions of the estimators through independently distributed random variables. We conduct an extensive simulation study where we vary the heterogeneity of the treatment effect as well as the proportion of participants assigned to the active treatment group. We illustrate the methods using an observational study of critical ill patients on the use of right heart catherization.
BibTeX
@inproceedings{matsouaka2022variance,
  author = {Roland Matsouaka and Yi Liu and Yunji Zhou},
  title = {{Variance estimation for the average treatment effects on the treated and on the controls}},
  booktitle = {Statistical},
  year = {2022},
  doi = {10.1177/09622802221142532},
}
Figure