On the Comparative Analysis of Average Treatment Effects Estimation via Data Combination

Peng Wu, Shanshan Luo, Zhi GengJournal of the American Statistical Association 2024 · DOI

TL;DR

This paper tackles the challenge of estimating causal effects in a target population by combining data from a source dataset and a target dataset. It provides a comprehensive comparative analysis of different data structures and assumptions, revealing key factors that drive efficiency gains, particularly highlighting the often-overlooked role of "posterior drift." The authors also introduce novel methods for sensitivity analysis to assess the robustness of these estimations.

What Problem Does This Paper Address?

Estimating causal effects (like the impact of a drug or policy) in a specific "target" population is crucial, but often challenging due to data limitations. Researchers increasingly have access to multiple datasets from different sources or studies. Combining these datasets can improve the precision of causal effect estimates and help generalize findings. However, existing methods for data combination are often tailored to specific scenarios and lack a thorough comparative analysis across various data structures and assumptions. This paper addresses this gap by systematically investigating how different data availability scenarios and assumptions (like covariate shift and posterior drift) impact the efficiency of average treatment effect (ATE) estimation. It aims to uncover why certain data combinations and assumptions lead to better results and to provide tools for assessing the reliability of these estimates.

Methods

The core of this paper lies in a detailed theoretical analysis using semiparametric efficiency bounds. The authors derive efficient influence functions (EIFs) for the ATE under various settings, which represent the best possible statistical precision achievable. They then compare these efficiency bounds across different scenarios to quantify the impact of data availability and assumptions.

Key methodological contributions include:

  • Systematic Setting Design: Six distinct settings are defined, varying in the information available in the target dataset (covariates only, covariates and treatment, etc.) and incorporating both covariate shift and posterior drift.
  • Semiparametric Efficiency Bounds: The authors derive theoretical efficiency bounds for the ATE under these settings, providing a principled way to compare estimation strategies.
  • Identification of Key Factors: The analysis reveals that efficiency gains are influenced by factors like the variance ratio of potential outcomes between datasets and the derivatives of the posterior drift function.
  • Novel Sensitivity Analysis: A new approach is proposed to assess how sensitive the ATE estimates are to violations of the transportability assumptions between datasets.
  • Double Machine Learning Estimators: The paper proposes and analyzes the large-sample properties of efficient estimators for the ATE, leveraging machine learning techniques.

Datasets

The paper does not introduce new datasets. Instead, it uses simulated data for its extensive simulation studies. The authors also demonstrate their methods on two real-world applications:

  • Job Training Program: Using experimental data from the National Supported Work (NSW) program as the source and non-experimental data from the Population Survey of Income Dynamics (PSID) as the target.
  • MIMIC-III and eICU Databases: Using data from these clinical databases for patients with sepsis, with MIMIC-III as the target and eICU as the source.

Key Insights and Ideas

  • Posterior Drift Matters: The paper strongly emphasizes the crucial, and often overlooked, role of "posterior drift" (differences in the conditional distribution of potential outcomes) in influencing estimation efficiency. Ignoring this can lead to suboptimal results.
  • "Effective Sample Size" is Dynamic: The efficiency gains from data combination are not simply additive. The "effective sample size" is influenced by complex interactions, including the variance ratio of potential outcomes and the posterior drift, acting as an "optimal transfer rate."
  • Assumptions vs. Data: The study clarifies that while having more data is beneficial, the assumptions made about the relationship between datasets (e.g., transportability, ignorability) are equally, if not more, critical for achieving efficient and reliable causal inference.
  • Sensitivity Analysis is Essential: For real-world applications, especially when transportability assumptions are uncertain, performing sensitivity analysis is crucial to understand the range of plausible causal effects.

Key Results

  • The paper provides a comprehensive comparative analysis of efficiency bounds across 12 different settings (combinations of data structures and assumptions).
  • The theoretical derivations show that under covariate shift alone, additional data (treatment and outcome) can enhance efficiency, but the impact of assumptions is paramount.
  • When posterior drift is considered, the variance ratio of potential outcomes and the derivatives of the posterior drift function become critical drivers of efficiency.
  • Simulations confirm that the proposed estimators achieve significant efficiency gains compared to using only the target data, especially when sampling scores are small.
  • The sensitivity analysis in real-world applications demonstrates that estimates can be robust to certain variations in posterior drift assumptions, but highlights the importance of exploring these variations.

Limitations and Future Work

  • Focus on ATE: The primary focus is on the Average Treatment Effect (ATE). Extending the analysis to other estimands like the Average Treatment Effect on the Treated (ATT) is mentioned and explored in later sections.
  • Two Datasets: The current framework primarily considers a single source and a single target dataset. Extending this to multiple source and target datasets is identified as a key area for future research.
  • Partial Covariates: The paper acknowledges that scenarios where only partial covariates are observed in the target dataset (a subset of those in the source) warrant further investigation.
  • Generalizability of Findings: While the paper provides a theoretical framework and simulations, the practical applicability of the findings will depend on the ability to accurately model posterior drift and perform robust sensitivity analyses in specific domains.

Summary updated

BibTeX
@article{wu2024comparative,
  author = {Peng Wu and Shanshan Luo and Zhi Geng},
  title = {{On the Comparative Analysis of Average Treatment Effects Estimation via Data Combination}},
  journal = {Journal of the American Statistical Association},
  year = {2024},
  doi = {10.1080/01621459.2024.2435656},
}
Figure