Combining Experimental and Observational Data for Identification and Estimation of Long-Term Causal Effects

AmirEmad Ghassami, Alan Hao Yang, David J. Richardson, Ilya Shpitser, Eric Tchetgen TchetgenarXiv 2022 · DOI · arXiv

Introduction

Estimating the long-term causal impact of an intervention is a fundamental challenge in social science, medicine, and public policy. While Randomized Controlled Trials (RCTs) are the gold standard for establishing causality, they are often limited by short follow-up periods. Conversely, observational studies provide long-term data but are frequently plagued by unobserved confounding. This paper presents a principled framework for "data fusion," combining the strengths of both data sources to identify long-term effects that neither could reveal alone.

The Data Fusion Challenge

The authors address a scenario where researchers have access to two distinct domains:

  1. Experimental Domain ($G=E$): Subjects are randomized into treatment ($A$), and short-term outcomes ($M$) are recorded. However, long-term outcomes ($Y$) are missing.
  2. Observational Domain ($G=O$): Both short-term ($M$) and long-term ($Y$) outcomes are recorded, but the treatment assignment is non-random and influenced by unobserved confounders ($U$).

The goal is to estimate the Average Treatment Effect (ATE) or the Effect of Treatment on the Treated (ETT) on the long-term outcome $Y$. The central difficulty lies in the fact that the observational data is biased, and the experimental data is incomplete.

Causal structures for pooled, experimental, and observational domains, including Single World Intervention Graphs (SWIGs).

Proposed Methodologies

The paper introduces three novel identification strategies, each relying on different structural assumptions about the relationship between the domains.

1. Equi-Confounding Bias

This approach assumes that the magnitude of the confounding bias introduced by $U$ is identical for both the short-term outcome $M$ and the long-term outcome $Y$. By using the experimental data to identify the "true" (unbiased) effect on $M$, researchers can calculate the bias in the observational domain and subtract it from the observed long-term effect. This generalizes the Difference-in-Differences (DiD) logic to settings where the "pre-test" variable ($M$) is itself affected by the treatment.

2. Bespoke Instrumental Variable (BSIV)

Recognizing that bias may not always be perfectly equal across time, the BSIV approach leverages a specific observed covariate. This covariate must share a partial additive association with both the short-term and long-term potential outcomes. This provides a more flexible bridge between the two outcomes than the strict equi-confounding assumption.

3. Proximal Data Fusion

This method extends proximal causal inference to the data fusion setting. It utilizes a "proxy" variable that tracks the latent confounder $U$ but does not directly influence the outcome. A key innovation here is that while standard proximal methods typically require two proxies (one for treatment and one for outcome), the presence of an experimental "anchor" allows identification with only a single proxy.

Robust Estimation

For each identification strategy, the authors derive influence function-based estimators. These estimators possess "multiple robustness" properties, meaning they remain consistent even if some components of the statistical model (such as the propensity score or the outcome bridge functions) are misspecified, provided a specific subset of the models is correct.

Empirical Application: Class Size and Student Achievement

The methods were applied to estimate the effect of class size on 8th-grade SAT scores. The authors fused data from Project STAR (an RCT measuring effects through 3rd grade) with the Early Childhood Longitudinal Study (ECLS-K) (an observational study tracking students through 8th grade).

While naive observational models or previous data fusion techniques often yielded negligible or counter-intuitive results, the proposed methods identified a significant positive causal effect of smaller class sizes on long-term academic achievement. These findings align with established educational research, demonstrating the practical utility of the framework.

Conclusion

By treating short-term experimental results as an "anchor" for confounded observational data, this work provides a robust path forward for long-term causal inference. The proposed estimators offer a principled way to navigate the trade-offs between the internal validity of experiments and the longitudinal scope of observational studies.

Summary updated

Abstract

We study identifying and estimating the causal effect of a treatment variable on a long-term outcome using data from an observational and an experimental domain. The observational data are subject to unobserved confounding. Furthermore, subjects in the experiment are only followed for a short period; thus, long-term effects are unobserved, though short-term effects are available. Consequently, neither data source alone suffices for causal inference on the long-term outcome, necessitating a principled fusion of the two. We propose three approaches for data fusion for the purpose of identifying and estimating the causal effect. The first assumes equal confounding bias for short-term and long-term outcomes. The second weakens this assumption by leveraging an observed confounder for which the short-term and long-term potential outcomes share the same partial additive association with this confounder. The third approach employs proxy variables of the latent confounder of the treatment-outcome relationship, extending the proximal causal inference framework to the data fusion setting. For each approach, we develop influence function-based estimators and analyze their robustness properties. We illustrate our methods by estimating the effect of class size on 8th-grade SAT scores using data from the Project STAR experiment combined with observational data from the Early Childhood Longitudinal Study.
BibTeX
@misc{ghassami2022combining,
  author = {AmirEmad Ghassami and Alan Hao Yang and David J. Richardson and Ilya Shpitser and Eric Tchetgen Tchetgen and Tchetgen, Eric Tchetgen},
  title = {{Combining Experimental and Observational Data for Identification and Estimation of Long-Term Causal Effects}},
  year = {2022},
  eprint = {2201.10743},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.10743},
}
Figure