Causal inference methods for combining randomized trials and observational studies: a review

Awa DiengOpenReview 2022 · arXiv · OpenReview

TL;DR

This paper reviews methods for combining data from randomized controlled trials (RCTs) and observational studies to improve causal inference. By leveraging the strengths of both data types, researchers can achieve better generalizability and more robust estimates of treatment effects, especially when dealing with unmeasured confounding or limited sample sizes.

What Problem Does This Paper Address?

Randomized controlled trials (RCTs) are the gold standard for establishing causal effects due to their ability to minimize confounding. However, RCTs often suffer from limited external validity because their highly selected participant pools may not represent the broader target population. Conversely, large observational studies offer better representativeness but are prone to confounding bias, making it difficult to isolate the true treatment effect. This paper addresses the challenge of combining these two data sources to gain the benefits of both: the internal validity of RCTs and the external validity and representativeness of observational data. It explores methods to generalize RCT findings to target populations and to improve the credibility and efficiency of causal inference by integrating experimental and observational evidence.

Methods

The paper provides a comprehensive review of methods for combining RCTs and observational studies, categorizing them based on the type of data available and the specific causal inference problem addressed. Key methods discussed include:

  • Generalizability/Transportability: Techniques to extend RCT findings to a target population using observational data. This involves methods like Inverse Probability of Sampling Weighting (IPSW), stratification, and plug-in g-formula estimators. These methods aim to adjust for differences in covariate distributions between the RCT sample and the target population.
  • Dealing with Unmeasured Confounding: Approaches that use RCTs as a "negative control" to help deconfound observational data or to improve estimation efficiency when observational data might be biased.
  • Doubly Robust Estimators: Methods like Augmented IPSW (AIPSW) that combine models for both treatment assignment (or sampling) and outcome to achieve consistent estimates if at least one of the models is correctly specified.
  • Structural Causal Models (SCMs): The paper also connects these methods to the SCM framework, highlighting how graphical models and do-calculus can provide a principled way to express causal assumptions and derive identification strategies, particularly for transportability and data fusion.

The review also discusses practical considerations like handling missing data and the choice of covariates.

Datasets

The paper uses two primary datasets for its application and simulation studies:

  • CRASH-3 Trial: A large, multi-center, randomized, placebo-controlled trial investigating the effect of tranexamic acid (TXA) on head-injury-related death in patients with traumatic brain injury (TBI).
  • Traumabase: A registry of over 30,000 French trauma patients, from which 8,270 TBI patients were extracted for analysis. This dataset represents a more general, observational population.

The authors also provide code for implementing many of the discussed methods, facilitating reproducible research.

Key Insights and Ideas

  • Complementary Strengths: RCTs offer internal validity, while observational studies offer external validity and representativeness. Combining them can mitigate the weaknesses of each.
  • Generalizability is Key: A major focus is on how to generalize findings from a potentially unrepresentative RCT to a broader, more representative target population described by observational data.
  • Unmeasured Confounding is a Major Hurdle: The paper highlights how RCTs can be instrumental in addressing unmeasured confounding in observational data, a persistent challenge in causal inference.
  • SCM Framework Offers a Unified View: Structural Causal Models provide a powerful graphical language to express complex causal assumptions and derive identification strategies, bridging different approaches to causal inference.
  • Practical Implementation Matters: The review emphasizes the need for accessible software and clear guidance on applying these methods, especially given the variety of approaches and their specific assumptions.

Key Results

The paper's primary contribution is a comprehensive review of the literature on combining RCTs and observational data for causal inference. The application to TXA's effect on TBI mortality demonstrated:

  • Discrepancies and Concordance: When generalizing RCT findings to the Traumabase population, some estimators supported the CRASH-3 trial's conclusion of no significant effect, while others suggested a deleterious effect. This highlights the sensitivity of results to the chosen methods and assumptions.
  • Impact of Missing Data: The analysis showed that missing values, particularly prevalent in the observational Traumabase data, significantly impact the performance and reliability of estimators, posing an ongoing research challenge.
  • Model Specification is Crucial: Simulations illustrated that while doubly robust estimators (like AIPSW) are robust to some model mis-specification, other methods (like IPSW or plug-in g-formula) can be biased if their underlying models are incorrect. Calibration weighting showed some robustness to slight mis-specification.
  • Stratification Benefits: For stratification estimators, increasing the number of strata generally improved prediction accuracy.

Limitations and Future Work

  • Handling Missing Values: The paper acknowledges that dealing with missing values, especially when they differ in patterns between data sources, remains a significant challenge and an open research question.
  • Covariate Selection: Identifying the right set of covariates for adjustment and generalization is complex and requires careful consideration of domain expertise and causal graphs. The impact of including non-treatment-effect-modifying but shifted covariates on variance inflation is noted.
  • Target Population Definition: Defining a clinically relevant target population, especially when considering specific strata (e.g., severe TBI patients), can be challenging and may require further methodological development for transportability.
  • Assumptions: The validity of all methods relies on strong assumptions (e.g., ignorability, positivity, S-admissibility), which need careful assessment in practice.
  • Future Work: The authors suggest further research into covariate selection, handling missing values in combined data analyses, and developing methods for generalizing effects to stratified subpopulations. They also point to the ongoing development of software and theoretical guarantees for these methods.

Summary updated

Abstract

With increasing data availability, causal effects can be evaluated across different datasets, both randomized controlled trials (RCTs) and observational studies. RCTs isolate the effect of the treatment from that of unwanted (confounding) co-occurring effects. But they may struggle with inclusion biases, and thus lack external validity. On the other hand, large observational samples are often more representative of the target population but can conflate confounding effects with the treatment of interest. In this paper, we review the growing literature on methods for causal inference on combined RCTs and observational studies, striving for the best of both worlds. We first discuss identification and estimation methods that improve generalizability of RCTs using the representativeness of observational data. Classical estimators include weighting, difference between conditional outcome models, and doubly robust estimators. We then discuss methods that combine RCTs and observational data to improve (conditional) average treatment effect estimation, handling possible unmeasured confounding in the observational data. We also connect and contrast works developed in both the potential outcomes literature and the structural causal model literature. Finally, we compare the main methods using a simulation study and real world data to analyze the effect of tranexamic acid on the mortality rate in major trauma patients. Code to implement many of the methods is provided.
BibTeX
@misc{dieng2022causal,
  author = {Awa Dieng},
  title = {{Causal inference methods for combining randomized trials and observational studies: a review}},
  year = {2022},
  eprint = {2011.08047},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.08047},
  howpublished = {OpenReview},
}
Figure