Improve Sensitivity Analysis Synthesizing Randomized Clinical Trials With Limited Overlap

Kuan Jiang, Wenjie Hu, Shu Yang, Xinxing Lai, Xiao‐Hua ZhouarXiv 2024 · DOI · arXiv

TL;DR

This paper introduces a "synthesis estimator" that combines the internal validity of Randomized Clinical Trials (RCTs) with the broad reach of Observational Studies (OS). By using RCT data to "anchor" uncertainty in trial-eligible populations, the authors provide a way to generate much tighter bounds for treatment effects in the real world, even when the RCT and the general population only partially overlap.

What Problem Does This Paper Address?

In medical research, we face a classic trade-off. Randomized Clinical Trials (RCTs) are the gold standard for proving a drug works because randomization eliminates "confounding" (hidden factors that influence both treatment and outcome). However, RCTs often have strict inclusion/exclusion criteria, meaning their results might not apply to the broader, "messier" real-world population.

Observational Studies (OS), on the other hand, reflect the real world but are plagued by unmeasured confounders. To deal with this, researchers use sensitivity analysis to estimate a range (bounds) of possible treatment effects. The problem? Without extra information, these bounds are often so wide they are practically useless for decision-making.

Existing methods to bridge this gap usually require "positivity"—the assumption that everyone in the real world had a non-zero chance of being in the RCT. In reality, many patients (e.g., the elderly or those with comorbidities) are explicitly excluded from trials, breaking these models.

Methods

The authors propose a synthesis formulation that splits the real-world population into two groups:

  1. The Trial-Eligible Group ($V^*=0$): People who could have participated in the RCT.
  2. The Trial-Ineligible Group ($V^*=1$): People excluded from the RCT.

The core innovation is the synthesis sensitivity analysis estimator. It calculates the Average Treatment Effect (ATE) for the eligible group using the RCT data (which has no unmeasured confounding). It then uses sensitivity analysis only for the ineligible group using OS data.

To make this work, they introduce a "separability" condition. This theoretical framework allows researchers to decompose the global uncertainty parameters into sub-population parameters. By "fixing" the uncertainty for the trial-eligible portion using the RCT's "ground truth," the overall bounds for the entire population naturally shrink.

Datasets

The researchers validated their approach using:

  • Simulated Data: Two scenarios designed to test the estimator under different levels of population overlap and confounding strengths.
  • Real-World Application: A study comparing the effectiveness of Songling Xuemaikang Capsule (SXC), a traditional Chinese medicine, against Losartan for lowering blood pressure.
  • RCT Dataset: 602 participants (strictly aged $\leq$ 60 with grade I hypertension).
  • OS Dataset: 3,000 hypertensive patients from a real-world registry with no age or grade restrictions.

Key Insights and Ideas

  • RCTs as Anchors: The most powerful idea here is that an RCT doesn't need to represent the whole population to be useful. It acts as an anchor that eliminates uncertainty for a specific slice of the population, which mathematically forces the bounds for the total population to narrow.
  • Overcoming Positivity Violation: Unlike previous "generalizability" methods that fail when people are excluded from trials, this method embraces the exclusion. It acknowledges that we have different levels of evidence for different people and synthesizes them accordingly.
  • Model Agnostic: The synthesis approach works with various standard estimators (Outcome Regression, Inverse Probability Weighting, etc.), making it a flexible tool for different types of data.

Key Results

  • Tighter Bounds: In simulations, the synthesis method consistently produced narrower bounds than OS-only analysis. For example, in one scenario, the mean bound width dropped from 23.79 (OS only) to roughly 8.3 (Synthesis).
  • Real-World Impact: In the blood pressure drug study, the OS-only sensitivity analysis gave a wide bound for Systolic Blood Pressure reduction: [-3.55, 4.96]. This range includes zero, meaning we couldn't be sure which drug was better.
  • The Synthesis Result: By incorporating the RCT data, the bounds narrowed to [-2.61, 3.31]. While the result still crossed zero (indicating no statistically significant difference), the uncertainty was reduced by nearly 30%, providing much clearer evidence for clinicians.

Limitations and Future Work

  • The Generalizability Assumption: The method relies on "mean generalizability," assuming that the trial-eligible people in the OS behave similarly to those in the RCT. If there are hidden "implicit" exclusion criteria in the RCT that aren't captured in the OS data, the results could be biased.
  • Separability Constraints: The current proof focuses on "separable" sensitivity parameters. The authors note that more complex, non-separable parameters (like those used in some non-linear models) require further research.
  • Outcome Complexity: The study focused on continuous outcomes (blood pressure). The authors suggest that extending this to survival analysis (time-to-event data) is a necessary next step for oncology and chronic disease research.

Summary updated

Abstract

Randomized clinical trials are the gold standard when estimating the average treatment effect. However, they are usually not a random sample from the real-world population because of the inclusion/exclusion rules. Meanwhile, observational studies typically consist of representative samples from the real-world population. However, due to unmeasured confounding, sensitivity analysis is often used to estimate bounds for the average treatment effect without relying on stringent assumptions of other existing methods. This article introduces a synthesis estimator that improves sensitivity analysis in observational studies by incorporating randomized clinical trial data, even when overlap in covariate distribution is limited due to inclusion/exclusion criteria. We show that the proposed estimator will give a tighter bound when a "separability" condition holds for the sensitivity parameter. Theoretical proofs and simulations show that this method provides a tighter bound than the sensitivity analysis using only observational study. We apply this method to combine an observational study on drug effectiveness with a partially overlapping RCT dataset, yielding improved average treatment effect bounds.
BibTeX
@misc{jiang2024improve,
  author = {Kuan Jiang and Wenjie Hu and Shu Yang and Xinxing Lai and Xiao‐Hua Zhou},
  title = {{Improve Sensitivity Analysis Synthesizing Randomized Clinical Trials With Limited Overlap}},
  year = {2024},
  eprint = {2409.07391},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.07391},
}
Figure