A Practical Analysis Procedure on Generalizing Comparative Effectiveness in the Randomized Clinical Trial to the Real-world Trialeligible Population

Kuan Jiang, Xinxing Lai, Shu Yang, Ying Gao, Xiao‐Hua Zhou

TL;DR

This paper introduces a robust statistical framework to bridge the gap between controlled clinical trials and real-world medical practice. By combining "doubly robust" estimation with advanced sensitivity analysis, the authors demonstrate how to accurately project trial results onto real-world populations, accounting for both known demographic shifts and potential unmeasured biases.

What Problem Does This Paper Address?

Randomized Controlled Trials (RCTs) are the "gold standard" for proving a drug works because randomization eliminates bias. However, RCTs often have strict inclusion criteria, resulting in a "trial population" that looks very different from the average patient in a clinic. This creates a gap in external validity: a drug might appear highly effective in a trial but perform differently in the real world due to "covariate distribution shifts" (e.g., real-world patients being older or having more comorbidities).

The authors tackle two main challenges:

  1. Generalization: How do we mathematically "transport" RCT results to a target real-world population?
  2. Robustness: How can we be sure our results aren't being skewed by "unmeasured confounders"—important variables that weren't recorded in the data?

Methods

The authors propose a two-step procedure:

  • Estimation via AIPSW: They use the Augmented Inverse Probability of Sampling Weighting (AIPSW) estimator. This is a "doubly robust" method, meaning it provides a consistent estimate if either the model for trial participation or the model for the drug's outcome is correctly specified. It essentially re-weights the trial participants to match the demographic profile of the real-world population.
  • Sensitivity Analysis: To address the risk of unobserved variables, they adapt an omitted variable bias framework. They calculate "Robustness Values" and create bias contour plots to determine how strong an unmeasured factor (a "killer confounder") would need to be to flip their conclusions.

Datasets

The study applies this framework to a comparative effectiveness study of hypertension treatments:

  • RCT Data: 602 patients with Grade I hypertension, randomized to receive either Songling Xuemaikang Capsule (SXC, a traditional Chinese medicine) or Losartan (a standard Western drug).
  • Real-World Data (RWD): A registry study of 3,000 patients. After "trimming" the data to include only those who would have been eligible for the trial (the "trial-eligible population"), 804 patients remained.
  • Variables: The models utilized seven common covariates: Age, Sex, BMI, Marriage status, Smoking status, and baseline systolic/diastolic blood pressure.

Key Insights and Ideas

  • The "Real World" is Older: A key finding was that the real-world trial-eligible population was significantly older and had lower baseline diastolic blood pressure than the RCT participants. This demographic shift is exactly why generalization is necessary.
  • Time-Dependent Efficacy: The discrepancy between the RCT and the real-world projection was most visible in the short term. While the RCT suggested the drugs were comparable throughout, the real-world projection showed SXC was initially less effective but caught up to Losartan by week 8.
  • Quantifying the Unknown: The sensitivity analysis provided a "benchmarking" approach. For example, to invalidate the long-term results for systolic blood pressure, an unmeasured variable would need to be 58 times stronger than gender or 24 times stronger than age. Since researchers usually collect the most important variables first, the existence of such a massive "killer confounder" is highly unlikely, making the results robust.

Key Results

  • Systolic Blood Pressure (SBP): In the RCT, SXC appeared non-inferior to Losartan. However, when generalized to the real world, SXC was significantly less effective at weeks 2, 4, and 6. By week 8, the difference disappeared, and the two treatments were comparable.
  • Diastolic Blood Pressure (DBP): Similar to SBP, the generalized results showed SXC was inferior at week 4 but reached parity by weeks 6 and 8.
  • Conclusion: The study suggests that for real-world trial-eligible populations, SXC is a viable long-term alternative to Losartan, though it may work more slowly in the initial weeks.

Limitations and Future Work

  • Statistical vs. Evidence-Based: The authors acknowledge they relied on statistical modeling rather than direct real-world evidence (which would require a controlled trial in the real-world setting—a difficult and expensive task).
  • Trial-Eligible Only: The study dropped real-world patients who didn't meet the RCT's original inclusion/exclusion criteria. This means the findings don't necessarily apply to the entire real-world population, only the "trial-eligible" slice.
  • Future Directions: The authors suggest exploring "non-positivity" problems—scenarios where certain real-world subgroups have zero representation in the trial data, requiring even more complex synthesis models.

Summary updated

Abstract

When evaluating the effectiveness of a drug, a Randomized Controlled Trial (RCT) is often considered the gold standard due to its perfect randomization. While RCT assures strong internal validity, its restricted external validity poses challenges in extending treatment effects to the broader real-world population due to possible heterogeneity in covariates. In this paper, we introduce a procedure to generalize the RCT findings to the real-world trial-eligible population based on the adaption of existing statistical methods. We utilized the augmented inversed probability of sampling weighting (AIPSW) estimator for the estimation and omitted variable bias framework to assess the robustness of the estimate against the assumption violation caused by potentially unmeasured confounders. We analyzed an RCT comparing the effectiveness of lowering hypertension between Songling Xuemaikang Capsule (SXC), a traditional Chinese medicine (TCM), and Losartan as an illustration. The generalization results indicated that although SXC is less effective in lowering blood pressure than Losartan on week 2, week 4, and week 6, there is no statistically significant difference among the trial-eligible population at week 8, and the generalization is robust against potential unmeasured confounders.
BibTeX
@article{jiang2025practical,
  author = {Kuan Jiang and Xinxing Lai and Shu Yang and Ying Gao and Xiao‐Hua Zhou},
  title = {{A Practical Analysis Procedure on Generalizing Comparative Effectiveness in the Randomized Clinical Trial to the Real-world Trialeligible Population}},
  journal = {Journal of Biopharmaceutical Statistics},
  year = {2025},
  doi = {10.1080/10543406.2025.2489282},
}
Figure