A practical analysis procedure on generalizing comparative effectiveness in the randomized clinical trial to the real-world trial-eligible population

Kuan Jiang, Xinxing Lai, Shu Yang, Ying Gao, Xiao‐Hua Zhou

TL;DR

This paper introduces a practical statistical framework to bridge the gap between randomized controlled trial (RCT) results and real-world patient populations. By combining an advanced estimation technique (AIPSW) with sensitivity analysis, the authors demonstrate how to generalize drug effectiveness findings, ensuring they remain robust even when unmeasured factors might be at play.

What Problem Does This Paper Address?

Randomized controlled trials (RCTs) are the gold standard for evaluating drug efficacy because their randomization helps control for many confounding factors, leading to strong internal validity. However, this controlled environment often doesn't perfectly reflect the diverse patient populations encountered in real-world clinical practice. This "external validity" problem means that findings from an RCT might not directly translate to the broader population eligible for treatment. Differences in patient characteristics (covariates) between the RCT and the real-world population can lead to different treatment effects. This paper tackles the challenge of generalizing RCT findings to a real-world, trial-eligible population, accounting for these potential discrepancies.

Methods

The core of the paper's methodology lies in two key components:

  • Augmented Inverse Probability of Sampling Weighting (AIPSW) Estimator: This statistical technique is used to adjust for the differences in covariate distributions between the RCT sample and the real-world trial-eligible population. AIPSW models both the probability of being selected into the RCT and the treatment assignment within the RCT simultaneously. This "doubly robust" approach means that the estimation remains consistent if either the selection model or the outcome model is correctly specified, offering a degree of resilience.
  • Omitted Variable Bias Framework for Sensitivity Analysis: To address the concern of unmeasured confounders (factors not accounted for in the data), the authors employ a sensitivity analysis based on Huang's method. This approach assesses how much unmeasured confounding would be needed to overturn the study's conclusions. It involves quantifying the "confounding strength" of potential unmeasured variables relative to observed covariates, helping to determine if the generalization results are robust.

Datasets

The study utilizes two datasets:

  • A Randomized Controlled Trial (RCT): This multicenter, double-blind, non-inferiority trial compared Songling Xuemaikang Capsule (SXC) with Losartan for mild essential hypertension in patients aged 18-65.
  • A Real-World Registry Study: This dataset comprises patients from real-world practice who used SXC (alone or in combination with other medications) for hypertension.

The real-world data was preprocessed to create a "trial-eligible population" by selecting individuals whose covariates were comparable to those in the RCT data.

Key Insights and Ideas

  • RCTs are not always representative: The paper strongly reinforces the idea that the controlled environment of an RCT, while excellent for internal validity, can lead to findings that don't directly apply to the broader patient population.
  • Covariate shift matters: Differences in patient characteristics (like age, BMI, etc.) between RCTs and real-world settings can significantly alter treatment effect conclusions.
  • AIPSW offers a robust solution: The AIPSW estimator provides a statistically sound way to adjust for these covariate differences, allowing for more reliable generalization of RCT findings.
  • Sensitivity analysis is crucial: Simply adjusting for observed covariates isn't enough. The paper highlights the importance of assessing how unmeasured factors might influence the results, ensuring the conclusions are not overly sensitive to unobserved confounders.
  • "Killer confounders" are a concern: The sensitivity analysis framework helps identify the threshold of confounding strength that could completely reverse the study's findings, allowing researchers to assess the risk of such "killer confounders."

Key Results

The study analyzed the effectiveness of Songling Xuemaikang Capsule (SXC) versus Losartan for hypertension.

  • Short-term (weeks 2-6): While the RCT showed no significant difference in lowering systolic blood pressure (SBP) between SXC and Losartan, the generalized results to the real-world population indicated that SXC was significantly less effective. For diastolic blood pressure (DBP), SXC was inferior to Losartan at week 4 but similar at weeks 6 and 8.
  • Long-term (week 8): For SBP, the generalized results showed no statistically significant difference between SXC and Losartan at week 8. For DBP, the conclusion from both RCT and generalized results was that SXC was not inferior to Losartan.
  • Robustness: Sensitivity analysis indicated that the generalization results were robust against potential unmeasured confounders, meaning that even with unobserved factors, the conclusions about the lack of significant difference in long-term efficacy (at week 8) were likely to hold.

Limitations and Future Work

  • Statistical vs. Evidence-Based Generalization: The study relied on statistical generalization, meaning they didn't have direct access to real-world trial-eligible population data to verify their findings. The authors acknowledge that obtaining such evidence would be challenging.
  • Exclusion of Outpatients: Patients in the real-world cohort who did not meet the RCT's inclusion/exclusion criteria were dropped, limiting the generalization to only the trial-eligible population.
  • Future Directions: The paper suggests that their framework provides a guideline for using SXC to treat hypertension in real-world settings and that a combination of SXC and Losartan might offer a better balance of efficacy and side effects. Further exploration of the mechanisms behind SXC's effectiveness, particularly in older populations, is implied.

Summary updated

Abstract

When evaluating the effectiveness of a drug, a randomized controlled trial (RCT) is often considered the gold standard due to its ability to balance effect modifiers through randomization. While RCT assures strong internal validity, its restricted external validity poses challenges in extending treatment effects to the broader real-world population due to possible heterogeneity in covariates. In this paper, we introduce a procedure to generalize the RCT findings to the real-world trial-eligible population based on the adaption of existing statistical methods. We utilized the augmented inversed probability of sampling weighting (AIPSW) estimator for the estimation and omitted variable bias framework to assess the robustness of the estimate against the assumption violation caused by potentially unmeasured confounders. We analyzed an RCT comparing the effectiveness of lowering hypertension between Songling Xuemaikang Capsule (SXC) - a traditional Chinese medicine (TCM), and Losartan as an illustration. Based on current evidence, the generalization results indicated that by adjusting covariates distribution shift, although SXC is less effective in lowering blood pressure than Losartan on week 2, there is no statistically significant difference among the trial-eligible population at weeks 4-8. In addition, sensitivity analysis further demonstrated that the generalization is robust against potential unmeasured confounders.
BibTeX
@article{jiang2025practical,
  author = {Kuan Jiang and Xinxing Lai and Shu Yang and Ying Gao and Xiao‐Hua Zhou},
  title = {{A practical analysis procedure on generalizing comparative effectiveness in the randomized clinical trial to the real-world trial-eligible population}},
  journal = {Journal of Biopharmaceutical Statistics},
  year = {2025},
  doi = {10.1080/10543406.2025.2489282},
}
Figure