Abstract
The American Statistical Association Biopharmaceutical Section (ASA BIOP) scientific working group on real-world evidence (RWE) has been making continuous, extended efforts towards a goal of supporting and advancing regulatory science with respect to clinical studies intended to use real-world data for evidence generation for the purpose of medical product development and evaluation (i.e., RWE studies). In 2023, the working group published a manuscript delineating challenges and opportunities in constructing estimands for RWE studies following the framework in ICH E9(R1) guidance on estimand and sensitivity analysis. As a follow-up task, we describe the other issue, sensitivity analysis. Although the FDA's definition of RWE studies includes randomized trials using RWD as a primary source of evidence generation such as pragmatic trials, here we focus on non-randomized RWE studies and the issue of unmeasured confounding which is a major source of bias for most RWE studies. We review the availability and applicability of sensitivity analysis methods for different types of unmeasured confounding. We also provide some practical and regulatory considerations on using sensitivity analysis for such RWE studies. Then, we use a plasmode case study to demonstrate realistic ways to interpret sensitivity analysis results in supporting regulatory decision-making. We conclude with a brief discussion.
Introduction
In traditional Randomized Controlled Trials (RCTs), randomization ensures that both measured and unmeasured characteristics are balanced between treatment groups. However, the medical field is increasingly turning to Real-World Data (RWD)—like electronic health records and insurance claims—to generate evidence. The "Achilles' heel" of these non-randomized studies is the No Unmeasured Confounding (NUC) assumption. Researchers must assume they have measured every variable that influences both the treatment choice and the outcome. If a hidden factor (e.g., a genetic marker or a lifestyle habit not in the database) affects both, the resulting "treatment effect" might be a statistical illusion. This paper addresses how to quantitatively test the robustness of study results against these hidden biases to satisfy regulatory standards (like those from the FDA).
The Scope of Sensitivity Analysis
Sensitivity analysis in RWE is not a monolithic task but a multi-layered investigation into the reliability of causal inferences. It serves to explore how results might change under different assumptions or data limitations.
As illustrated in Figure 1, sensitivity analysis branches into evaluating underlying modeling assumptions and data limitations. Within the modeling domain, it specifically scrutinizes causal assumptions—including NUC, consistency, and positivity—alongside statistical modeling choices. This framework ensures that the "tipping point" of a study's conclusions is thoroughly understood.
Categorizing Sensitivity Analysis Methods
The authors categorize existing sensitivity analysis methods into three distinct buckets based on their requirements and the nature of the data used:
- Category One (Parametric/Internal): These methods rely on specific mathematical models and distributional assumptions about the hidden confounder. While efficient, they are "brittle"—if assumptions about the hidden variable's distribution are incorrect, the sensitivity analysis itself may be invalid.
- Category Two (Non-parametric/Internal): These are the "robust" choices, such as the E-value and G-value. They don't require a specific model for the hidden confounder. Instead, they calculate the minimum strength of association a hidden variable would need to have with both the treatment and the outcome to "explain away" the observed effect.
- Category Three (External Data): These methods use external datasets (like a small, high-detail validation study) to "calibrate" the results of a larger, less-detailed real-world study.
Case Study: The Plasmode Approach
The paper utilizes a plasmode dataset to demonstrate these methods. A plasmode dataset is a "hybrid" benchmark: it uses real-world covariates (patient characteristics) from an actual clinical trial but simulates the treatment assignment and outcomes. This study used data from the START trial (NCT00409188), involving the cancer vaccine Stimuvax in non-small cell lung cancer. The structure included 1,000 simulated treated subjects and 500 control subjects, using nine real-world confounders (age, sex, smoking status, histology, etc.) to create a realistic, complex baseline for testing.
Key Results and Interpretation
Using the lung cancer plasmode case study, the adjusted Relative Risk (RR) for survival was 1.68 (95% CI: 1.55, 1.82). Sensitivity metrics provided the following insights:
- E-value: The point estimate E-value was 2.75. This means a hidden confounder would need to increase the risk of both treatment and survival by 2.75-fold to nullify the result.
- G-value: The G-value was 0.55 (or 2.89 Adj units). This indicates that the residual hidden bias would need to be nearly three times stronger than the bias already accounted for by the nine measured variables to negate the findings.
Because a 2.75-fold effect is significantly larger than most known clinical risk factors in this space, the researchers could argue the finding is robust. This "benchmarking against the known" is a standout idea: if the most powerful known risk factor (like smoking) only has a 1.5x effect, and the E-value shows you'd need a 3.0x effect to flip the result, the study is likely robust.
Limitations and Future Work
While the mathematical frameworks provide concrete numbers, they still require expert judgment to determine if such powerful hidden variables are "plausible" in the real world. Furthermore, while E-values and G-values are becoming easier to calculate via R packages, many other complex sensitivity methods lack user-friendly, validated software. Finally, most current methods are designed for "point-exposure" (one-time treatment), leaving a significant research gap in sensitivity analysis for Dynamic Treatment Regimes where treatment changes over time.
Summary updated