Abstract
The assumption of no unmeasured confounders is a critical but unverifiable assumption required for causal inference, yet quantitative sensitivity analyses to assess the robustness of real-world evidence remain underutilized. The lack of use is likely due in part to the complexity of implementation and often specific and restrictive data requirements for the application of each method. With the advent of sensitivity analysis methods that are broadly applicable—in that they do not require identification of a specific unmeasured confounder—along with publicly available code for implementation, roadblocks toward broader use are decreasing. To spur greater application, here we present a best practice guidance to address the potential for unmeasured confounding at both the design and analysis stages, including a set of framing questions and an analytic toolbox for researchers. The questions at the design stage guide the research through steps evaluating the potential robustness of the design while encouraging the gathering of additional data to reduce uncertainty. At the analysis stage, the questions guide researchers in quantifying the robustness of the observed result, providing a clearer indication of the reliability of their conclusions. We demonstrate the application of the guidance using simulated data based on a real-world fibromyalgia study, applying multiple methods from our analytic toolbox for illustration purposes.
Introduction
In Real-World Evidence (RWE) research, observational data is frequently used to compare treatment outcomes. A fundamental requirement for valid causal inference in these studies is the "no unmeasured confounders" assumption—the premise that all variables influencing both treatment assignment and the outcome have been accounted for. However, this assumption is inherently unverifiable. If a significant confounder remains unmeasured, the observed treatment effect may be biased or entirely spurious.
Despite the availability of Quantitative Sensitivity Analysis (QSA) to test the robustness of findings against such bias, these methods are rarely applied in practice. This is often attributed to perceived mathematical complexity and restrictive data requirements. This paper introduces a practical, two-stage roadmap designed to integrate QSA into the standard research workflow, moving beyond qualitative acknowledgments of "unmeasured confounding" toward rigorous quantification.
A Two-Stage Framework for Robustness
The proposed guidance addresses unmeasured confounding at two critical junctures: study design and final analysis.
Stage 1: Study Design and Benchmarking
Before data collection or analysis, researchers should explicitly map out the causal structure of the problem. Directed Acyclic Graphs (DAGs) are essential tools for identifying known confounders and hypothesizing potential unmeasured ones. By using "benchmarking," researchers can compare the potential strength of a hypothetical unmeasured confounder to that of the strongest measured variables. This allows for an early assessment of whether a study design is likely to yield robust results.
Stage 2: Analysis and the QSA Toolbox
Once the primary analysis is complete, the framework guides researchers through a series of questions to quantify robustness. The "Toolbox" provides a suite of methods categorized by their specific utility, such as identifying "tipping points" or performing adjusted analyses.
Key tools highlighted in the toolbox include:
- E-values: The minimum strength an unmeasured confounder would need (on a risk ratio scale) to explain away an observed effect.
- Robustness Values: Measures based on the proportion of variance explained ($R^2$).
- Corrective Modeling: Techniques like Multiple Imputation or Bayesian Modeling that can adjust estimates if partial information on the unmeasured confounder is available from external sources or registries.
Case Study: Fibromyalgia Simulation
To demonstrate the framework, the authors used a simulated dataset (N=1000) modeled after the REFLECTIONS fibromyalgia study. In this scenario, a "true" treatment effect of zero was masked by the intentional omission of a key confounder ('U'), resulting in a false "significant" finding in standard analyses (estimates ranging from -0.20 to -0.47).
Applying the QSA framework revealed the fragility of this result:
- Sensitivity Analysis: The E-value for the lower confidence limit was only 1.2, indicating that even a very weak unmeasured confounder could nullify the finding.
- Benchmarking: When compared to the known effect of baseline pain (which has a Risk Ratio of 3.1), it became clear that the study's findings were not robust.
- Correction: When the authors used Bayesian modeling or Multiple Imputation to incorporate partial information about the "hidden" variable, the treatment effect estimate moved from a biased -0.30 back toward the true null (-0.14 to -0.17), correctly showing the treatment was likely ineffective.
Conclusion
Quantitative sensitivity analysis should be a standard component of RWE research. By adopting a structured approach—utilizing DAGs for design and a standardized toolbox for analysis—researchers can provide a clearer indication of the robustness of their conclusions. While this framework currently focuses on unmeasured confounding, future work should extend these best practices to address other common biases, such as selection bias and measurement error, further strengthening the reliability of real-world evidence.
Summary updated