Objective study validity diagnostics: a framework requiring pre-specified, empirical verification to increase trust in the reliability of real-world evidence

Mitchell Conover, Patrick B. Ryan, Yong Chen, Marc A. Suchard, George Hripcsak, Martijn J. SchuemieJ. Am. Medical Informatics Assoc 2025 · DOI

TL;DR

This paper introduces a framework of "objective study validity diagnostics" to rigorously assess the reliability of findings from observational health studies. By pre-specifying and empirically verifying these diagnostics before unblinding results, the authors aim to increase trust in real-world evidence (RWE) and reduce bias.

What Problem Does This Paper Address?

Real-world data (RWD) holds immense potential for understanding the effects of medical treatments, but observational studies using this data are often criticized for their potential for bias. Issues like confounding, selection bias, and p-hacking can lead to unreliable findings and a lack of trust in RWE. While methods exist to address these biases, their application and interpretation can be subjective, leading to a situation where studies with questionable validity might still be published if their results are compelling. This paper tackles the challenge of establishing a more objective and standardized way to evaluate the trustworthiness of RWE generated from observational studies.

Methods

The core of this paper is a proposed framework of "objective study validity diagnostics." These are pre-specified, empirical measures designed to assess the appropriateness of study designs, analytic assumptions, and potential threats to validity. The framework suggests two approaches: either interpret diagnostic results before unblinding study findings, or only unblind results from analyses that pass pre-specified diagnostic thresholds.

The paper details several key diagnostics:

  • Minimum Detectable Relative Risk (MDRR): Assesses if a study has sufficient statistical power to detect meaningful effect sizes, preventing misinterpretation of results from underpowered studies.
  • Empirical Equipoise: Measures the overlap in propensity or preference score distributions to evaluate the "positivity" assumption (i.e., sufficient variation in exposure across study groups). This helps ensure there's genuine uncertainty about treatment choices for comparable patients.
  • Covariate Balance (Maximum SDM): Quantifies the imbalance in baseline characteristics between study groups, serving as an indicator of potential confounding bias.
  • Generalizability (Maximum SDM): Compares the characteristics of the analytic cohort to the target population to assess how well the study findings can be generalized.
  • Expected Absolute Systematic Error (EASE): Uses "negative control outcomes" (exposure-outcome pairs where a null effect is expected) to quantify the overall systematic error in a study design.

The authors applied these diagnostics to a large-scale comparative new-user study of antihypertensive medications using negative control outcomes to demonstrate their impact.

Datasets

The study utilized data from the LEGEND for Hypertension (LEGEND-HTN) cohort study. This study involved a large-scale comparative new-user design and was conducted across a global network of nine databases: six administrative claims databases (Merative MarketScan Commercial Claims and Encounters, Optum ClinFormatics, Merative MarketScan Medicare Supplemental Beneficiaries, Merative MarketScan Multi-state Medicaid, Japan Medical Data Center, Korea National Health Insurance Service/National Sample Cohort) and three electronic health record databases (Optum Pan-Therapeutic, IMS/IQVIA Disease Analyzer Germany, Columbia University Medical Center). The paper also generated a set of negative control outcomes specifically for this evaluation, distinct from those used in the original LEGEND-HTN study.

Key Insights and Ideas

  • Pre-specification is Key: The emphasis on pre-specifying diagnostic thresholds before seeing study results is crucial for preventing p-hacking and subjective interpretation.
  • Blinding Invalid Findings: The framework allows for "blinding" unreliable findings by only proceeding with analyses that pass the objective diagnostics, thereby preventing the misinterpretation of potentially spurious results.
  • Complementary Nature of Diagnostics: While individual diagnostics can reduce bias, applying a suite of diagnostics together provides a more comprehensive and robust evaluation of study validity.
  • Empirical Equipoise is Highly Impactful: The study found that the equipoise diagnostic, particularly requiring >50% of patients to be in empirical equipoise, was the most restrictive but also the most impactful in reducing systematic error.
  • Standardization Facilitates Deployment: The use of standardized data models (OMOP CDM) and open-source software (HADES) makes the implementation of these objective diagnostics more feasible and reproducible.

Key Results

  • Applying objective diagnostics significantly reduces bias and improves the reliability of RWE.
  • Among 11,716 negative control analyses, 13.9% met pre-specified diagnostic thresholds.
  • The application of these diagnostics reduced the Expected Absolute Systematic Error (EASE) from 0.38 to 0.
  • When all objective diagnostics were applied together, the EASE was reduced to zero, and only 3.9% of negative control estimates had confidence intervals excluding the null, indicating a substantial improvement in reliability.
  • Restricting analyses based on covariate balance (SDM < 0.10) reduced the EASE from 0.38 to 0.28 and improved confidence interval coverage.

Limitations and Future Work

  • Negative Control Limitations: The use of negative control outcomes inherently limits the ability to quantify false negatives, as the true effect is known to be null. However, the authors argue that the framework would clearly reduce false negatives by classifying studies as "inestimable."
  • Generalizability SDM: The evaluation of the generalizability SDM diagnostic might be skewed because, in the context of negative control outcomes, effect modification between target and analytic cohorts is not expected.
  • Limited Diagnostic Thresholds Explored: The study explored a limited set of diagnostic thresholds. Further research is needed to determine optimal thresholds for various scenarios.
  • Interaction Between Diagnostics: While some interaction was observed, a comprehensive understanding of how different diagnostics interact and their relative value requires further investigation.
  • Future Work: The authors emphasize the need for further work in developing and evaluating more diagnostics, exploring different thresholds, and understanding the role of these diagnostics in meta-analyses. They also highlight the importance of standardizing programs and evidence generation to facilitate a common set of easily deployable and understandable diagnostic evaluations.

Summary updated

BibTeX
@article{conover2025objective,
  author = {Mitchell Conover and Patrick B. Ryan and Yong Chen and Marc A. Suchard and George Hripcsak and Martijn J. Schuemie},
  title = {{Objective study validity diagnostics: a framework requiring pre-specified, empirical verification to increase trust in the reliability of real-world evidence}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2025},
  doi = {10.1093/jamia/ocae317},
}
Figure