TL;DR
This paper introduces the EUMAEUS study, designed to rigorously evaluate and compare different epidemiological methods for vaccine safety surveillance using real-world, multi-database networks. The goal is to determine which methods are most precise and unbiased in detecting adverse events, bridging the gap between simulated study findings and real-world data applicability.
What Problem Does This Paper Address?
The rapid development and deployment of vaccines, particularly for COVID-19, necessitate robust post-marketing surveillance to detect rare adverse events that may not be apparent in clinical trials. While various epidemiological study designs (e.g., cohort, case-control, self-controlled) exist for this purpose, there's a lack of comprehensive understanding regarding their comparative performance, precision, and bias in real-world data settings. Existing comparisons are often based on simulations, which may not fully capture the complexities of real-world data, such as variations in data quality, healthcare systems, and population characteristics. This paper aims to address this gap by proposing the EUMAEUS study, which will evaluate these methods using large-scale, real-world observational data.
Methods
The EUMAEUS study will employ a multi-database network approach, leveraging existing observational health data (claims and electronic health records) from the US. The core of the methodology involves evaluating four main study designs:
- Cohort studies: Using either contemporary non-users or historical rates as comparators.
- Case-control studies: Comparing cases with event-free individuals.
- Self-controlled designs: Specifically, Self-Controlled Case Series (SCCS) and Self-Controlled Risk Intervals (SCRI), which use individuals as their own controls.
The study will focus on evaluating performance metrics such as:
- Counterfactual construction: Estimating expected event rates.
- Time-at-Risk (TAR): Defining periods when adverse events might be attributed to vaccination.
- Test statistics: To estimate differences between observed and expected counts (e.g., incidence rate ratio, hazard ratio, odds ratio, log likelihood ratio).
- Decision rules: To classify true safety signals from non-signals, using criteria like Type I and Type II error rates, sensitivity, and specificity.
The study will also assess timeliness in signal detection, reporting performance metrics monthly.
Datasets
The EUMAEUS study will utilize a combination of administrative claims and electronic health record (EHR) databases from the United States. The paper mentions that details of these databases can be found in their online protocol. The study will focus on six groups of vaccines:
- A (H1N1)pdm09
- Seasonal flu (Fluvirin)
- Seasonal flu (Fluzone)
- Seasonal flu (All)
- Zoster (Shingrix)
- Human papillomavirus (Gardasil 9)
For outcomes, they will study both "negative control outcomes" (believed to be unrelated to vaccines) and "imputed positive controls" (simulated to be caused by vaccines) to evaluate method performance.
Key Insights and Ideas
- Bridging the simulation-real-world gap: A major insight is the recognition that simulation studies, while valuable, don't fully replicate the complexities and biases inherent in real-world observational data. EUMAEUS aims to bridge this by evaluating methods on actual data.
- The importance of multi-database networks: The use of a multi-country, multi-database network (like OHDSI) is highlighted as crucial for robust evaluation, allowing for the examination of data heterogeneity, generalizability of findings, and pooling of results.
- Self-controlled designs offer advantages: SCCS and SCRI designs are presented as promising alternatives that inherently control for time-invariant confounders and can minimize selection bias by using individuals as their own controls.
- Bias is pervasive: The paper emphasizes that all study designs are susceptible to various biases (selection, misclassification, confounding), and understanding these is critical for accurate interpretation of results.
Key Results
Not discussed in the paper. The paper focuses on the rationale and design of the EUMAEUS study, which is ongoing. It reviews existing literature on study designs and simulation studies but does not present results from the EUMAEUS study itself.
Limitations and Future Work
Limitations: * Generalizability: The reliance on US-based databases for EUMAEUS may limit the generalizability of findings to other countries with different healthcare systems, policies, and vaccine utilization patterns. * Data quality and availability: Real-world data can be subject to variations in quality, completeness, and accessibility, which can pose challenges for any study design. * Bias in real-world data: The paper acknowledges that "observational real-world" data is prone to confounding and misclassifications, which need to be identified and addressed.
Future Work: * The EUMAEUS study itself represents the future work, aiming to provide empirical evidence on the performance of different vaccine safety surveillance methods in real-world settings. * The authors hope that the EUMAEUS project will help address remaining questions about how, what, and when to reliably identify vaccine safety signals in real-world settings. * The paper suggests that future research should incorporate more complex time-varying factors and adjust for various biases and confounders to provide more robust results.
Summary updated