Current Approaches to Vaccine Safety Using Observational Data: A Rationale for the EUMAEUS (Evaluating Use of Methods for Adverse Events Under Surveillance-for Vaccines) Study Design

Lana Yin Hui Lai, Faaizah Arshad, Carlos Areia, Thamir M. Alshammari, Heba Alghoul, Paula Casajust, Xintong Li, Dalia Dawoud, Fredrik Nyberg, Nicole Pratt, George Hripcsak, Marc A. Suchard, Daniel Prieto‐Alhambra, Patrick Ryan, Martijn J. SchuemieFrontiers in Pharmacology 2022 · DOI

TL;DR

This paper introduces the EUMAEUS study, designed to rigorously evaluate and compare different epidemiological methods for vaccine safety surveillance using real-world, multi-database networks. The goal is to determine which methods are most precise and unbiased in detecting adverse events, bridging the gap between simulated study findings and real-world data applicability.

What Problem Does This Paper Address?

The rapid development and deployment of vaccines, particularly for COVID-19, necessitate robust post-marketing surveillance to detect rare adverse events that may not be apparent in clinical trials. While various epidemiological study designs (e.g., cohort, case-control, self-controlled) exist for this purpose, there's a lack of comprehensive understanding regarding their comparative performance, precision, and bias in real-world data settings. Existing comparisons are often based on simulations, which may not fully capture the complexities of real-world data, such as variations in data quality, healthcare systems, and population characteristics. This paper aims to address this gap by proposing the EUMAEUS study, which will evaluate these methods using large-scale, real-world observational data.

Methods

The EUMAEUS study will employ a multi-database network approach, leveraging existing observational health data (claims and electronic health records) from the US. The core of the methodology involves evaluating four main study designs:

  • Cohort studies: Using either contemporary non-users or historical rates as comparators.
  • Case-control studies: Comparing cases with event-free individuals.
  • Self-controlled designs: Specifically, Self-Controlled Case Series (SCCS) and Self-Controlled Risk Intervals (SCRI), which use individuals as their own controls.

The study will focus on evaluating performance metrics such as:

  • Counterfactual construction: Estimating expected event rates.
  • Time-at-Risk (TAR): Defining periods when adverse events might be attributed to vaccination.
  • Test statistics: To estimate differences between observed and expected counts (e.g., incidence rate ratio, hazard ratio, odds ratio, log likelihood ratio).
  • Decision rules: To classify true safety signals from non-signals, using criteria like Type I and Type II error rates, sensitivity, and specificity.

The study will also assess timeliness in signal detection, reporting performance metrics monthly.

Datasets

The EUMAEUS study will utilize a combination of administrative claims and electronic health record (EHR) databases from the United States. The paper mentions that details of these databases can be found in their online protocol. The study will focus on six groups of vaccines:

  • A (H1N1)pdm09
  • Seasonal flu (Fluvirin)
  • Seasonal flu (Fluzone)
  • Seasonal flu (All)
  • Zoster (Shingrix)
  • Human papillomavirus (Gardasil 9)

For outcomes, they will study both "negative control outcomes" (believed to be unrelated to vaccines) and "imputed positive controls" (simulated to be caused by vaccines) to evaluate method performance.

Key Insights and Ideas

  • Bridging the simulation-real-world gap: A major insight is the recognition that simulation studies, while valuable, don't fully replicate the complexities and biases inherent in real-world observational data. EUMAEUS aims to bridge this by evaluating methods on actual data.
  • The importance of multi-database networks: The use of a multi-country, multi-database network (like OHDSI) is highlighted as crucial for robust evaluation, allowing for the examination of data heterogeneity, generalizability of findings, and pooling of results.
  • Self-controlled designs offer advantages: SCCS and SCRI designs are presented as promising alternatives that inherently control for time-invariant confounders and can minimize selection bias by using individuals as their own controls.
  • Bias is pervasive: The paper emphasizes that all study designs are susceptible to various biases (selection, misclassification, confounding), and understanding these is critical for accurate interpretation of results.

Key Results

Not discussed in the paper. The paper focuses on the rationale and design of the EUMAEUS study, which is ongoing. It reviews existing literature on study designs and simulation studies but does not present results from the EUMAEUS study itself.

Limitations and Future Work

Limitations: * Generalizability: The reliance on US-based databases for EUMAEUS may limit the generalizability of findings to other countries with different healthcare systems, policies, and vaccine utilization patterns. * Data quality and availability: Real-world data can be subject to variations in quality, completeness, and accessibility, which can pose challenges for any study design. * Bias in real-world data: The paper acknowledges that "observational real-world" data is prone to confounding and misclassifications, which need to be identified and addressed.

Future Work: * The EUMAEUS study itself represents the future work, aiming to provide empirical evidence on the performance of different vaccine safety surveillance methods in real-world settings. * The authors hope that the EUMAEUS project will help address remaining questions about how, what, and when to reliably identify vaccine safety signals in real-world settings. * The paper suggests that future research should incorporate more complex time-varying factors and adjust for various biases and confounders to provide more robust results.

Summary updated

Abstract

Post-marketing vaccine safety surveillance aims to detect adverse events following immunization in a population. Whether certain methods of surveillance are more precise and unbiased in generating safety signals is unclear. Here, we synthesized information from existing literature to provide an overview of the strengths, weaknesses, and clinical applications of epidemiologic and analytical methods used in vaccine monitoring, focusing on cohort, case-control and self-controlled designs. These designs are proposed to be evaluated in the EUMAEUS (Evaluating Use of Methods for Adverse Event Under Surveillance-for vaccines) study because of their widespread use and potential utility. Over the past decades, there have been an increasing number of epidemiological study designs used for vaccine safety surveillance. While traditional cohort and case-control study designs remain widely used, newer, novel designs such as the self-controlled case series and self-controlled risk intervals have been developed. Each study design comes with its strengths and limitations, and the most appropriate study design will depend on availability of resources, access to records, number and distribution of cases, and availability of population coverage data. Several assumptions have to be made while using the various study designs, and while the goal is to mitigate any biases, violations of these assumptions are often still present to varying degrees. In our review, we discussed some of the potential biases (i.e., selection bias, misclassification bias and confounding bias), and ways to mitigate them. While the types of epidemiological study designs are well established, a comprehensive comparison of the analytical aspects (including method evaluation and performance metrics) of these study designs are relatively less well studied. We summarized the literature, reporting on two simulation studies, which compared the detection time, empirical power, error rate and risk estimate bias across the above-mentioned study designs. While these simulation studies provided insights on the analytic performance of each of the study designs, its applicability to real-world data remains unclear. To bridge that gap, we provided the rationale of the EUMAEUS study, with a brief description of the study design; and how the use of real-world multi-database networks can provide insights into better methods evaluation and vaccine safety surveillance.
BibTeX
@inproceedings{lai2022current,
  author = {Lana Yin Hui Lai and Faaizah Arshad and Carlos Areia and Thamir M. Alshammari and Heba Alghoul and Paula Casajust and Xintong Li and Dalia Dawoud and Fredrik Nyberg and Nicole Pratt and George Hripcsak and Marc A. Suchard and Daniel Prieto‐Alhambra and Patrick Ryan and Martijn J. Schuemie},
  title = {{Current Approaches to Vaccine Safety Using Observational Data: A Rationale for the EUMAEUS (Evaluating Use of Methods for Adverse Events Under Surveillance-for Vaccines) Study Design}},
  booktitle = {Frontiers in Pharmacology},
  year = {2022},
  doi = {10.3389/fphar.2022.837632},
}
Figure