Empirical evaluation of the sensitivity of background incidence rate characterization for adverse events across an international observational data network

Anna Ostropolets, Xintong Li, Rupa Makadia, Gowtham Rao, Peter R. Rijnbeek, Talita Duarte‐Salles, Anthony G. Sena, Azza Shaoibi, Marc A. Suchard, Patrick Ryan, Daniel Prieto‐Alhambra, George HripcsakmedRxiv 2021 · DOI

Abstract

Background incidence rates are routinely used in safety studies to evaluate the association of an exposure and an outcome. Systematic research on the sensitivity of background rates to the choice of the study parameters is lacking. We used 12 electronic health record and administrative claims data sources to calculate incidence rates of 15 adverse events. We examined the influence of age, race, sex, database, time-at-risk start (anchoring) event and duration, season and year, prior observation and clean window. For binary comparisons, we calculated incidence rate ratios and performed random-effect model meta-analysis. Background rates were highly sensitive to demographic characteristics of the population, especially age, with rates varying up to a factor of 1,000 across age groups. Rates varied by up to a factor of 100 by database. Incidence rates were highly influenced by the choice of anchoring (e.g., health visit, vaccination, or arbitrary date) for the time-at-risk start, especially at short times at risk, and less influenced by secular or seasonal trends. Therefore, comparing background to observed rates requires appropriate adjustment, and results should be interpreted in the context of design choices.

Introduction

In pharmacovigilance, researchers monitor drug and vaccine safety by comparing observed medical events to expected background incidence rates. This "Observed vs. Expected" (O/E) analysis is a cornerstone of safety signaling. However, there is no universal standard for calculating these background rates. Researchers must make numerous design choices regarding database selection, demographic stratification, and the definition of the "at-risk" period.

This study addresses a critical gap in the field: quantifying how sensitive background rates are to these specific design choices. Without understanding this sensitivity, safety monitoring risks producing false alarms or missing real signals due to poorly calibrated expectations.

Methods

The authors performed an empirical sensitivity analysis across an international observational data network, calculating incidence rates for 15 "Adverse Events of Special Interest" (AESIs) relevant to COVID-19 vaccine monitoring, such as Bell’s palsy, myocarditis, and appendicitis.

The baseline incidence rate is calculated by dividing new cases by the person-time of the at-risk population, with each component subject to various design choices.

As illustrated in Figure 1, the study systematically varied several parameters to measure their impact on the final rate:

  • Target Population: Stratification by age, sex, and race.
  • Time-at-Risk (TAR) Interval: Varying the "anchoring" event (random date, healthcare visit, or vaccination) and the duration (1 to 365 days).
  • New Case Definition: Implementing phenotype algorithms and "clean windows" to ensure only incident cases were counted.

The analysis utilized 12 diverse data sources, including administrative claims (e.g., IBM MarketScan) and Electronic Health Records (EHR) from the US, UK, Germany, Spain, Japan, and other regions, representing millions of patients.

Key Results and Insights

The study revealed that background incidence rates are not fixed constants but are highly fluid depending on the study's parameters.

The Dominance of Age and Database Selection

Age emerged as the most significant factor, with incidence rates for the same condition varying by up to 1,000-fold across different age groups. Furthermore, even when using identical definitions, rates varied by up to 100-fold between different databases. This suggests that a single "global" background rate is a myth; surveillance should ideally use background rates derived from the same database as the observed data.

Incidence rates for 15 adverse events across 12 databases, demonstrating significant heterogeneity by data source and sex.

Figure 2 visualizes this massive heterogeneity. The spread of data points for conditions like anaphylaxis or acute myocardial infarction across different databases (indicated by color) and sexes (indicated by shape) highlights the risk of using unadjusted or non-specific background rates.

The "Anchoring" Effect

A major finding was the impact of the "anchoring" event. When the observation period began on the day of a healthcare visit, the recorded incidence in the following 24 hours was up to 100 times higher than when starting on a random calendar date. This is likely because patients at a doctor's office are more likely to be diagnosed or are already symptomatic. This effect was most pronounced in short windows (0–1 day) and attenuated over longer periods.

Discussion and Limitations

The findings suggest that safety protocols must strictly standardize anchoring events and demographic adjustments. While seasonality and secular trends had some impact, they were secondary to age and database choice.

Limitations: As an observational study, it is subject to the limitations of EHR and claims data, such as miscoding. Race data was only available for a subset of US databases, and the study focuses on measuring sensitivity rather than explaining the underlying causal reasons for database-specific variations. Future work should focus on providing "prediction intervals" in meta-analyses to better account for this inherent variability across data networks.

Summary updated

Abstract

ABSTRACT Background incidence rates are routinely used in safety studies to evaluate the association of an exposure and an outcome. Systematic research on the sensitivity of background rates to the choice of the study parameters is lacking. We used 12 electronic health record and administrative claims data sources to calculate incidence rates of 15 adverse events. We examined the influence of age, race, sex, database, time-at-risk start (anchoring) event and duration, season and year, prior observation and clean window. For binary comparisons, we calculated incidence rate ratios and performed random-effect model meta-analysis. Background rates were highly sensitive to demographic characteristics of the population, especially age, with rates varying up to a factor of 1,000 across age groups. Rates varied by up to a factor of 100 by database. Incidence rates were highly influenced by the choice of anchoring (e.g., health visit, vaccination, or arbitrary date) for the time-at-risk start, especially at short times at risk, and less influenced by secular or seasonal trends. Therefore, comparing background to observed rates requires appropriate adjustment, and results should be interpreted in the context of design choices. Short Figure
BibTeX
@misc{ostropolets2021empirical,
  author = {Anna Ostropolets and Xintong Li and Rupa Makadia and Gowtham Rao and Peter R. Rijnbeek and Talita Duarte‐Salles and Anthony G. Sena and Azza Shaoibi and Marc A. Suchard and Patrick Ryan and Daniel Prieto‐Alhambra and George Hripcsak},
  title = {{Empirical evaluation of the sensitivity of background incidence rate characterization for adverse events across an international observational data network}},
  year = {2021},
  doi = {10.1101/2021.06.27.21258701},
  howpublished = {medRxiv},
}
Figure