Abstract
Background incidence rates are routinely used in safety studies to evaluate the association of an exposure and an outcome. Systematic research on the sensitivity of background rates to the choice of the study parameters is lacking. We used 12 electronic health record and administrative claims data sources to calculate incidence rates of 15 adverse events. We examined the influence of age, race, sex, database, time-at-risk start (anchoring) event and duration, season and year, prior observation and clean window. For binary comparisons, we calculated incidence rate ratios and performed random-effect model meta-analysis. Background rates were highly sensitive to demographic characteristics of the population, especially age, with rates varying up to a factor of 1,000 across age groups. Rates varied by up to a factor of 100 by database. Incidence rates were highly influenced by the choice of anchoring (e.g., health visit, vaccination, or arbitrary date) for the time-at-risk start, especially at short times at risk, and less influenced by secular or seasonal trends. Therefore, comparing background to observed rates requires appropriate adjustment, and results should be interpreted in the context of design choices.
Introduction
In pharmacovigilance, researchers monitor drug and vaccine safety by comparing observed medical events to expected background incidence rates. This "Observed vs. Expected" (O/E) analysis is a cornerstone of safety signaling. However, there is no universal standard for calculating these background rates. Researchers must make numerous design choices regarding database selection, demographic stratification, and the definition of the "at-risk" period.
This study addresses a critical gap in the field: quantifying how sensitive background rates are to these specific design choices. Without understanding this sensitivity, safety monitoring risks producing false alarms or missing real signals due to poorly calibrated expectations.
Methods
The authors performed an empirical sensitivity analysis across an international observational data network, calculating incidence rates for 15 "Adverse Events of Special Interest" (AESIs) relevant to COVID-19 vaccine monitoring, such as Bell’s palsy, myocarditis, and appendicitis.
As illustrated in Figure 1, the study systematically varied several parameters to measure their impact on the final rate:
- Target Population: Stratification by age, sex, and race.
- Time-at-Risk (TAR) Interval: Varying the "anchoring" event (random date, healthcare visit, or vaccination) and the duration (1 to 365 days).
- New Case Definition: Implementing phenotype algorithms and "clean windows" to ensure only incident cases were counted.
The analysis utilized 12 diverse data sources, including administrative claims (e.g., IBM MarketScan) and Electronic Health Records (EHR) from the US, UK, Germany, Spain, Japan, and other regions, representing millions of patients.
Key Results and Insights
The study revealed that background incidence rates are not fixed constants but are highly fluid depending on the study's parameters.
The Dominance of Age and Database Selection
Age emerged as the most significant factor, with incidence rates for the same condition varying by up to 1,000-fold across different age groups. Furthermore, even when using identical definitions, rates varied by up to 100-fold between different databases. This suggests that a single "global" background rate is a myth; surveillance should ideally use background rates derived from the same database as the observed data.
Figure 2 visualizes this massive heterogeneity. The spread of data points for conditions like anaphylaxis or acute myocardial infarction across different databases (indicated by color) and sexes (indicated by shape) highlights the risk of using unadjusted or non-specific background rates.
The "Anchoring" Effect
A major finding was the impact of the "anchoring" event. When the observation period began on the day of a healthcare visit, the recorded incidence in the following 24 hours was up to 100 times higher than when starting on a random calendar date. This is likely because patients at a doctor's office are more likely to be diagnosed or are already symptomatic. This effect was most pronounced in short windows (0–1 day) and attenuated over longer periods.
Discussion and Limitations
The findings suggest that safety protocols must strictly standardize anchoring events and demographic adjustments. While seasonality and secular trends had some impact, they were secondary to age and database choice.
Limitations: As an observational study, it is subject to the limitations of EHR and claims data, such as miscoding. Race data was only available for a subset of US databases, and the study focuses on measuring sensitivity rather than explaining the underlying causal reasons for database-specific variations. Future work should focus on providing "prediction intervals" in meta-analyses to better account for this inherent variability across data networks.
Summary updated