Factors Influencing Background Incidence Rate Calculation: Systematic Empirical Evaluation Across an International Network of Observational Databases

Anna Ostropolets, Xintong Li, Rupa Makadia, Gowtham Rao, Peter R. Rijnbeek, Talita Duarte‐Salles, Anthony G. Sena, Azza Shaoibi, Marc A. Suchard, Patrick Ryan, Daniel Prieto‐Alhambra, George HripcsakFrontiers in Pharmacology 2022 · DOI

Abstract

Objective: Background incidence rates are routinely used in safety studies to evaluate an association of an exposure and outcome. Systematic research on sensitivity of rates to the choice of the study parameters is lacking. Materials and Methods: We used 12 data sources to systematically examine the influence of age, race, sex, database, time-at-risk, season and year, prior observation and clean window on incidence rates using 15 adverse events of special interest for COVID-19 vaccines as an example. For binary comparisons we calculated incidence rate ratios and performed random-effect meta-analysis. Results: We observed a wide variation of background rates that goes well beyond age and database effects previously observed. While rates vary up to a factor of 1,000 across age groups, even after adjusting for age and sex, the study showed residual bias due to the other parameters. Rates were highly influenced by the choice of anchoring (e.g., health visit, vaccination, or arbitrary date) for the time-at-risk start. Anchoring on a healthcare encounter yielded higher incidence comparing to a random date, especially for short time-at-risk. Incidence rates were highly influenced by the choice of the database (varying by up to a factor of 100), clean window choice and time-at-risk duration, and less so by secular or seasonal trends. Conclusion: Comparing background to observed rates requires appropriate adjustment and careful time-at-risk start and duration choice. Results should be interpreted in the context of study parameter choices.

Introduction

In vaccine safety monitoring, "observed-to-expected" analyses are a critical tool for identifying potential safety signals. By comparing the number of adverse events observed following vaccination to the "background rate" expected in a similar population, regulators can determine if an association exists. However, calculating these background rates is fraught with methodological choices. Researchers must decide which databases to query, how to define age cohorts, and when to start the "time-at-risk" (TAR) window. This study addresses the significant gap in systematic research regarding how sensitive these background rates are to such design parameters.

Methods

The study utilized the Observational Health Data Sciences and Informatics (OHDSI) network to perform a systematic empirical evaluation across 12 diverse international databases. These sources included administrative claims (e.g., IBM MarketScan CCAE) and electronic health records (e.g., Optum EHR, CUMC) from the US, Australia, Europe, and Japan.

The evaluation focused on 15 Adverse Events of Special Interest (AESI) relevant to COVID-19 vaccines. Researchers systematically varied parameters including demographics (age, sex, race), temporal factors (season, year, TAR duration), and study design elements like the "anchor" date and "clean window" duration.

Figure 1: Conceptual framework for baseline incidence rate calculation, illustrating the interplay between new cases, time-at-risk intervals, and target population parameters.

Key Results and Insights

The findings reveal that background rates are highly sensitive to study design choices, fluctuating by up to 1,000-fold based on seemingly minor parameters.

The Impact of Anchoring and Database Choice

A primary driver of variation was the "anchor" event used to start the TAR. Anchoring on a healthcare visit rather than a random date significantly increased calculated incidence—by nearly 27-fold for short-term windows (pooled IRR 26.8). This occurs because patients at a clinic are under active observation. Furthermore, even with identical definitions, rates varied by a factor of 10 to 100 across different databases, reflecting heterogeneity in how healthcare systems record data.

Demographic Drivers

Age remained the most significant driver of variation, with rates differing by up to 1,000-fold across age groups. While sex-adjustment is standard, the study found that sex-specific IRRs ranged from 0.76 to 2.17, indicating that adjustment alone does not eliminate residual bias from other parameters.

Figure 2: Incidence rates per 100 person-years across different age groups and databases for 15 adverse events of interest, highlighting the significant influence of age and database heterogeneity.

Discussion and Conclusion

The results underscore that "background rates" are not universal constants but are deeply influenced by methodological choices. The massive "anchor effect" suggests that safety studies must carefully match the background rate's anchor to the exposure event (e.g., a vaccination visit) to ensure a fair comparison.

Limitations include the inherent variability in observational data quality and the limited availability of race data outside the US. Additionally, the "healthy vaccinee" bias remains a factor that background rates cannot fully account for. Ultimately, comparing background to observed rates requires appropriate adjustment and careful time-at-risk start and duration choice. Results should always be interpreted in the context of these study parameter choices.

Summary updated

Abstract

<b>Objective:</b> Background incidence rates are routinely used in safety studies to evaluate an association of an exposure and outcome. Systematic research on sensitivity of rates to the choice of the study parameters is lacking. <b>Materials and Methods:</b> We used 12 data sources to systematically examine the influence of age, race, sex, database, time-at-risk, season and year, prior observation and clean window on incidence rates using 15 adverse events of special interest for COVID-19 vaccines as an example. For binary comparisons we calculated incidence rate ratios and performed random-effect meta-analysis. <b>Results:</b> We observed a wide variation of background rates that goes well beyond age and database effects previously observed. While rates vary up to a factor of 1,000 across age groups, even after adjusting for age and sex, the study showed residual bias due to the other parameters. Rates were highly influenced by the choice of anchoring (e.g., health visit, vaccination, or arbitrary date) for the time-at-risk start<i>.</i> Anchoring on a healthcare encounter yielded higher incidence comparing to a random date, especially for short time-at-risk. Incidence rates were highly influenced by the choice of the database (varying by up to a factor of 100), clean window choice and time-at-risk duration, and less so by secular or seasonal trends. <b>Conclusion:</b> Comparing background to observed rates requires appropriate adjustment and careful time-at-risk start and duration choice. Results should be interpreted in the context of study parameter choices.
BibTeX
@inproceedings{ostropolets2022factors,
  author = {Anna Ostropolets and Xintong Li and Rupa Makadia and Gowtham Rao and Peter R. Rijnbeek and Talita Duarte‐Salles and Anthony G. Sena and Azza Shaoibi and Marc A. Suchard and Patrick Ryan and Daniel Prieto‐Alhambra and George Hripcsak},
  title = {{Factors Influencing Background Incidence Rate Calculation: Systematic Empirical Evaluation Across an International Network of Observational Databases}},
  booktitle = {Frontiers in Pharmacology},
  year = {2022},
  doi = {10.3389/fphar.2022.814198},
}
Figure