A signal detection method for temporal variation of adverse effect with vaccine adverse event reporting system data

Yi Cai, Jingcheng Du, Jing Huang, Susan S. Ellenberg, Sean Hennessy, Cui Tao, Yong ChenBMC Medical Informatics Decis. Mak 2017 · DOI

Abstract

Traditional vaccine safety surveillance often aggregates data over long periods, potentially masking safety signals that vary year-to-year. This paper introduces a statistical framework using a random effects model and Likelihood Ratio Test (LRT) to detect temporal heterogeneity in adverse event reporting. Applied to the Vaccine Adverse Event Reporting System (VAERS) data for influenza vaccines, the method successfully identifies significant fluctuations in reporting rates that traditional "lumping" approaches might overlook.

Introduction

Vaccine safety monitoring relies heavily on passive surveillance systems like the Vaccine Adverse Event Reporting System (VAERS). Standard data mining tools typically aggregate reports over many years to identify disproportionately high reporting of specific vaccine-event pairs. However, this approach ignores the temporal dimension, which is critical for vaccines that change annually, such as the seasonal influenza shot. If a specific year's formulation leads to a unique adverse event, the signal may be "diluted" when averaged over decades of data. This research addresses the need for a rigorous method to detect temporal heterogeneity—significant variations in reporting rates across different years.

Methodology

The researchers developed a multi-step process to identify temporal shifts in vaccine safety data.

Flowchart of the proposed test process

The core of the methodology involves a random effects model paired with a Likelihood Ratio Test (LRT). By structuring VAERS data (1990–2013) into a three-dimensional array—vaccine type, adverse event group (MedDRA System Organ Classes), and reporting year—the model tests for a single variance parameter ($\tau^2$). The null hypothesis assumes a constant reporting rate ($\tau^2 = 0$), while the alternative hypothesis suggests significant temporal variation ($\tau^2 > 0$). This approach is more statistically efficient than fixed-effects models, as it reduces the degrees of freedom required for the test.

Results and Visualization

The method was applied to the synthetic trivalent influenza vaccine (FLU3). The LRT successfully identified several System Organ Classes (SOCs) with highly significant temporal variation ($p < 0.0001$), most notably Infections and infestations (SOC1), Nervous system disorders (SOC8), and General disorders (SOC22).

To make these complex results interpretable, the authors introduced a "bubble plot" visualization.

Bubble plot summarizing temporal variation and trends across 26 SOCs

In this plot, bubble size represents statistical significance (p-value), while color indicates the temporal trend (e.g., green for decreasing trends, red for increasing). For instance, SOC1 showed a clear decreasing trend over the 24-year period. This visualization effectively separates stable reporting from heterogeneous reporting, providing a roadmap for clinical review.

Discussion and Future Work

The primary insight of this study is that temporal variation itself is a safety signal. Sudden spikes or long-term trends suggest external drivers, such as changes in vaccine ingredients or viral strains, that warrant deeper investigation.

While promising, the model has limitations. VAERS data is often "sparse," with many zero-report years for specific vaccine-event pairs. Future iterations could incorporate zero-inflated Poisson distributions to improve accuracy. Additionally, accounting for data dependencies—such as surges in reporting following media coverage—using composite likelihood functions remains an area for future research. Finally, moving from broad SOC levels to more specific Preferred Terms (PTs) will allow for more granular medical insights.

Conclusion

By moving beyond aggregate snapshots, this temporal signal detection method provides a more sensitive "smoke detector" for vaccine safety. It is particularly valuable for seasonal vaccines, ensuring that year-specific safety issues are identified and investigated rather than lost in the noise of long-term data.

Summary updated

BibTeX
@article{cai2017signal,
  author = {Yi Cai and Jingcheng Du and Jing Huang and Susan S. Ellenberg and Sean Hennessy and Cui Tao and Yong Chen},
  title = {{A signal detection method for temporal variation of adverse effect with vaccine adverse event reporting system data}},
  journal = {BMC Medical Informatics Decis. Mak.},
  year = {2017},
  doi = {10.1186/s12911-017-0472-y},
}
Figure