A Discovery and Verification Approach for Pharmacovigilance using Electronic Health Care Data

Lj Dijkstra, Tania Schink, Roland Linder, Markus Schwaninger, Iris Pigeot, Marvin N. Wright, Ronja ForaitamedRxiv 2022 · DOI

TL;DR

Researchers have developed a robust three-step framework for "signal detection"—identifying potential side effects of drugs—by combining multiple statistical methods through a consensus-based voting system (Borda count). By aggregating diverse algorithms and following up with rigorous pharmacoepidemiological verification, the team successfully identified known risks of the anticoagulant rivaroxaban while filtering out "spurious" signals that single-method approaches might have misidentified.

What Problem Does This Paper Address?

The gold standard for drug safety, Randomized Clinical Trials (RCTs), often fails to catch rare or long-term adverse drug reactions (ADRs) because their sample sizes are too small and their participants are too "idealized" (excluding the elderly or those with multiple conditions). Once a drug hits the market, we rely on pharmacovigilance.

Traditionally, this meant "spontaneous reporting"—doctors or patients voluntarily reporting a suspected side effect. However, these systems suffer from massive under-reporting. The field is now shifting toward Electronic Health Care (EHC) data (insurance claims, medical records). The challenge? Most researchers use only a single statistical method to find "signals" in this data. Since different algorithms have different biases, relying on just one can lead to false positives or missed risks.

Methods

The authors propose a novel "Discovery and Verification" pipeline:

  1. Ensemble Signal Detection: Instead of one algorithm, they use four:
  2. Univariate: Longitudinal Gamma Poisson Shrinker (LGPS) and Bayesian Confidence Propagation Neural Network (BCPNN). These look at simple drug-event pairs.
  3. Multivariate: LASSO (sparse regression) and Random Forests (machine learning). These control for multiple variables simultaneously to find complex associations.
  4. Borda Count Aggregation: They treat each algorithm as a "voter." The Borda count ranks drug-ADR pairs based on their average performance across all methods. This identifies a "consensus" rather than just a majority, highlighting signals that are consistently strong across different statistical lenses.
  5. Expert Triage: A committee reviews the top-ranked signals using a custom web tool to assess biological plausibility.
  6. Verification Study: The most interesting signals are subjected to a formal "nested active comparator case-control study." This is a rigorous epidemiological deep-dive designed to minimize bias and confounding.

Datasets

The study utilized the German Pharmacoepidemiological Research Database (GePaRD).

  • Scale: Contains claims data for approximately 25 million people (about 30% of the German population).
  • Case Study: They focused on Rivaroxaban (RVX), a common blood thinner, comparing it against an older comparator, Phenprocoumon (PPC), using data from 2011 to 2017.

Key Insights and Ideas

  • Algorithms Disagree: One of the most striking findings (shown in Figure 2) is that these popular algorithms often don't agree. For example, three methods ranked acute cystitis in the top 2%, while LASSO didn't even put it in the top 10%. This justifies the need for an ensemble approach.
  • Consensus Over Majority: The Borda count is powerful because it doesn't require the raw scores to be on the same scale (e.g., comparing a p-value to a machine learning importance score). It only cares about the rank.
  • The "Innocent Bystander" Problem: Some drugs appear to cause an event simply because they are prescribed to sick people. The verification step is crucial here; it revealed that while "acute liver injury" (ALI) appeared as a signal in the detection phase, the rigorous verification study showed RVX users actually had a lower risk than the comparator group, suggesting the initial signal was likely due to underlying patient characteristics rather than the drug itself.

Key Results

  • Known Risks Confirmed: The system successfully ranked known side effects, like gastrointestinal and intracranial bleeding, at the very top of the list.
  • Quantitative Verification: In the formal study of ~170,000 new users, Rivaroxaban showed a significantly higher risk for gastrointestinal bleeding (Adjusted Odds Ratio: 1.37) compared to the active comparator.
  • Filtering False Leads: The study identified sepsis and acute liver injury as potential signals during the "discovery" phase, but the "verification" phase correctly identified them as likely spurious or not drug-related in this specific context.

Limitations and Future Work

  • The Time Gap: EHC data (claims) usually has a reporting lag. Spontaneous reporting systems are still faster for catching "immediate" disasters with newly released drugs.
  • Verification Bottleneck: While the signal detection phase is "fast" and automated, the verification study is "time-consuming" and requires bespoke epidemiological design. The authors suggest that only a limited number of high-priority signals can be verified this way.
  • Future Directions: The authors suggest that weighting the "votes" of different algorithms (giving more weight to methods known to perform better in specific scenarios) could further refine the Borda ranking.

Summary updated

Abstract

Abstract Introduction Pharmacovigilance shifted its focus from spontaneous reporting systems to electronic health care (EHC) data. Usually, a single statistical method is used to detect signals, i.e., potential adverse drug reactions (ADRs). Objective and Method We present a novel approach to detect ADRs in EHC databases. It aggregates the results of multiple statistical signal detection methods applying Borda count ranking, a preference voting system, which results are used by an expert committee to select plausible signals. The obtained signals are afterwards investigated in tailored pharmacoepidemiological studies to provide support of plausibility or spuriousness of the signal. We showcase the approach using data from the German Pharmacoepidemiological Research Database on drug reactions of the direct oral anticoagulant rivaroxaban. Results of four statistical methods are aggregated into Borda count rankings: longitudinal Gamma Poisson shrinker, Bayesian confidence propagation neural network, random forests and LASSO. A verification study designed as nested active comparator case-control study was conducted. We included patients diagnosed with atrial fibrillation who initiated anticoagulant treatment with rivaroxaban or with phenprocoumon as active comparator between 2011 and 2017. Results The case study highlights that our Borda ranking approach ( https://borda.bips.eu ) is fast, able to retrieve known ADRs and find other interesting signals. Hasty false conclusions are avoided by a verification study, which is, however, time-consuming. Conclusion Post-market signal detection in EHC data is useful to identify and validate safety signals, particularly a few years after first admission to the market, when spontaneous reports are less frequent and more EHC data are available.
BibTeX
@misc{dijkstra2022discovery,
  author = {LJ Dijkstra and Tania Schink and Roland Linder and Markus Schwaninger and Iris Pigeot and Marvin N. Wright and Ronja Foraita},
  title = {{A Discovery and Verification Approach for Pharmacovigilance using Electronic Health Care Data}},
  year = {2022},
  doi = {10.1101/2022.05.10.22274885},
  howpublished = {medRxiv},
}
Figure