A time-indexed reference standard of adverse drug reactions

Rave Harpaz, David Odgers, Greg Gaskin, William DuMouchel, Rainer Winnenburg, Olivier Bodenreider, Anna Ripple, Ana Szarfman, Alfred Sorbello, Eric Horvitz, Ryen W. White, Nigam H. ShahScientific 2014 · DOI

Abstract

Undetected adverse drug reactions (ADRs) pose a major burden on the health system. Data mining methodologies designed to identify signals of novel ADRs are of deep importance for drug safety surveillance. The development and evaluation of these methodologies requires proper reference benchmarks. While progress has recently been made in developing such benchmarks, our understanding of the performance characteristics of the data mining methodologies is limited because existing benchmarks do not support prospective performance evaluations. We address this shortcoming by providing a reference standard to support prospective performance evaluations. The reference standard was systematically curated from drug labeling revisions, such as new warnings, which were issued and communicated by the US Food and Drug Administration in 2013. The reference standard includes 62 positive test cases and 75 negative controls, and covers 44 drugs and 38 events. We provide usage guidance and empirical support for the reference standard by applying it to analyze two data sources commonly mined for drug safety surveillance.

Introduction

Identifying adverse drug reactions (ADRs) after a drug has entered the market is a critical component of public health surveillance. While data mining algorithms are frequently employed to scan patient records and spontaneous reports for safety signals, evaluating the efficacy of these algorithms is challenging. Most existing benchmarks are retrospective, relying on drug-event pairs that have been established for decades. This introduces a "hindsight bias," where public knowledge of a side effect increases reporting and publication rates, making it artificially easy for algorithms to "detect" the signal.

To improve drug safety, researchers require prospective evaluation tools that test whether an algorithm can detect an emerging threat before it is officially recognized. This paper introduces a curated, time-indexed reference standard based on FDA label changes from 2013 to address this gap.

Summary of the study design and metadata for the pharmacovigilance reference standard.

Methodology

The creation of this "gold standard" involved a rigorous multi-step pipeline to ensure data quality and machine readability:

  1. Source Selection: Every safety-related label change was extracted from the FDA’s MedWatch site for the year 2013.
  2. Filtering: Out of 975 candidate changes, only those representing truly novel safety signals were retained. Filtering rules excluded events already mentioned in older labels, overly general terms (e.g., "hypersensitivity"), or associations with multi-ingredient drugs where the specific culprit was already known.
  3. Normalization: Drugs were mapped to RxNorm codes and adverse events to MedDRA (Medical Dictionary for Regulatory Activities) codes to ensure interoperability with standard data mining tools.
  4. Negative Controls: To provide a balanced benchmark, 75 negative controls were created by pairing drugs and events with no known association, manually verified against existing drug labels.
Workflow for curating the reference standard from FDA safety labeling changes, illustrated with the example of Ampyra.

The 2013 Reference Standard

The resulting benchmark consists of 137 drug-event pairs, including 62 positive test cases and 75 negative controls. These cases cover 44 distinct drugs and 38 unique adverse events. This dataset is specifically designed to measure the "lead time" of safety models—testing if an algorithm could have identified a signal years before the FDA took regulatory action.

Empirical Validation and Key Insights

The reference standard was validated using two major data sources: the FDA Adverse Event Reporting System (FAERS) and MEDLINE (biomedical literature). Key findings include:

  • Lead Time Decay: Signal detection accuracy (AUC) decreases as models attempt to predict further into the past. In FAERS, the AUC was 0.81 at the time of the label change but dropped to 0.70 when using data from five years prior.
  • The "Newness" Filter: Only 9% of 2013 FDA label changes were suitable for a prospective benchmark, as most updates are minor or reflect known "class effects."
  • Literature Mining: In MEDLINE, using broad event definitions (including related symptoms) outperformed narrow, specific terms. Furthermore, simple article counts proved more predictive of future ADRs than complex statistical measures like Odds Ratios.

Discussion

While the 2013 Reference Standard provides a vital tool for prospective evaluation, it is limited by its sample size (62 positive cases) due to strict quality filtering. Additionally, the FDA often begins internal investigations up to two years before a label change is finalized, meaning the practical lead time for algorithms may be shorter than the benchmark suggests. Future work should focus on continuous maintenance of such standards through community initiatives like OHDSI to keep pace with evolving drug safety data.

Summary updated

Abstract

Undetected adverse drug reactions (ADRs) pose a major burden on the health system. Data mining methodologies designed to identify signals of novel ADRs are of deep importance for drug safety surveillance. The development and evaluation of these methodologies requires proper reference benchmarks. While progress has recently been made in developing such benchmarks, our understanding of the performance characteristics of the data mining methodologies is limited because existing benchmarks do not support prospective performance evaluations. We address this shortcoming by providing a reference standard to support prospective performance evaluations. The reference standard was systematically curated from drug labeling revisions, such as new warnings, which were issued and communicated by the US Food and Drug Administration in 2013. The reference standard includes 62 positive test cases and 75 negative controls, and covers 44 drugs and 38 events. We provide usage guidance and empirical support for the reference standard by applying it to analyze two data sources commonly mined for drug safety surveillance.
BibTeX
@inproceedings{harpaz2014time,
  author = {Rave Harpaz and David Odgers and Greg Gaskin and William DuMouchel and Rainer Winnenburg and Olivier Bodenreider and Anna Ripple and Ana Szarfman and Alfred Sorbello and Eric Horvitz and Ryen W. White and Nigam H. Shah},
  title = {{A time-indexed reference standard of adverse drug reactions}},
  booktitle = {Scientific},
  year = {2014},
  doi = {10.1038/sdata.2014.43},
}
Figure