Quantifying the Severity of Adverse Drug Reactions Using Social Media: Network Analysis

Adam Lavertu, Tymor Hamamsy, Russ B. AltmanJournal of Medical Internet Research 2021 · DOI

TL;DR

This paper introduces a novel network-based approach using social media word embeddings to quantify the severity of adverse drug reactions (ADRs), significantly expanding the number of ADRs with quantitative severity estimates. The developed SAEDR (Severity of Adverse Events Derived from Reddit) scores show strong correlations with real-world outcomes and human judgment, offering a valuable tool for pharmacovigilance.

What Problem Does This Paper Address?

Adverse drug reactions (ADRs) are a major public health concern, leading to significant morbidity, mortality, and economic costs. Current methods for classifying ADR severity rely on qualitative and categorical systems (e.g., boxed warnings, warnings, adverse reactions), which can be imprecise and make it difficult to compare the relative risks of different drugs. While previous efforts have attempted to quantify ADR severity, they were limited in scope due to the time and cost involved. This paper aims to address this gap by developing a more scalable and quantitative method to estimate ADR severity, thereby improving the quality and breadth of severity assessments.

Methods

The core of this research lies in a semisupervised network-based label propagation method. Here's a breakdown:

  • Word Embeddings: The authors leverage pre-trained word embeddings from the RedMed project, which were specifically trained on a large corpus of Reddit comments enriched with medical terms. This allows them to represent ADR terms as dense numerical vectors that capture semantic meaning.
  • Lexical Network Creation: A network is constructed where nodes represent ADR terms. Edges connect terms based on their proximity in the embedding space (k-nearest neighbors), with edge weights reflecting cosine similarity.
  • Seed Selection: A crucial step is identifying initial "seed" terms for "severe" and "benign" ADRs. The authors explore two strategies: using crowdworker-ranked seeds (from a previous study) and using seeds derived from real-world outcomes (like death or serious outcomes from FAERS data). They also employ lexical filtering to reduce redundancy within seed sets.
  • Label Propagation: A modified node2vec algorithm is used to perform weighted random walks on the constructed network, starting from the labeled seed nodes. The idea is that terms semantically similar to severe ADRs will be visited more often by walks originating from severe seeds, and vice versa.
  • SAEDR Score Calculation: The Severity of Adverse Events Derived from Reddit (SAEDR) score for each ADR is calculated based on the relative number of times it's encountered during random walks from severe versus benign seed nodes. This score is normalized to a 0-1 range.
  • Drug-Risk Profile (DRIP) Scores: To assess drug-level risk, DRIP scores are calculated by combining the SAEDR score of each ADR associated with a drug with the frequency of that ADR on the drug's label.

Datasets

  • RedMed Word Embeddings: Used for representing ADR terms. These were trained on over 580 million health-enriched Reddit comments from more than 10 million users.
  • Medical Dictionary for Regulatory Activities (MedDRA): Version 22.0 was used as the source for ADR terms and phrases. The study focused on terms with specific semantic types (e.g., Disease or Syndrome, Finding, Sign or Symptom).
  • Gottlieb et al. Crowdsourced Severity Data: Used for validating the SAEDR scores and for selecting seed terms. This dataset contained severity estimates for 2929 ADRs.
  • Food and Drug Administration Adverse Event Reporting System (FAERS): Data up to June 30, 2020, was used to derive real-world outcome statistics for ADRs (death, serious outcome, no outcome) and to inform seed selection.
  • Side Effect Resource (SIDER): Used to retrieve ADR frequency information for calculating DRIP scores.

Key Insights and Ideas

  • Social Media as a Rich Source for Severity: The study demonstrates that publicly available social media data (Reddit) can be effectively leveraged, through word embeddings, to capture nuanced perceptions of ADR severity, going beyond traditional clinical data.
  • Network Analysis for Quantifying Abstract Concepts: The application of network-based label propagation to quantify something as abstract as "severity" is a powerful demonstration of the technique's versatility in scientific domains.
  • Real-World Outcomes as a Strong Signal: Using FAERS data (cases of death or serious outcomes) as a basis for seed selection yielded SAEDR scores that were highly correlated with both human judgments and other real-world metrics.
  • Postmarketing ADRs are More Severe: A significant finding is that ADRs discovered after a drug has been approved and is on the market tend to be more severe than those identified during clinical trials. This underscores the importance of postmarketing surveillance.
  • Sex Differences in Severity: ADRs disproportionately reported in men were found to be more severe than those disproportionately reported in women, aligning with previous research.
  • Nuance in Severity: The method can distinguish severity even within seemingly similar categories, such as benign versus malignant neoplasms, or different types of heart failure.

Key Results

  • SAEDR Score Performance: The SAEDR scores achieved a Spearman correlation of 0.735 with the crowdworker-ranked test set and 0.595, 0.633, and -0.748 for correlations with death, serious outcome, and no outcome in FAERS data, respectively. These correlations were statistically significant (P<.001).
  • Expanded Severity Estimates: The study generated quantitative severity estimates for 12,198 ADR concepts, a significant increase from previous efforts.
  • DRIP Score Performance: The DRIP scores for 968 drugs showed a Spearman correlation of 0.377 with drugs ranked by FAERS cases resulting in death.
  • Boxed Warnings: ADRs included in boxed warnings had a median SAEDR score of 0.624, significantly higher than those not included (median 0.538).
  • Postmarketing vs. Clinical Trial: Postmarketing ADRs were significantly more severe (P<.001) than clinical trial ADRs.

Limitations and Future Work

  • Social Media Bias: The authors acknowledge that Reddit's user base may be skewed towards certain demographics (e.g., younger, male), potentially introducing biases in the word embeddings and subsequent severity estimates. Future work could involve incorporating embeddings from other social media platforms or biomedical literature.
  • FAERS Data Limitations: The FAERS data is limited by its case-level reporting and potential reporting biases. The absence of a "no outcome" category in FAERS can also affect severity interpretation.
  • ADR Modifier Terms: The current MedDRA terminology doesn't capture intensity modifiers (e.g., "excruciating" pain). The authors suggest repurposing general word sentiment analysis techniques to assign severity to these modifier terms.
  • Sex-Specific ADRs at HLGT Level: Sex-specific ADRs were analyzed at the High-Level Group Term (HLGT) level, which averages preferred terms (PTs). Analyzing at the PT level could reveal more granular differences.
  • DRIP Score Limitations: The DRIP scores' correlation with real-world outcomes is modest, partly due to the limitations of available ADR frequency data (especially for postmarketing drugs) and potential reporting biases in FAERS.

The authors have made the SAEDR and DRIP scores publicly available, enabling further quantitative analysis in pharmacovigilance.

Summary updated

Abstract

Our SAEDR and DRIP scores are well correlated with the real-world outcomes of the entities they represent and have demonstrated utility in pharmacovigilance research. We make the SAEDR scores for 12,198 ADRs and the DRIP scores for 968 drugs publicly available to enable more quantitative analysis of pharmacovigilance data.
BibTeX
@article{lavertu2021quantifying,
  author = {Adam Lavertu and Tymor Hamamsy and Russ B. Altman},
  title = {{Quantifying the Severity of Adverse Drug Reactions Using Social Media: Network Analysis}},
  journal = {Journal of Medical Internet Research},
  year = {2021},
  doi = {10.2196/27714},
}
Figure