Adverse drug reaction signal detection via the long short-term memory model

Ming Cao, Yanna Chi, Jinyang Yu, Yang Yu, Ruogu Meng, Jinzhu JiaFrontiers in Pharmacology 2025 · DOI

TL;DR

Researchers have developed a Long Short-Term Memory (LSTM) deep learning model that significantly improves the detection of Adverse Drug Reaction (ADR) signals. By treating entire clinical reports as "free text" rather than just structured data points, the model achieved a sensitivity of 95.16%, vastly outperforming traditional statistical methods and standard machine learning classifiers.

What Problem Does This Paper Address?

Adverse Drug Reactions (ADRs) are a leading cause of hospital mortality and impose a massive financial burden on healthcare systems—costing an estimated $75 billion annually. To catch these signals early, health authorities use "spontaneous reporting systems."

However, the traditional tools used to analyze this data—known as disproportionality analysis (e.g., ROR, PRR)—suffer from high false-positive rates. These methods typically only look at a tiny subset of variables (like drug name and reaction type) and ignore the rich, descriptive narratives provided by healthcare workers. The challenge is: how do we build a system that utilizes the full context of a report to identify true safety signals without drowning in "noise"?

Methods

The authors propose moving away from simple frequency counts toward sequence-based deep learning.

  1. Data Transformation: Instead of treating the 48 selected variables (like dosage, patient history, and reaction descriptions) as separate columns in a spreadsheet, they concatenated them into a single string of "long free text."
  2. The LSTM Model: They employed a Long Short-Term Memory (LSTM) network. LSTMs are a type of Recurrent Neural Network (RNN) specifically designed to remember information over long sequences, making them ideal for processing the semantic patterns in clinical text.
  3. Benchmarking: The researchers compared the LSTM against:
  4. Traditional Statistical Methods: ROR, PRR, BCPNN, and MGPS.
  5. Standard Machine Learning: Logistic Regression (LR), Random Forest (RF), K-Nearest Neighbor (KNN), and Multilayer Perceptron (MLP).

Datasets

The study utilized real-world data from the Center for ADR Monitoring of Guangdong, China, spanning 2018 to 2019.

  • Initial Pool: Over 281,000 reports.
  • Final Dataset: 2,376 ADR signals, consisting of 448 "positive" signals (manually validated by experts) and 1,928 "negative" signals (sampled using traditional statistical methods).
  • Features: 72 initial variables were pruned down to 48 relevant features for model training.

Key Insights and Ideas

  • The Power of Narrative: The most striking insight is that converting structured data into a text-like format for an LSTM works better than keeping it structured for a Random Forest. This suggests that the relationships and sequences between variables (e.g., the reason for medication followed by a specific dosage and then a specific reaction) carry more weight than the individual variables in isolation.
  • Contextual Predictors: Through chi-square analysis, the authors found that "Reasons for taking medication" and "Dosage" were among the most important features. Traditional methods often ignore the "why" behind a prescription, but this model shows that intent is a critical filter for identifying true ADRs.
  • Overcoming the "Black Box" of MLP: Interestingly, the Multilayer Perceptron (a standard neural network) performed poorly (AUC 0.59), while the LSTM excelled (AUC 0.98). This highlights that for clinical reports, the order and long-term dependencies of information are more important than simple non-linear feature mapping.

Key Results

The LSTM model demonstrated a clear superiority across almost every metric:

  • Sensitivity: The LSTM reached 95.16%, compared to 86.83% for Logistic Regression and only 73.80% for the best traditional statistical method (BCPNN).
  • F1-Score: The LSTM achieved 0.9706, significantly higher than Logistic Regression (0.9063) and Random Forest (0.8647).
  • Area Under the Curve (AUC): The LSTM achieved a near-perfect 0.98, while the MLP struggled at 0.59.
  • False Positive Rate: In the disproportionality analysis comparison, the LSTM maintained a remarkably low false positive rate of 3.69%, whereas traditional methods like ROR and PRR saw rates exceeding 41%.

Limitations and Future Work

While promising, the authors note several hurdles:

  • Geographic Bias: The data comes from a single province in China. Reporting habits and demographic profiles might differ elsewhere, requiring multicenter validation to ensure the model generalizes globally.
  • Selection Bias: The "negative" cases used for training were selected using traditional methods, which might inadvertently exclude novel or atypical ADR patterns that those traditional methods already miss.
  • Data Quality: The model is sensitive to the quality of the input text. If reports are sparse or contain irregular medical terminology, the LSTM's performance may degrade.

The authors suggest that future work should focus on "active learning" to capture more atypical cases and integrating data from diverse international healthcare systems.

Summary updated

Abstract

The application of deep learning models shows potential to improve the detection performance in ADR monitoring.
BibTeX
@inproceedings{cao2025adverse,
  author = {Ming Cao and Yanna Chi and Jinyang Yu and Yang Yu and Ruogu Meng and Jinzhu Jia},
  title = {{Adverse drug reaction signal detection via the long short-term memory model}},
  booktitle = {Frontiers in Pharmacology},
  year = {2025},
  doi = {10.3389/fphar.2025.1554650},
}
Figure