Multi-Task Pharmacovigilance Mining from Social Media Posts

Shaika Chowdhury, Chenwei Zhang, Philip S. Yu

TL;DR

This paper introduces a multi-task neural network framework designed to monitor drug safety by analyzing social media posts. By jointly learning to classify posts, extract adverse drug reactions (ADRs), and identify drug "indications" (the reason a drug is taken), the model significantly improves detection accuracy—especially for complex, multi-word descriptions of side effects.

What Problem Does This Paper Address?

Pharmacovigilance—the science of monitoring drug safety—traditionally relies on clinical trials and voluntary reporting by doctors. However, these methods are often slow and suffer from massive under-reporting. Social media (specifically Twitter) offers a real-time, "in-the-wild" look at patient experiences, but mining this data is notoriously difficult.

The authors identify two major hurdles:

  1. Diverse Expressions: Patients rarely use medical jargon. Instead of "insomnia," they might write "never sleeping" or "sleep deprived." These "phrasal" ADRs are hard for standard models to capture.
  2. Ambiguity: A medical condition mentioned in a tweet might be the reason someone took the drug (an Indication) or a result of taking it (an ADR). For example, in "I took Paroxetine for my panic attacks but it made me fat," panic attacks is the indication, while fat is the ADR.

Methods

The researchers proposed a Multi-Task RNN Encoder-Decoder framework. Instead of treating classification and extraction as separate problems, they trained one model to do three things at once:

  • ADR Classification: Is this tweet reporting a side effect? (Binary)
  • ADR Labeling: Which specific words describe the side effect? (Sequence labeling)
  • Indication Labeling: Which words describe why the drug was taken? (Sequence labeling)

Technical Highlights:

  • Shared Encoder: A Bidirectional LSTM (Bi-LSTM) processes the tweet, creating a shared representation that captures the semantic overlap between the three tasks.
  • Hybrid Embeddings: To handle the slang and typos of Twitter, the model combines standard word embeddings (GloVe) with character-level embeddings.
  • Coverage-based Attention: This is the "secret sauce." Standard attention mechanisms often focus on a single "keyword." The authors added a coverage mechanism that keeps track of which words the model has already "attended to" within a sliding window. This forces the model to recognize long, descriptive phrases (like "gain unwanted pounds and water weight") rather than just single words.

Datasets

The study utilized the PSB 2016 Social Media Shared Task Twitter dataset.

  • ADR Classification: A balanced set of 2,162 tweets.
  • ADR/Indication Labeling: A subset of 1,081 tweets available for download, supplemented with 203 additional tweets from a 2015 study to bolster the training data.
  • The authors manually labeled the supplemental data to ensure consistency across the three tasks.

Key Insights and Ideas

  • The Power of "Also Knowing": The most significant insight is that learning to identify Indications actually makes the model better at identifying ADRs. By understanding what a "reason for taking a drug" looks like, the model is less likely to mislabel it as a side effect.
  • Attention Needs Memory: Standard attention is "forgetful." By implementing a coverage vector, the model maintains a memory of the context, which is essential for capturing the colloquial, rambling way people describe health issues online.
  • Multi-Tasking as a Regularizer: Training on multiple related tasks prevents the model from over-fitting on the quirks of a single dataset, leading to better generalization.

Key Results

The multi-task framework consistently outperformed state-of-the-art baselines (like CRNN and CNNA) and single-task versions of itself:

  • ADR Labeling: Achieved an F1-score of 79.24%, a significant jump over the 68.47% achieved by a non-attention baseline.
  • ADR Classification: Reached an F1-score of 70.69%, outperforming specialized models like CRNN (51.00%) and CNNA (49.00%) on the same dataset.
  • Qualitative Success: Case studies showed the model correctly identifying "zombie" and "heart was hurting" as ADRs in contexts where standard models failed or mislabeled them as indications.

Limitations and Future Work

  • Data Sparsity: The scores for "Indication Labeling" (F1: 48.82%) were lower than the other tasks. The authors attribute this to the fact that many people tweet about their side effects without mentioning why they took the drug in the first place.
  • Twitter Volatility: The authors noted difficulty in downloading the full original dataset because many tweets had been deleted or accounts made private, a common "link rot" issue in social media research.
  • Future Directions: The authors suggest that incorporating even more tasks—such as identifying the specific drug-ADR relationship or drug-drug interactions—could further refine the model's performance.

Summary updated

Abstract

Social media has grown to be a crucial information source for pharmacovigilance studies where an increasing number of people post adverse reactions to medical drugs that are previously unreported. Aiming to effectively monitor various aspects of Adverse Drug Reactions (ADRs) from diversely expressed social medical posts, we propose a multi-task neural network framework that learns several tasks associated with ADR monitoring with different levels of supervisions collectively. Besides being able to correctly classify ADR posts and accurately extract ADR mentions from online posts, the proposed framework is also able to further understand reasons for which the drug is being taken, known as »indications», from the given social media post. A coverage-based attention mechanism is adopted in our framework to help the model properly identify »phrasal» ADRs and Indications that are attentive to multiple words in a post. Our framework is applicable in situations where limited parallel data for different pharmacovigilance tasks are available. We evaluate the proposed framework on real-world Twitter datasets, where the proposed model outperforms the state-of-the-art alternatives of each individual task consistently.
BibTeX
@inproceedings{chowdhury2018multi,
  author = {Shaika Chowdhury and Chenwei Zhang and Philip S. Yu},
  title = {{Multi-Task Pharmacovigilance Mining from Social Media Posts}},
  booktitle = {The Web Conference},
  year = {2018},
  url = {https://openreview.net/forum?id=HJNMUzZuZS},
}

Loading related content...

Figure