Using Machine Learning and Deep Learning Methods to Find Mentions of Adverse Drug Reactions in Social Media

Pilar López Úbeda, Manuel Carlos Díaz Galiano, Maite Martin, L. Alfonso Urena LopezACL 2019

TL;DR

This paper explores how machine learning and deep learning can automatically identify mentions of adverse drug reactions (ADRs) in informal social media text, specifically Twitter. The authors developed and evaluated several models for both classifying tweets that contain ADRs and extracting the specific text spans indicating ADRs, achieving competitive results against other participants in the SMM4H workshop.

What Problem Does This Paper Address?

Monitoring public health is crucial, and social media platforms like Twitter have become rich sources of real-time health-related information. However, extracting meaningful insights from this informal text presents a significant challenge. Specifically, identifying mentions of Adverse Drug Reactions (ADRs) – injuries occurring after a drug is used as recommended – is vital for pharmacovigilance and understanding drug safety. Traditional methods for ADR detection are often manual and time-consuming. This paper addresses the need for automated systems that can effectively mine social media for ADR mentions, contributing to the broader goal of Social Media Mining for Health Applications (SMM4H).

Methods

The researchers employed a combination of machine learning and deep learning techniques. For Task 1 (Automatic Classification of ADR Mentions), they explored:

  • SVM (Support Vector Machines): A baseline model using TF-IDF features and linear kernel.
  • SVM + Features: The SVM baseline augmented with external medical knowledge, specifically using the MetaMap tool to extract semantic types of medical concepts (e.g., "Disease or Syndrome," "Sign or Symptom") and incorporating these as features.
  • CNN (Convolutional Neural Network): A deep learning approach utilizing an embedding layer, convolutional layers, and pooling layers, trained with pre-trained Twitter word vectors (GloVe).

For Task 2 (Extraction of ADR Mentions), they focused on Conditional Random Fields (CRF):

  • CRF: A baseline CRF model incorporating contextual features (previous/next words), part-of-speech tags, and lemmas.
  • CRF + W2V: The CRF model enhanced with pre-trained GloVe word embeddings (50 dimensions) as additional features.
  • CRF + BC + W2V: Further augmenting the CRF + W2V model with features derived from Brown clustering, which groups words with similar contexts.

Preprocessing steps for both tasks included text normalization (lowercase, HTML entity replacement), expansion of contractions, removal of hashtags and @mentions, and filtering for alphanumeric words.

Datasets

The study utilized datasets provided for the SMM4H workshop.

  • Task 1 Dataset: Contained tweets with binary annotations indicating the presence or absence of ADRs. The training set comprised 25,672 tweets (2,374 positive, 23,298 negative), and the test set had 4,5175 tweets.
  • Task 2 Dataset: A subset of Task 1 tweets tagged as having ADRs, along with an equal number of non-ADR tweets. This dataset included tweet IDs, start/end spans of ADR mentions, ADR annotations, and the text covered by the span. The training set had 2,367 tweets (1,212 positive, 1,155 negative), and the test set contained 1,573 tweets.

Key Insights and Ideas

  • Value of External Medical Knowledge: Augmenting traditional ML models (like SVM) with domain-specific features derived from tools like MetaMap significantly improved performance, highlighting the importance of incorporating medical ontologies.
  • Context is King for Extraction: For ADR mention extraction (Task 2), features related to word context (previous/next words) and semantic grouping (Brown clustering) proved crucial for improving performance, especially in recall.
  • Deep Learning Shows Promise: The CNN model for classification performed competitively, often outperforming ML-based approaches, suggesting deep learning's potential for handling the nuances of informal social media text.
  • Informal Text is Challenging: The relatively low average scores across participants in Task 1 suggest that classifying ADRs in informal tweets is a difficult task, underscoring the need for robust NLP techniques.

Key Results

  • Task 1 (Classification): The CNN model achieved an F1 score of 0.4969, slightly below the average participant score of 0.5019 but demonstrating strong performance. The SVM + features model also showed improvement over the baseline SVM.
  • Task 2 (Extraction):
    • Relaxed Matching: The CRF + BC + W2V model achieved an F1 score of 0.486, surpassing the average participant score of 0.3169. While precision was high, recall was noted as a weakness.
  • Strict Matching: The CRF + BC + W2V model achieved an F1 score of 0.36, also exceeding the average participant score of 0.326. Similar to relaxed matching, recall was lower than precision.

Limitations and Future Work

The authors acknowledge that while feature engineering improved SVM performance, it didn't lead to a substantial increase, and some systems even saw a decrease in precision. They suggest exploring different feature sets for future work. For Task 2, they noted that while precision was often high, recall was a consistent challenge, indicating a need to improve exhaustiveness in identifying relevant instances. Future work includes exploring deep learning neural networks for ADR extraction (instead of CRF), further improving neural network architectures, and incorporating additional text features. The team also plans to participate in all proposed tasks in future workshops to further test and refine their systems.

Summary updated

BibTeX
@inproceedings{ubeda2019using,
  author = {Pilar López Úbeda and Manuel Carlos Díaz Galiano and Maite Martin and L. Alfonso Urena Lopez},
  title = {{Using Machine Learning and Deep Learning Methods to Find Mentions of Adverse Drug Reactions in Social Media}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure