Identifying Adverse Drug Events Mentions in Tweets Using Attentive, Collocated, and Aggregated Medical Representation

Xinyan Zhao, Deahan Yu, V.G.Vinod VydiswaranACL 2019

TL;DR

This paper introduces CLAPA, a novel neural network architecture that significantly improves the identification of adverse drug events (ADEs) in tweets by leveraging contextual information through collocated word embeddings and aggregated representations. By focusing on the relationships between words within tweets, CLAPA achieves state-of-the-art performance, outperforming existing methods and average system scores in a recent shared task.

What Problem Does This Paper Address?

Identifying mentions of medical concepts, particularly adverse drug events (ADEs), in social media is a challenging task due to the inherent variability and informal nature of language used in platforms like Twitter. Traditional Natural Language Processing (NLP) methods often struggle with this noisy data. While pre-trained word embeddings provide a general understanding of words, they may not capture the specific nuances of medical terminology in a social media context. Existing approaches have attempted to address this by using domain-specific knowledge graphs or by fine-tuning embeddings, but there's still room for improvement in effectively representing medical concepts and their relationships within short, informal text. This paper aims to bridge this gap by proposing a method that enriches word representations using local contextual information derived directly from the training data.

Methods

The core of this paper is the CLAPA (Collocated LSTM with Attentive Pooling and Aggregated representation) architecture. CLAPA builds upon a bidirectional LSTM (BiLSTM) for sentence encoding and incorporates several key innovations:

  • Medical Collocation Embedding: This is a novel embedding method that goes beyond standard word embeddings. It leverages a "collocation graph" built from the training data, where nodes are words and edges represent co-occurrence. For each word, it learns an "attentive neighborhood embedding" that captures the semantic meaning of its neighbors in the graph, weighted by attention scores. This allows the model to understand how a medical concept is used in its local context.
  • Aggregated Medical Representation: This component further enriches the sentence representation by aggregating the medical concept information within a sentence. It uses the attentive embeddings of medical concepts and a "delta" function to create a residual representation that is added to the final output.
  • Attentive Pooling: The model employs multi-head attention mechanisms within the LSTM encoder. This allows the model to focus on the most relevant parts of the sentence when generating its representation. Max pooling is then applied to extract important signals from these attentive hidden states.

These components are combined to create a comprehensive representation that is then fed into a fully connected network for ADE mention classification.

Datasets

The authors utilized the dataset provided for Task 1 of the SMM4H (Social Media Mining for Health Applications) 2019 shared task. This dataset consists of 25,678 annotated tweets, split into a training set (80%) and a validation set (20%). The tweets were cleaned by removing punctuation, special characters, and normalizing mentions, URLs, and numbers. The paper also mentions using fastText for pre-trained word embeddings, trained on this dataset.

Key Insights and Ideas

  • Local Context is King: The paper strongly suggests that understanding the immediate linguistic neighborhood of a medical term is crucial for accurate identification of ADEs in social media. Simply relying on general pre-trained embeddings is insufficient.
  • Collocation Graphs as a Rich Source: Building a collocation graph directly from the training data provides a valuable, domain-specific resource for learning word relationships. This is more tailored than general knowledge graphs.
  • Attention for Focus: The use of attention mechanisms, both in embedding neighbors and in sentence encoding, is vital for highlighting the most informative parts of the text, leading to better representations.
  • Aggregation Adds Value: The aggregated medical representation acts as a useful "residual" that further boosts performance, particularly in improving recall.
  • Concept Vocabulary Matters: The study found that training with concepts identified only from ADE tweets, rather than all tweets, led to a higher F1 score, suggesting that focusing on the specific language used in positive examples can be beneficial.

Key Results

The CLAPA model achieved the best performance among the models tested. Specifically:

  • The full CLAPA model (CLAPA) achieved an F1 score of 0.5998 on the validation set.
  • When evaluated on the test set of the SMM4H 2019 shared task, the CLAPA model achieved an F1 score of 0.5676.
  • This score is significantly better than the average system score of 0.5019 on the same test set, indicating CLAPA's superior performance.
  • The paper also demonstrated that the model's performance is robust even with reduced training data (down to 10%), maintaining a consistently higher F1 score than the baseline LAP model.

Limitations and Future Work

The authors acknowledge several limitations:

  • MetaMap Limitations: MetaMap, used for concept extraction, might not perfectly identify relevant concepts from social media text due to linguistic differences compared to clinical documents.
  • Precision vs. Recall Trade-off: While the proposed methods improved recall and overall F1, they slightly reduced precision. Future work could focus on improving precision.
  • Collocation Graph Dependency: The collocation graph is built solely on the training data. If the training data is not representative, the quality of neighborhood information might suffer.

Future work directions include:

  • Exploring pre-trained word embeddings trained on medical corpora or larger text datasets.
  • Investigating the use of domain-based knowledge bases like UMLS to expand concept coverage.
  • Further improving precision and exploring methods to address the trade-off.
  • Testing the approach on other domain-specific tasks to assess its generalizability.
  • Exploring the use of unique nodes in different classes (ADE vs. non-ADE) for concept identification.

Summary updated

BibTeX
@inproceedings{zhao2019identifying,
  author = {Xinyan Zhao and Deahan Yu and V.G.Vinod Vydiswaran},
  title = {{Identifying Adverse Drug Events Mentions in Tweets Using Attentive, Collocated, and Aggregated Medical Representation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure