CLaC at SMM4H Task 1, 2, and 4

Parsa Bagherzadeh, Nadia Sheikh, Sabine BerglerEMNLP 2018

TL;DR

This paper presents CLaC Labs' participation in the SMM4H shared tasks, focusing on a unified recurrent neural network architecture across three distinct tasks. They explore the impact of linguistic features and a drug gazetteer, finding that while their architecture shows promise, it may be prone to overfitting on simpler tasks, and the effectiveness of added features varies significantly by task.

What Problem Does This Paper Address?

The Social Media Mining for Health Applications (SMM4H) shared tasks aim to leverage social media data for health-related insights, such as identifying drug mentions, distinguishing actual medication intake from mentions, and recognizing flu vaccine intent. This paper addresses the challenge of building a robust and adaptable system that can perform well across these diverse, yet related, health-focused natural language processing tasks. The authors explore how a single, consistent architecture, augmented with various linguistic features and external knowledge sources, can be applied to these different problems and how its performance is affected by task complexity and data characteristics.

Methods

The core of CLaC's approach is a feed-forward neural network with a recurrent neuron, specifically an LSTM cell. They experimented with three main variations of this architecture:

  • Full: A standard LSTM followed by a 3-layer feed-forward network with soft-max activation for classification.
  • NR (Network Removed): Only the LSTM and decision neurons are used, omitting the feed-forward network.
  • Full+At (Full with Attention): The LSTM outputs the sequence of activations at each time step, allowing for attention mechanisms.

They also investigated the impact of augmenting word vectors with:

  • Gazetteer: A list of drug names scraped from DrugBank.
  • Linguistic Features: Extracted using a CLaC pipeline, including part-of-speech tags, verb clusters (voice, tense, aspect), negation triggers, and modality terms.

The neural networks were implemented using Keras with TensorFlow as the backend. Tweets were normalized to a fixed size and represented by Word2Vec embeddings.

Datasets

The paper primarily discusses the datasets provided for the SMM4H shared tasks:

  • Task 1: A binary categorization task to identify drug mentions in a medical sense. The training data consisted of over 9000 tweets, balanced across categories.
  • Task 2: A 3-way classification task to distinguish between actual medication intake, potential intake, and mere mentions. The training data comprised 14482 tweets, which were highly imbalanced.
  • Task 4: A binary task to identify tweets indicating the receipt or intention to receive a flu vaccine. Only 4502 tweets were available for training out of an initial 8000.

For evaluation, a validation set of 1000 tweets was randomly selected from the training data for Tasks 1 and 2.

Key Insights and Ideas

  • Architecture Versatility vs. Overfitting: The unified architecture proved useful for comparing performance across tasks. However, for simpler tasks like Task 1, the system exhibited high training accuracy, suggesting significant overfitting, and the performance difference between the full architecture and a simpler one (NR) was minimal.
  • Linguistic Features and Gazetteers: Task-Dependent Impact: The effectiveness of adding gazetteer lists and linguistic features varied greatly. For Task 1, the gazetteer improved training accuracy but not necessarily validation accuracy. For Task 4, the drug gazetteer was not useful, but the "Full+All" configuration (likely including linguistic features) significantly outperformed simpler models, suggesting these features were crucial for semantic understanding.
  • Attention's Role: The "Full+At" configuration showed promise, particularly in validation runs for Task 2, indicating that attending to the sequence of LSTM outputs can be beneficial.
  • Task Definition Matters: The richer semantic component of Task 2 led to greater variance in team performance compared to Task 1, highlighting the challenge of nuanced classification.

Key Results

  • Task 1: The best competition run for CLaC was the "Full" architecture. Interestingly, the performance difference between the best and worst system was 0.1399, suggesting a relatively stable performance landscape for this task.
  • Task 2: The "Full+All+At" configuration achieved a validation accuracy of 0.85. However, their official results were lower, with a micro-averaged F1 score of 0.383, placing them towards the bottom of the competition. This was attributed to a low recall score.
  • Task 4: The "Full+All" configuration was CLaC's best run, achieving a recall of 0.89 and performing closest to the competition mean, despite the small training data size.

Limitations and Future Work

  • Overfitting: The authors acknowledge that their architecture showed signs of overfitting, particularly on Task 1, leading to a potentially brittle system.
  • Input Normalization: They suggest that input binning and normalization techniques need careful review, as they might be ignoring key terms.
  • Parameter Interplay: The detailed interplay between the LSTM network, gazetteer, and linguistic features requires further study.
  • Future Directions: The paper implicitly suggests exploring more sophisticated attention mechanisms, refining feature engineering, and potentially developing task-specific architectural modifications to mitigate overfitting and improve performance on nuanced tasks.

Summary updated

BibTeX
@inproceedings{bagherzadeh2018clac,
  author = {Parsa Bagherzadeh and Nadia Sheikh and Sabine Bergler},
  title = {{CLaC at SMM4H Task 1, 2, and 4}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure