IIITN NLP at SMM4H 2021 Tasks: Transformer Models for Classification on Health-Related Imbalanced Twitter Datasets

Varad Pimpalkhute, Prajwal Nakhate, Tausif DiwanNAACL 2021

TL;DR

This paper tackles the challenge of classifying health-related tweets, particularly those with imbalanced datasets, using transformer-based models. The authors demonstrate that data augmentation, combined with models like RoBERTa and BioBERT, significantly improves performance on these difficult tasks, offering a practical approach for extracting valuable health insights from social media.

What Problem Does This Paper Address?

Analyzing health-related information on social media platforms like Twitter holds immense potential for improving public health services. However, a major hurdle is the scarcity of well-annotated and balanced datasets. Many health-related tweets are imbalanced, meaning one class (e.g., normal tweets) vastly outnumbers another (e.g., tweets about adverse drug effects). This class imbalance can lead to models that are biased towards the majority class and perform poorly on the minority class, which often contains the most critical information. This paper addresses this challenge by developing transformer-based models specifically designed to handle imbalanced health-related tweet classification tasks.

Methods

The core of this work lies in employing transformer-based deep learning models for binary classification. The authors experimented with several established transformer architectures, including:

  • RoBERTa: Used for Task 1a (Adverse Drug Effects) and Task 4 (Adverse Potential Outcomes).
  • BioBERT: A BERT variant pre-trained on biomedical literature, used for Task 8 (Breast Cancer Self-reports).
  • Other models: BERT, DistilBERT, XLNET, BERT-Epi, and BERTweet were also explored.

To combat class imbalance, they applied three key techniques:

  • Undersampling: Reducing the number of instances in the majority class.
  • Oversampling: Duplicating instances in the minority class, either directly or using Synthetic Minority Over-Sampling Technique (SMOTE).
  • Data Augmentation: Generating synthetic tweets for the minority class using techniques like synonym replacement, word embedding, and contextual word embeddings. This is highlighted as distinct from oversampling because it introduces variations in the text.

The models were trained and evaluated using the Flair framework, with hyperparameter tuning focusing on learning rate, epochs, batch size, and optimizer. A weighted loss function was also used, penalizing the rare class loss more heavily.

Datasets

The paper utilizes datasets from the Sixth Social Media Mining for Health Applications (SMM4H) 2021 workshop, specifically for three shared tasks:

  • Task 1a: Distinguishing tweets mentioning Adverse Drug Effects (ADE) from other tweets (NoADE). This dataset is highly imbalanced (approx. 1:13 ratio).
  • Task 4: Distinguishing tweets mentioning Adverse Potential Outcomes (APO) from other tweets (NoAPO). This dataset is described as comparatively balanced.
  • Task 8: Segregating tweets containing self-reports of breast cancer (S) from other tweets (NR). This dataset is also imbalanced (approx. 1:3 ratio).

The paper provides detailed characteristics of these datasets, including the number of instances in training, validation, and test sets for each class.

Key Insights and Ideas

  • Data Augmentation is Key for Imbalance: The most striking insight is the consistent superiority of data augmentation, especially for highly imbalanced datasets. It appears to provide the necessary variations to help models generalize better than simple oversampling or undersampling.
  • Augmentation Benefits Even Balanced Datasets: Interestingly, data augmentation also showed benefits on the comparatively balanced Task 4, suggesting its utility extends beyond extreme imbalance.
  • Undersampling Leads to Underfitting, Oversampling to Overfitting: For Task 1a, undersampling resulted in underfitting, while oversampling led to overfitting. This highlights the delicate balance needed when dealing with sparse data.
  • Domain-Specific Models Can Help: While general-purpose transformers like RoBERTa performed well, the use of BioBERT for Task 8 (a health-specific task) suggests that domain-adapted models can be advantageous.
  • Ensembling Not Always Superior: The authors found that ensembling multiple transformer models did not yield better results than using a single, well-tuned model, likely because errors were often common across models.

Key Results

  • Task 1a: RoBERTa with data augmentation achieved the best performance, with an F1 score of 0.8433 on the augmented dataset, significantly outperforming original, undersampled, and oversampled versions.
  • Task 4: RoBERTa also performed best on Task 4. While augmentation didn't yield a higher F1 score than the original dataset, it still performed competitively.
  • Task 8: BioBERT with data augmentation achieved an F1 score of 0.8947 on the validation set and 0.85 on the test set, demonstrating strong performance on this imbalanced task.

Limitations and Future Work

  • Limitations: The authors acknowledge that while data augmentation improved performance, the F1 scores for Task 1a and Task 8, despite improvements, are still relatively low, indicating the inherent difficulty of these imbalanced tasks. The paper also notes that undersampling led to underfitting and oversampling to overfitting on Task 1a, suggesting that finding the optimal balance is challenging.
  • Future Work: The authors plan to conduct further experiments to explore additional strategies for improving model performance on these challenging datasets. This could involve investigating more advanced data augmentation techniques or exploring different model architectures and training paradigms.

Summary updated

BibTeX
@inproceedings{pimpalkhute2021iiitn,
  author = {Varad Pimpalkhute and Prajwal Nakhate and Tausif Diwan},
  title = {{IIITN NLP at SMM4H 2021 Tasks: Transformer Models for Classification on Health-Related Imbalanced Twitter Datasets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure