Classification of Tweets Self-reporting Adverse Pregnancy Outcomes and Potential COVID-19 Cases Using RoBERTa Transformers

Lung-Hao Lee, Man-Chen Hung, Chien-Huan Lu, Chang-Hao Chen, Po-Lei Lee, Kuo-Kai ShyuNAACL 2021

TL;DR

This paper presents a RoBERTa-based system for classifying tweets related to adverse pregnancy outcomes and potential COVID-19 cases. The model achieved strong performance, slightly exceeding the median scores for both tasks, demonstrating the effectiveness of fine-tuning pre-trained language models for specialized health-related social media analysis.

What Problem Does This Paper Address?

The authors tackle the challenge of extracting critical health information from the vast and noisy landscape of social media. Specifically, they focus on two crucial areas:

  1. Adverse Pregnancy Outcomes: Automatically identifying tweets where individuals self-report personal experiences of adverse pregnancy events like miscarriage, stillbirth, preterm birth, low birthweight, or neonatal intensive care. This is vital for public health surveillance and understanding the prevalence and impact of these events.
  2. Potential COVID-19 Cases: Distinguishing tweets that indicate a user or their household might have COVID-19 (e.g., denied testing, symptomatic, exposed, high-risk exposure) from general COVID-19 discussions. This is crucial for real-time monitoring of the pandemic's spread and impact.

The core problem is the need for accurate and scalable methods to process unstructured text data from social media for health-related insights, which traditional methods struggle with due to the informal language, abbreviations, and context-dependent nature of tweets.

Methods

The NCUEE-NLP system leverages the power of pre-trained transformer models, specifically RoBERTa (Robustly Optimized BERT Pretraining Approach). The approach involves two main stages:

  1. RoBERTa Transformers: The core of the system uses a pre-trained RoBERTa-Large model. This model has been trained on a massive corpus of text and possesses a strong understanding of language nuances.
  2. Fine-tuning:
    • Language Model Fine-tuning: For Task 5 (COVID-19), the language model itself was fine-tuned using a combination of the Task 5 dataset and additional tweets from Task 6 (which involved a three-class COVID-19 classification). This was done because COVID-19 related tweets are relatively rare, and leveraging more data for LM fine-tuning can improve representation.
  3. Classifier Fine-tuning: After the language model is adapted, a downstream classifier is fine-tuned. For both Task 4 and Task 5, this classifier was trained using the respective task's training set that contained tweets with corresponding labels.

The system architecture is depicted in Figure 1, showing the input tweets being processed by the RoBERTa model, followed by a classifier to predict the class.

Datasets

The study utilizes datasets provided by the SMM4H 2021 shared tasks:

  • Task 4 (Adverse Pregnancy Outcomes):
    • Training Set: 5,514 tweets (2,484 positive, 3,030 negative)
    • Validation Set: 973 tweets (438 positive, 535 negative)
    • Test Set: 10,000 tweets
  • Task 5 (Potential COVID-19 Cases):
    • Training Set: 6,465 tweets (1,026 positive, 5,439 negative)
    • Validation Set: 716 tweets (122 positive, 594 negative)
    • Test Set: 10,000 tweets
  • Additional Data for Task 5 LM Fine-tuning: 16,067 tweets from Task 6 (three-class COVID-19 classification).

The paper also mentions that tweets were pre-processed to convert emojis into Unicode codes.

Key Insights and Ideas

  • RoBERTa's Efficacy: The study reinforces the effectiveness of RoBERTa transformers for health-related tweet classification, building on previous successes in SMM4H tasks.
  • Fine-tuning is Key: The performance boost observed from fine-tuning the language model and classifier highlights the importance of adapting general-purpose models to specific, domain-relevant tasks.
  • Task 5 Data Scarcity: The authors acknowledge the rarity of COVID-19 positive tweets and their strategy to augment the language model fine-tuning with data from a related task (Task 6) is a practical approach to mitigate this challenge.
  • Consistent Performance: The system achieved consistent results across both tasks, suggesting a robust methodology.

Key Results

The primary evaluation metric for both tasks was the F1-score of the positive class.

  • Task 4 (Adverse Pregnancy Outcomes): The system achieved a best F1-score of 0.93, which slightly exceeded the median score of 0.925 from all submissions.
  • Task 5 (Potential COVID-19 Cases): The system achieved a best F1-score of 0.75, which slightly exceeded the median score of 0.745 from all submissions.

The paper notes that the performance boost from fine-tuning the language model was evident, and their best results slightly surpassed the respective median scores of all submissions by 0.005.

Limitations and Future Work

  • Limitations:
    • The paper doesn't explicitly detail limitations beyond the data scarcity for Task 5.
  • The performance for Task 5 (0.75 F1-score) is notably lower than Task 4 (0.93 F1-score), suggesting that distinguishing potential COVID-19 cases from general discussions is a more challenging task, likely due to the ambiguity and broader scope of "potential cases."
  • Future Work:
    • Not explicitly discussed in the paper, but the success of fine-tuning suggests further exploration of different transformer architectures or more advanced fine-tuning strategies could yield improvements.
  • Investigating methods to better handle the class imbalance and ambiguity in Task 5 would be a logical next step.

Summary updated

BibTeX
@inproceedings{lee2021classification,
  author = {Lung-Hao Lee and Man-Chen Hung and Chien-Huan Lu and Chang-Hao Chen and Po-Lei Lee and Kuo-Kai Shyu},
  title = {{Classification of Tweets Self-reporting Adverse Pregnancy Outcomes and Potential COVID-19 Cases Using RoBERTa Transformers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure