Fine-tuning BERT to classify COVID19 tweets containing symptoms

Rajarshi Roychoudhury, Sudip NaskarNAACL 2021

TL;DR

This paper tackles the challenge of accurately classifying COVID-19-related tweets into self-reports, non-personal reports, or literature/news mentions. They achieve this by fine-tuning a BERT model with custom preprocessing, demonstrating a strong F1 score of 93% on a challenging real-world dataset.

What Problem Does This Paper Address?

Identifying and understanding public health information shared on social media, particularly during a pandemic like COVID-19, is crucial. Twitter, with its vast stream of user-generated content, offers a rich source of real-time insights. However, analyzing this data presents significant hurdles. Firstly, distinguishing genuine personal experiences from discussions about news articles or scientific findings is difficult. Secondly, users often report symptoms experienced by others (e.g., family members), making it hard to differentiate between self-reporting and non-personal observations. Furthermore, the informal nature of tweets—characterized by slang, misspellings, contractions, and emojis—adds another layer of complexity to natural language processing tasks. This paper aims to build a robust system for classifying these nuanced COVID-19 symptom tweets.

Methods

The authors employ a fine-tuning approach using a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model, specifically a "Small_BERT" variant. The core of their method involves:

  • Custom Preprocessing: Before feeding tweets into BERT, they perform several preprocessing steps. These include expanding contractions, removing punctuation, converting text to lowercase, removing emojis, and stripping hashtags, mentions (words starting with '@'), and URLs.
  • BERT Encoding: The preprocessed tweets are then passed through the Small_BERT encoder to generate vector representations.
  • Classification Layer: The pooled outputs from the BERT encoder are fed into a fully connected dense layer, followed by a dropout layer (for regularization) and a final dense layer with a softmax activation function to produce the class probabilities.
  • Fine-tuning: The entire model is then fine-tuned on the specific task of classifying COVID-19 symptom tweets. They experimented with learning rates and found optimal results with a learning rate of 3e-5 and the Adam optimizer, training for 9 epochs.

Datasets

The study utilizes data from Task 6 of the SMM4H (Social Media Mining for Health Applications) Shared Task 2021. This task involved classifying tweets into three categories:

  1. Self-reports
  2. Non-personal reports
  3. Literature/news mentions

The dataset was split into training, validation, and test sets. Table 1 in the paper provides statistics on the distribution of these classes across the splits.

Key Insights and Ideas

  • The Power of Contextual Embeddings: The paper highlights the importance of contextual representations, which BERT excels at, for understanding the nuances of social media text. Unlike older unidirectional models, BERT's ability to consider both left and right context is crucial for accurately interpreting symptom descriptions.
  • Preprocessing is a Double-Edged Sword: While aggressive preprocessing (like removing hashtags and mentions) can improve overall performance by simplifying the input, it can also lead to misclassifications. The authors found that for certain tweets, retaining mentions was necessary for proper classification, suggesting a need for more sophisticated preprocessing strategies that balance noise reduction with context preservation.
  • Subtlety in Self-Reporting: The challenge of distinguishing true self-reports from non-personal reports is significant. The authors observed that tweets containing ambiguous language or discussing symptoms experienced by others can easily be misclassified, underscoring the difficulty of this task.

Key Results

The model achieved strong performance on the SMM4H Shared Task 2021:

  • Validation Set: An F1 score of 0.968.
  • Test Set: An F1 score of 0.9325.

These results indicate a highly effective system for classifying COVID-19 symptom tweets.

Limitations and Future Work

  • Preprocessing Trade-offs: The authors acknowledge that their preprocessing strategy, while effective overall, led to misclassifications in specific cases. They suggest that future work should explore methods to retain relevant contextual information, such as mentions, for improved accuracy on challenging tweets.
  • Ambiguous Cases: The paper points out that certain tweets are inherently ambiguous, making perfect classification difficult. Future research could focus on developing models that can better handle such ambiguity or provide confidence scores for their predictions.
  • Generalizability: While the model performed well on the SMM4H dataset, its generalizability to other health-related topics or different social media platforms might require further investigation.

Summary updated

BibTeX
@inproceedings{roychoudhury2021fine,
  author = {Rajarshi Roychoudhury and Sudip Naskar},
  title = {{Fine-tuning BERT to classify COVID19 tweets containing symptoms}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure