UACH-INAOE at SMM4H: a BERT based approach for classification of COVID-19 Twitter posts

Alberto Valdes, Jesus Lopez, Manuel MontesNAACL 2021

TL;DR

This paper presents a BERT-based approach for classifying COVID-19-related tweets, focusing on identifying self-reporting cases and symptom mentions. The study highlights that a BERT model fine-tuned on a smaller, domain-specific COVID-19 corpus (CT-BERT) can outperform a model trained on a much larger general corpus, achieving top scores in the SMM4H 2021 shared task.

What Problem Does This Paper Address?

The COVID-19 pandemic generated an unprecedented volume of information on social media, particularly Twitter. Effectively sifting through this data to identify crucial public health signals, such as self-reported cases or symptom discussions, is a significant challenge for Natural Language Processing (NLP). The SMM4H (Social Media Mining for Health Applications) workshop aims to develop methods for extracting such insights from social media. This paper specifically tackles two classification tasks within SMM4H 2021: identifying tweets where users self-report potential COVID-19 cases and classifying tweets that mention COVID-19 symptoms. The core question is whether a model trained on a specialized dataset can outperform one trained on a general, albeit larger, dataset for these health-related classification tasks.

Methods

The researchers employed transformer-based models, specifically BERT (Bidirectional Encoder Representations from Transformers), a state-of-the-art architecture for NLP tasks. They implemented two variations:

  • BERT-Large (uncased): A model pretrained on a massive general-domain corpus (Wikipedia).
  • CT-BERT: A model based on BERT-Large but specifically pretrained on a smaller corpus of COVID-19-related tweets.

The core idea was to compare the performance of these two models, hypothesizing that the domain-specific CT-BERT would perform better. The models were fine-tuned using a random search approach with the Optuna framework to optimize hyperparameters like weight decay, learning rate, and epochs. The general architecture involved taking the hidden state of the [CLS] token from the final BERT layer as the input to a fully-connected classifier with a softmax activation function for class prediction.

Data preprocessing involved standard steps like replacing user mentions with a "user" token, handling multiple occurrences of "user" and "url" tokens, converting emojis to text aliases, standardizing text to ASCII, and lowercasing all text.

Datasets

The study utilized datasets provided by the SMM4H 2021 shared task for two specific tasks:

  • Task 5: Classification of tweets self-reporting potential cases of COVID-19. The dataset included training, validation, and test sets with a significant class imbalance (approximately 1:5 for "1" and "0" classes).
  • Task 6: Classification of COVID-19 tweets containing symptoms into three categories: self-reports, non-personal reports, and literature/news mentions. This dataset also comprised training, validation, and test sets.

The paper does not introduce new benchmarks but rather uses the official datasets for the SMM4H 2021 tasks.

Key Insights and Ideas

  • Domain-Specific Pretraining Matters: The most significant insight is that pretraining a BERT model on a smaller, highly relevant domain corpus (COVID-19 tweets) leads to superior performance compared to using a model pretrained on a much larger, general-domain corpus. This suggests that specialized knowledge embedded during pretraining is crucial for nuanced classification tasks in specific domains.
  • Data Preprocessing Impact: The authors found that data preprocessing steps had a greater impact on model performance than hyperparameter tuning. This emphasizes the importance of meticulous data cleaning and preparation in NLP tasks, especially with noisy social media data.
  • Class Imbalance Challenges: For Task 5, the inherent class imbalance was a challenge. While attempts to balance the classes were made, they did not improve results, indicating that the models could learn effectively from the imbalanced data with appropriate fine-tuning.
  • Error Analysis Reveals Nuances: Analyzing misclassifications provided insights into the model's limitations. For Task 5, tweets with symptom-related keywords were sometimes misclassified, and for Task 6, distinguishing between literature/news mentions and non-personal reports proved difficult, often hinging on the perceived audience size or impersonal style of the tweet.

Key Results

  • Task 5: CT-BERT achieved a macro F1 score of 0.77 on the test set, outperforming the general BERT model (0.68) and the median submission score (0.74).
  • Task 6: CT-BERT achieved a macro F1 score of 0.95 on the test set, while the general BERT model scored 0.94. CT-BERT achieved the highest score among all participants for this task.
  • Both models achieved very high scores on the validation sets for both tasks, with CT-BERT consistently performing slightly better or on par with BERT.

Limitations and Future Work

  • BioBERT Comparison: The researchers intended to compare their models with BioBERT (a BERT model pretrained on a medical corpus) but could not due to time constraints. This remains a potential avenue for future work.
  • Further Domain-Specific Models: The authors suggest exploring the performance of other domain-specific models for predicting COVID-19 infection risk.
  • Wide & Deep Learning: Based on error analysis, they plan to investigate wide and deep learning techniques to improve generalization by incorporating a wider range of features.
  • Model Interpretability: While error analysis provided some qualitative insights, deeper interpretability methods could further illuminate why certain misclassifications occur.

Summary updated

BibTeX
@inproceedings{valdes2021uach,
  author = {Alberto Valdes and Jesus Lopez and Manuel Montes},
  title = {{UACH-INAOE at SMM4H: a BERT based approach for classification of COVID-19 Twitter posts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure