OCHADAI at SMM4H-2021 Task 5: Classifying self-reporting tweets on potential cases of COVID-19 by ensembling pre-trained language models

Ying Luo, Lis Pereira, Kobayashi IchiroNAACL 2021

TL;DR

This paper presents a system for automatically identifying tweets that self-report potential COVID-19 cases. The researchers achieved this by ensembling three pre-trained language models, demonstrating that combining specialized models can outperform individual ones for this specific task.

What Problem Does This Paper Address?

The COVID-19 pandemic generated a massive amount of information and personal expression on social media, particularly Twitter. Distinguishing between tweets that genuinely report potential COVID-19 symptoms or cases and those that don't is crucial for public health monitoring and understanding public sentiment. This task is challenging due to the informal language, abbreviations, and potential for misinformation present in tweets. The SMM4H-2021 Task 5 specifically aimed to tackle this by classifying English tweets related to self-reporting potential COVID-19 cases.

Methods

The Ochadai team employed an ensemble approach, combining the strengths of three pre-trained language models:

  • COVID-Twitter-BERT (CT-BERT): A BERT-LARGE model specifically trained on a large corpus of COVID-19-related Twitter messages.
  • RoBERTa-LARGE: A general-purpose, robustly optimized BERT pre-training approach.
  • Twitter-RoBERTa: A RoBERTa-BASE model pre-trained on a broad corpus of English tweets, potentially including COVID-19 content.

The text pre-processing involved standard techniques like lowercasing, replacing user tags with "@USER", URLs with "URL", converting emoticons to their ASCII representations, and removing unicode symbols and accented characters.

The models were fine-tuned using 5-fold cross-validation. Crucially, instead of simply taking the majority vote of predicted labels (max-voting), the researchers averaged the prediction probabilities obtained after the softmax layer. This "average probability" method proved to be more effective. They also experimented with ensembling these models together.

Datasets

The paper focuses on the dataset provided for the SMM4H-2021 Task 5, which consists of English tweets for classifying self-reporting of potential COVID-19 cases. The specific details of the dataset's creation or size are not elaborated upon beyond its purpose for the task.

Key Insights and Ideas

  • Ensembling for Specialization: Combining models that have been pre-trained on different, but relevant, corpora (like CT-BERT for COVID-19 tweets and general Twitter-trained models) can lead to improved performance on a specialized downstream task.
  • Averaging Probabilities > Max Voting: For this task, averaging the predicted probabilities from the models after softmax yielded better results than simply taking the majority vote of the predicted labels. This suggests that the nuanced probability distributions from the models carry more useful information than just their final class predictions.
  • Cross-Validation Benefits: Performing cross-validation and averaging results across folds consistently improved performance compared to single-fold training, highlighting the robustness gained from this technique.

Key Results

The Ochadai system achieved an F1-score of 76% on the test set during the evaluation phase and 77.5% in the post-evaluation phase. Individual models showed varying performance, with CT-BERT achieving an F1-score of 79% on the validation set, Twitter-RoBERTa-base reaching up to 93%, and RoBERTa-large also reaching 93% on the validation set. The ensembled models, particularly "Ensemble 2" (which achieved 77.5% on the test set), demonstrated superior performance.

Limitations and Future Work

The authors acknowledge that their best performing model achieved an F1-score of 77.5%. They plan to further improve their model by exploring other pre-trained language models and different ensemble techniques. The paper does not explicitly detail limitations beyond the scope for improvement.

Summary updated

BibTeX
@inproceedings{luo2021ochadai,
  author = {Ying Luo and Lis Pereira and Kobayashi Ichiro},
  title = {{OCHADAI at SMM4H-2021 Task 5: Classifying self-reporting tweets on potential cases of COVID-19 by ensembling pre-trained language models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure