Approaching SMM4H with auto-regressive language models and back-translation

Joseph Cornelius, Tilia Ellendorff, Fabio RinaldiNAACL 2021

TL;DR

This paper tackles the challenge of identifying potential COVID-19 cases from noisy social media text by employing XLNet, a powerful auto-regressive language model, and back-translation for data augmentation. Their approach successfully improved performance by balancing the dataset and leveraging advanced NLP techniques.

What Problem Does This Paper Address?

The Social Media Mining for Health Applications (SMM4H) shared task aims to extract valuable health information from the vast and often messy world of social media. Specifically, this paper focuses on Task 5: classifying tweets that self-report potential COVID-19 cases. The core challenge lies in the inherent noisiness of social media text, which includes misspellings, abbreviations, emojis, and non-standard language. Furthermore, the dataset is highly imbalanced, with significantly more negative (non-COVID-19 related) tweets than positive ones, making it harder for models to learn to identify the minority class.

Methods

The authors employed a two-pronged approach:

  • Preprocessing: To combat the noisy nature of social media text, they implemented several preprocessing steps: stripping hash symbols, removing most punctuation (keeping only ".,!?"), eliminating URLs, removing emojis, and converting all text to lowercase.
  • XLNet for Classification: They utilized XLNet, a pre-trained auto-regressive transformer language model known for its strong performance on various NLP tasks. Unlike BERT, XLNet's pre-training objective involves predicting masked tokens conditioned on all permutations of tokens within a sentence, allowing it to learn bidirectional context more effectively. They used the "XLNet-large-cased" model, fine-tuning it with a consistent set of hyperparameters, including a maximum tweet length of 256 characters, a batch size of four, and a learning rate of 3e-6.
  • Back-Translation for Data Augmentation: To address the dataset imbalance and increase the number of positive samples, they used back-translation. This technique involves translating a tweet into a different language (e.g., English to Spanish) and then translating it back to the original language (Spanish to English). This process generates paraphrased versions of the original tweets, effectively augmenting the training data. They leveraged the Google Translation API and used ten different languages as pivot languages to increase the diversity of the generated paraphrases.

Datasets

The primary dataset used is from the SMM4H 2021 shared task, specifically for Task 5 (COVID-19 case classification). It consists of:

  • Training Set: 6,465 samples (1,026 positive, 5,439 negative).
  • Validation Set: 716 samples (122 positive, 594 negative).
  • Test Set: 10,000 samples for evaluation.

The dataset is characterized by its imbalance, with approximately five times more negative tweets than positive ones.

Key Insights and Ideas

  • Back-translation is effective for imbalance: The study demonstrates that back-translation can significantly improve performance on imbalanced datasets by generating synthetic positive examples.
  • Optimal augmentation is key: Counter-intuitively, simply increasing the number of back-translated samples (higher k values) did not consistently lead to better results. The F-score peaked at k=1, suggesting that there's an optimal balance for augmentation.
  • XLNet is a strong baseline: The XLNet model, even without extensive augmentation, provided a robust baseline for this challenging task.
  • Parameter merging's limitations: While parameter merging (averaging weights from models trained on different data folds) is a known technique for improving robustness, it didn't yield significant improvements in this specific case, potentially due to limited GPU resources restricting the number of cross-validation folds.

Key Results

The paper reports the following key results:

  • The XLNet model with back-translation (k=1) achieved the best performance, with an F-score of 0.72 on the official test set and 0.84 on the validation set.
  • The XLNet model without any back-translation achieved an F-score of 0.69 on the official test set and 0.82 on the unofficial test set.
  • The parameter-merged XLNet system achieved an F-score of 0.80 on the validation dataset, which was not as high as the best single XLNet model with k=1 back-translation.

Limitations and Future Work

  • Limited GPU resources: The authors acknowledge that limited GPU computing power restricted the number of folds used for parameter merging, potentially hindering its effectiveness.
  • Optimal augmentation: The study highlights that the number of additional generated samples through back-translation plays a decisive role, and further research could explore more sophisticated methods for determining the optimal amount of augmented data.
  • Parameter Merging: Future work could investigate parameter merging with a larger number of folds if computational resources allow, to see if it can yield better results.

Summary updated

BibTeX
@inproceedings{cornelius2021approaching,
  author = {Joseph Cornelius and Tilia Ellendorff and Fabio Rinaldi},
  title = {{Approaching SMM4H with auto-regressive language models and back-translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure