BERT Goes Brrr: A Venture Towards the Lesser Error in Classifying Medical Self-Reporters on Twitter

Alham Fikri Aji, Made Nindyatama Nityasya, Haryo Akbarianto Wibowo, Radityo Eko Prasojo, Tirana FatyanosaNAACL 2021

TL;DR

This paper presents Kata.ai's winning submission for the SMM4H 2021 shared task, achieving top performance in classifying adverse drug effects and COVID-19 self-reports on Twitter. Their approach leverages a domain-specific BERT model, enhanced through meticulous data cleaning, augmentation, hyperparameter optimization, and model ensembling.

What Problem Does This Paper Address?

Social media platforms like Twitter have become invaluable sources for monitoring public health trends, from flu outbreaks to adverse drug reactions and the spread of diseases like COVID-19. However, analyzing this data presents a significant challenge due to its inherent noisiness, characterized by linguistic errors, typos, and informal language. Traditional methods for extracting health-related information from such data are often costly and time-consuming. This paper tackles the problem of accurately classifying medical self-reports and adverse drug effects from noisy Twitter data, aiming to improve the efficiency and effectiveness of public health surveillance.

Methods

The core of Kata.ai's system is the BERT (Bidirectional Encoder Representations from Transformers) model. They began with a standard BERT-base model and progressively enhanced its performance through several key techniques:

  • Domain-Specific Pre-training: Recognizing that Twitter data has unique characteristics, they explored pre-trained BERT models specifically trained on relevant domains. They found that BERTweet-Covid19, a model pre-trained on English tweets related to COVID-19, yielded the best results, even for non-COVID-19 related tasks like adverse drug effect classification.
  • Data Cleaning: To mitigate the impact of noisy text, they implemented several cleaning strategies. The most effective involved masking Twitter handles, URLs, email addresses, phone numbers, and money symbols. They also performed routine HTML tag cleanup and hashtag expansion.
  • Data Augmentation: To address class imbalance in the training data, they experimented with oversampling (duplicating minority class samples) and class weighting. They also employed data augmentation through round-trip translation (English to another language and back to English) to create paraphrased training examples.
  • Hyperparameter Optimization (HPO): They utilized a Tree-structured Parzen Estimator (TPE) approach via the Hyperopt package to automatically search for optimal hyperparameters, including batch size, learning rate, optimizer, epsilon, and weight decay.
  • Model Ensembling: Finally, they combined predictions from multiple well-performing models. They explored both full ensembles and subset ensembles, finding that a carefully selected subset of models often outperformed a larger, more comprehensive ensemble.

Datasets

The study utilized datasets from the Social Media Mining for Health (SMM4H) 2021 shared task. These datasets consist of tweets related to:

  • Task 1a: Classification of adverse drug effects (ADE).
  • Task 5: Classification of tweets self-reporting potential cases of COVID-19.
  • Task 6: Classification of COVID-19 tweets containing symptoms.

The paper provides examples of these tweet classifications in Table 1 and a distribution of the training and validation data in Table 2.

Key Insights and Ideas

  • Domain-Specific BERT is Crucial: The study strongly suggests that using BERT models pre-trained on domain-specific data (like BERTweet-Covid19) significantly outperforms general-purpose BERT models, even for tasks not directly related to the pre-training domain.
  • Data Cleaning Pays Off: Simple yet effective data cleaning techniques, such as masking common noisy elements like URLs and handles, can substantially improve model performance.
  • Ensembling is Powerful, but Nuanced: While model ensembling is a well-established technique for boosting performance, the paper highlights that simply combining all available models isn't always optimal. Selecting a carefully curated subset of models can yield better results, suggesting that some models might introduce "noise" into the ensemble.
  • Data Augmentation's Situational Impact: Data augmentation and hyperparameter optimization showed significant improvements for some tasks (Task 1a and Task 5) but had less impact on others (Task 6), indicating that their effectiveness can be task-dependent.

Key Results

Kata.ai achieved the first rank in two of the three SMM4H 2021 tasks:

  • Task 1a (Adverse Drug Effects): Achieved a F1-Score of 0.54 on the test set, outperforming the median.
  • Task 5 (COVID-19 Self-Report): Achieved a F1-Score of 0.79 on the test set, outperforming the median.
  • Task 6 (COVID-19 Symptoms): Achieved a F1-Score of 0.94 on the test set, also outperforming the median.

The paper details incremental performance improvements at each stage of their methodology (Table 3), showing that the final ensembled system achieved a F1-Score of 87.80 for Task 1a and 86.27 for Task 5 on the validation set.

Limitations and Future Work

  • Test Set Analysis Limitations: The authors note that further in-depth analysis on the test set was not feasible due to the absence of provided labels.
  • HPO Bias and Overfitting: Hyperparameter optimization, while beneficial, can lead to overfitting on the validation set, as observed for Task 1a.
  • Ensemble Search Complexity: Exhaustively searching for the best ensemble combination (power set) is computationally expensive and scales exponentially with the number of models. Optimizing this search process is identified as future work.
  • Situational Augmentation: The paper suggests that data augmentation and HPO are more situational, implying that their application and effectiveness might need careful consideration for different tasks.

The authors propose that their developed training pipeline, combining domain-specific BERT, data cleaning, augmentation, HPO, and ensembling, can be a valuable approach for improving performance on general text classification tasks.

Summary updated

BibTeX
@inproceedings{aji2021bert,
  author = {Alham Fikri Aji and Made Nindyatama Nityasya and Haryo Akbarianto Wibowo and Radityo Eko Prasojo and Tirana Fatyanosa},
  title = {{BERT Goes Brrr: A Venture Towards the Lesser Error in Classifying Medical Self-Reporters on Twitter}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure