LT4SG@SMM4H’24: Tweets Classification for Digital Epidemiology of Childhood Health Outcomes Using Pre-Trained Language Models

Dasun Athukoralage, Thushari Atapattu, Menasha Thilakaratne, Katrina FalknerACL 2024

TL;DR

This paper tackles the challenge of automatically identifying tweets reporting childhood health disorders like ADHD and autism. They achieved state-of-the-art results by fine-tuning pre-trained language models, particularly excelling with an ensemble of BERTweet-large models, demonstrating the power of social media data for digital epidemiology.

What Problem Does This Paper Address?

Identifying and tracking childhood health conditions is crucial for public health, but traditional methods can be slow and resource-intensive. Social media platforms like Twitter offer a vast, real-time source of information, but extracting meaningful health insights from noisy, informal text is a significant NLP challenge. Specifically, this paper addresses the gap in identifying tweets where users report their children having specific medical disorders, a nuanced task that requires distinguishing between personal health reports and those concerning dependents. This is vital for early detection and understanding the prevalence of conditions like ADHD, ASD, delayed speech, and asthma in young populations.

Methods

The authors explored two main approaches using pre-trained language models (LLMs):

  1. Single Model Fine-tuning: They fine-tuned a single RoBERTa-large model on the task.
  2. Ensemble Learning: They fine-tuned three BERTweet-large models and then combined their predictions using a hard majority voting mechanism.

For hyperparameter optimization, they leveraged HuggingFace's Trainer API with Ray Tune's BasicVariantGenerator and a FIFO scheduler, enabling efficient exploration of the search space. Training was conducted on Google Colab Pro+ with NVIDIA A100 GPUs, utilizing FP-16 mixed precision for faster training and larger batch sizes. They experimented with BioLinkBERT-large, RoBERTa-large, and BERTweet-large, ultimately focusing on RoBERTa-large and BERTweet-large due to BioLinkBERT-large's comparatively lower performance on the validation set.

Datasets

The study utilized datasets provided for the SMM4H'24 Shared Task 5. These included:

  • Training Dataset: 7398 tweets.
  • Validation Dataset: 389 tweets.
  • Test Dataset: 1947 tweets.

All datasets were composed of tweets from users who had reported their pregnancy on Twitter, either mentioning a child with a disorder (positive class) or merely referring to a disorder (negative class). The test dataset was unlabeled, requiring the models to perform inference on unseen data.

Key Insights and Ideas

  • Ensembling for Robustness: The paper highlights that fine-tuning LLMs on small datasets can lead to performance instability. Ensembling multiple models (especially BERTweet-large) significantly improved robustness and generalization, leading to better performance on unseen test data.
  • BERTweet-large's Superiority: Despite RoBERTa-large performing well, the BERTweet-large ensemble consistently outperformed it, particularly on the test set. This suggests that models specifically pre-trained on Twitter data (like BERTweet-large) are better suited for social media text analysis, even when dealing with specialized domains like health.
  • Validation vs. Test Performance: A crucial insight is that models performing equally well on validation data can diverge significantly on test data. This underscores the importance of rigorous evaluation on unseen data and the potential pitfalls of overfitting to the validation set.
  • Hyperparameter Tuning Matters: The authors emphasize that thorough hyperparameter tuning, as performed with Ray Tune, is critical for achieving optimal performance, especially when compared to previous work that might have used less extensive tuning.

Key Results

  • The best-performing system, a BERTweet-large ensemble, achieved an F1-score of 0.938 on the test data.
  • This performance outperformed the benchmark classifier by 1.18%.
  • On the validation data, both the RoBERTa-large best-run and the BERTweet-large ensemble achieved an identical F1-score of 0.945055. However, the BERTweet-large ensemble showed better performance on the test data.
  • The BERTweet-large ensemble also outperformed the mean and median performance of all teams' submissions on the test data.

Limitations and Future Work

  • Model Diversity: While the authors combined multiple runs of BERTweet-large, they suggest that investigating the impact of adding more BERTweet-large iterations to the ensemble could further improve performance.
  • Generalizability: The study focused on specific childhood disorders. Future work could explore the system's performance on a wider range of health conditions and across different social media platforms.
  • Data Size and Noise: The performance of LLMs on social media data is inherently limited by the size and inherent noise of the datasets. Further research could explore techniques to better handle informal language, abbreviations, and misspellings common in tweets.

Summary updated

BibTeX
@inproceedings{athukoralage2024lt,
  author = {Dasun Athukoralage and Thushari Atapattu and Menasha Thilakaratne and Katrina Falkner},
  title = {{LT4SG@SMM4H’24: Tweets Classification for Digital Epidemiology of Childhood Health Outcomes Using Pre-Trained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure