SMM4H 2024: 5 Fold Cross Validation for Classification of tweets reporting children’s disorders

Lipika Dey, B Naik, Oppangi Poojita, Kovidh PothireddiACL 2024

TL;DR

This paper tackles the challenge of automatically identifying tweets that report children's medical disorders from those that merely mention them. Using a RoBERTa-Large model with 5-fold cross-validation and careful data preprocessing, the authors achieved a respectable F1-score of 0.823 on the test set, highlighting the potential of NLP for public health monitoring.

What Problem Does This Paper Address?

The Social Media Mining for Health (SMM4H) initiative aims to leverage the vast amount of information shared on social media for public health insights. Specifically, Task 05 of SMM4H 2024 focuses on a crucial distinction: differentiating between tweets that report a child's medical disorder and those that simply mention a disorder. This distinction is vital for accurate health surveillance, as a reported disorder implies a direct observation or experience, whereas a mention could be a casual reference. The challenge lies in the noisy and informal nature of social media text, making it difficult for automated systems to reliably make this classification.

Methods

The authors employed a robust approach centered around the RoBERTa-Large model, a powerful transformer-based language model known for its strong performance in natural language understanding tasks. To ensure reliable evaluation and mitigate the risk of overfitting, they implemented a 5-fold cross-validation strategy.

A significant portion of their effort was dedicated to data preprocessing. They utilized the neattext Python library to systematically remove common social media noise elements like usernames, hashtags, links, emojis, and dates. Recognizing that neattext's emoji removal wasn't exhaustive, they developed a custom Python function to ensure complete emoji removal. This meticulous cleaning aimed to allow the model to focus on the core textual content.

To address class imbalance (where tweets mentioning disorders significantly outnumbered those reporting them), they applied RandomUnderSampler. This technique reduced the majority class to match the minority class, creating a balanced dataset for training. Finally, they merged the training and validation datasets to further augment the data available for model training, aiming to improve generalization.

Datasets

The study utilized the dataset provided for the SMM4H 2024 Task 05. This dataset was divided into:

  • Training set: 7,398 tweets (5,118 labeled 0 - mention, 2,280 labeled 1 - report).
  • Validation set: 389 tweets (254 labeled 0 - mention, 135 labeled 1 - report).
  • Test set: Approximately 10,000 unlabeled tweets.

After preprocessing and merging, the combined dataset for training and validation contained 7,787 tweets. Following resampling, the dataset was balanced to 4,830 samples (2,415 for each class).

Key Insights and Ideas

  • The importance of meticulous preprocessing: The authors found that aggressive cleaning of social media noise (usernames, hashtags, emojis, URLs) significantly improved the model's performance by allowing it to focus on the core textual content.
  • Class imbalance is a critical challenge: The significant imbalance between "mention" and "report" tweets necessitated resampling techniques to prevent the model from being biased towards the majority class.
  • RoBERTa's effectiveness for nuanced classification: The choice of RoBERTa-Large demonstrates its capability in understanding the subtle differences in language used to report versus mention medical conditions.
  • Cross-validation is key for robust evaluation: The use of 5-fold cross-validation provided a more reliable assessment of the model's performance across different data splits.

Key Results

The model achieved the following performance metrics:

  • Validation Dataset: F1 Score: 0.886, Precision: 0.831, Recall: 0.948
  • Test Dataset: F1 Score: 0.823, Precision: 0.721, Recall: 0.959

Notably, the model consistently achieved a high recall score across all datasets, indicating its strength in identifying most of the relevant cases.

Limitations and Future Work

The authors acknowledge that their methodology did not match the baseline performance, attributing this to several factors:

  • Suboptimal learning rate: The learning rate was not effectively optimized, potentially hindering the model's convergence.
  • Lack of text lowercasing: Not converting text to lowercase might have limited vocabulary standardization, impacting training.
  • Tokenization length: Using a tokenization length shorter than the recommended maximum for RoBERTa (512 tokens) might have prevented the model from capturing the full contextual meaning of some tweets.

Future work could explore optimizing these parameters, implementing lowercasing, and experimenting with longer tokenization lengths to further improve precision. The high recall suggests that while the model is good at finding relevant tweets, further refinement is needed to reduce false positives.

Summary updated

BibTeX
@inproceedings{dey2024smm,
  author = {Lipika Dey and B Naik and Oppangi Poojita and Kovidh Pothireddi},
  title = {{SMM4H 2024: 5 Fold Cross Validation for Classification of tweets reporting children’s disorders}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure