Golden_Duck at #SMM4H 2024: A Transformer-based Approach to Social Media Text Classification

Md Ayon Mia, Mahshar Yahan, Hasan Murad, Muhammad KhanACL 2024

TL;DR

This paper presents a transformer-based approach for classifying social media text related to mental health. The team achieved strong results on two tasks from the SMM4H 2024 workshop: classifying Reddit posts about outdoor spaces and their relation to social anxiety, and identifying tweets about children's medical disorders. Their work highlights the effectiveness of advanced transformer models like RoBERTa for nuanced mental health text analysis.

What Problem Does This Paper Address?

Social media has become a ubiquitous communication platform, but it also hosts a wealth of information about mental health. Identifying and classifying this information is crucial for understanding public health trends, potential therapeutic interventions, and the prevalence of various conditions. However, the unstructured and often informal nature of social media text presents a significant challenge for accurate analysis. This paper tackles two specific challenges from the SMM4H 2024 workshop:

  1. Task 3: Social Anxiety and Outdoor Spaces: Classifying Reddit posts into categories indicating positive, neutral, negative, or unrelated effects of outdoor environments on social anxiety symptoms. This aims to explore the potential therapeutic benefits of nature.
  2. Task 5: Childhood Medical Disorders: Identifying tweets that describe children experiencing specific conditions like ADHD, ASD, delayed speech, or asthma, distinguishing these from tweets that merely mention these disorders. This contributes to understanding the public discourse and self-reporting of these conditions.

Methods

The core of this research relies on leveraging pre-trained transformer models, specifically RoBERTa and BERT, and fine-tuning them for the specific classification tasks.

  • Preprocessing: Standard text cleaning techniques were applied, including removing URLs, hashtags, user mentions, emojis, and punctuation, while also converting text to lowercase. For Task 5, contractions were expanded (e.g., "wouldn't" to "would not").
  • Data Augmentation (Task 3): Due to class imbalance in Task 3, random oversampling was used to balance the dataset.
  • Transformer Models:
    • Task 3: Several RoBERTa-base and BERT-base-uncased configurations were explored, including:
  • Mean pooling of the output.
  • Concatenation of Mean pooling, CLS token, and Attention Head outputs.
  • An additional Attention Head on top of the model.
  • Weighted layer pooling from the last 4 layers of BERT-base-uncased.
  • Task 5: RoBERTa-large and BERT-large models were employed using ensemble methods. These ensembles combined outputs from the last four hidden layers through various strategies:
  • Weighted ensemble of concatenated last 4 hidden layers.
  • Weighted ensemble of concatenated last 4 hidden layers with weighted pooling.
  • Mean pooling of the concatenated last 4 hidden layers.
  • Fine-tuning: Hyperparameters such as AdamW optimizer, learning rates (2e-5 for Task 3, 1e-5 for Task 5), batch sizes (16 for Task 3, 32 for Task 5), and dropout (0.3) were tuned. Training was conducted for 10 epochs with early stopping based on validation performance.

Datasets

The research utilized datasets provided by the SMM4H 2024 workshop for two shared tasks:

  • Task 3: A multi-class classification dataset consisting of Reddit posts related to outdoor spaces and social anxiety. The dataset was divided into 1800 training, 600 validation, and 1200 test samples. The classes were: (0) Positive effect, (1) Neutral or no effect, (2) Negative effect, and (3) Unrelated.
  • Task 5: A binary classification dataset of tweets reporting children's medical disorders. It included 7398 training, 389 validation, and 10,000 test samples. Class "1" represented tweets describing children with ADHD, ASD, delayed speech, or asthma, while class "0" represented tweets merely mentioning these disorders.

The test data for both tasks was not publicly provided.

Key Insights and Ideas

  • Transformer Power: The study strongly reinforces the efficacy of transformer architectures, particularly RoBERTa, for complex social media text classification tasks related to mental health.
  • Ensembling for Performance: For Task 5, ensemble methods combining multiple representations from RoBERTa-large significantly boosted performance, indicating that different layers capture complementary information.
  • Task-Specific Optimization: The choice of model configuration and fine-tuning strategy (e.g., weighted layer pooling vs. attention heads) had a notable impact on performance for each task, suggesting that task-specific tuning is critical.
  • Potential of Social Media Data: The research underscores the value of social media as a rich source for epidemiological studies and understanding public health issues, even for nuanced topics like the impact of outdoor environments on mental well-being.

Key Results

  • Task 3: The best performing model on the official test set was RoBERTa-base with the concatenation of Mean pooling, CLS, and Attention Head, achieving an F1-score of 0.596.
  • Task 5: The RoBERTa-large model with a weighted ensemble of the last 4 hidden layers concatenation and Mean pooling achieved the highest F1-score of 0.928 on the test dataset. This model also showed strong performance on the validation set with an F1-score of 0.923.

Limitations and Future Work

  • Limitations:
    • The paper does not explicitly discuss limitations beyond the inherent challenges of social media data.
  • The test sets were not publicly available, meaning direct comparison with other research on the exact same held-out data is not possible.
  • Future Work: The authors state they are "working on further improvements," but specific directions are not detailed in the provided text. Potential avenues could include exploring larger transformer models, more advanced data augmentation techniques, or incorporating multimodal information if available.

Summary updated

BibTeX
@inproceedings{mia2024golden,
  author = {Md Ayon Mia and Mahshar Yahan and Hasan Murad and Muhammad Khan},
  title = {{Golden\_Duck at \#SMM4H 2024: A Transformer-based Approach to Social Media Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure