ThangDLU at #SMM4H 2024: Encoder-decoder models for classifying text data on social disorders in children and adolescents

Thang Ta, Abu Rahman, Lotfollah Najjar, Alexander GelbukhACL 2024

TL;DR

This paper explores the use of pre-trained encoder-decoder models (BART and T5) for classifying social media text related to social anxiety and childhood disorders. They found that these models, especially when augmented with paraphrased data, offer a strong baseline and can outperform average team performance in the #SMM4H 2024 workshop.

What Problem Does This Paper Address?

Social disorders, like social anxiety disorder (SAD) and various childhood conditions (ADHD, ASD, speech delay, asthma), significantly impact young people. Social media platforms like Twitter and Reddit are rich sources of data where individuals discuss their experiences with these disorders. However, automatically analyzing this vast amount of text to understand prevalence, impact, and experiences is a challenging task. This paper addresses the need for effective text classification methods to analyze social media data for identifying and understanding social disorders in children and adolescents, specifically within the context of the #SMM4H 2024 workshop's shared tasks.

Methods

The authors leverage transfer learning with pre-trained encoder-decoder models, specifically BART-base and T5-small. These models, based on the Transformer architecture, are well-suited for sequence-to-sequence tasks and have shown strong performance in various natural language processing tasks.

They explored two main strategies:

  • Direct Training: Using the original datasets provided by the workshop.
  • Data Augmentation:
    • Paraphrased Data: Generating paraphrased versions of existing training and validation data using ChatGPT to increase the dataset size and diversity.
  • False Inferred Data: Training a model on the original data, then using it to predict labels for the validation set. Texts where the model's prediction was incorrect (false inferences) were then added back to the training set.
  • Training + Validation Data: Combining the original training and validation sets for training.

Interestingly, they found that training without extensive preprocessing (like tokenization or stemming) yielded better results, suggesting that raw text might contain valuable signals for these powerful models.

Datasets

The paper focuses on two tasks from the #SMM4H 2024 workshop:

  • Task 3: Classifying Reddit posts related to outdoor environments and their impact on social anxiety. The dataset consists of 3,000 annotated posts from the r/socialanxiety subreddit, filtered for users aged 12-25. The classification is multi-class, categorizing posts into "positive effect," "neutral/no effect," "negative effect," and "unrelated." The data is split into training/validation (80%) and testing (20%).
  • Task 5: Automatically classifying tweets from users reporting pregnancy, specifically identifying tweets that mention children with ADHD, autism, delayed speech, or asthma ("1") versus those merely mentioning a disorder ("0"). The dataset includes 7,398 training tweets, 389 validation tweets, and 1,947 test tweets.

Key Insights and Ideas

  • Encoder-decoder models are powerful baselines: BART and T5, even without extensive task-specific fine-tuning, provide a strong starting point for social media text classification related to health disorders.
  • Data augmentation can boost performance: Both paraphrased data and "false inferred" data showed potential to improve model performance, though the impact varied. Paraphrased data, in particular, seemed to offer a tangible benefit.
  • Raw text might be best: The observation that training without preprocessing yielded better results is significant. It suggests that the nuances and informal language present in social media posts can be effectively captured by these advanced models without explicit cleaning.
  • More data is generally better: The "Training + Validation" approach, which effectively increased the training data size, consistently led to better results, highlighting the importance of data quantity.

Key Results

  • Task 3: The best model, trained on the combined "Training + Validation" data, achieved an F1-macro score of 0.627. This outperformed the mean F1 score of other teams (0.518) but was lower than the median (0.579).
  • Task 5: The best model, trained on the "Training + Validation" data, achieved an F1 score of 0.87. All their models performed better than the mean F1 score of other teams (0.822) but were lower than the median (0.901). They noted their T5-small model's performance was lower than a benchmark using RoBERTa-Large, attributing this to the smaller scale of their chosen model.

Limitations and Future Work

  • Model Scale: The authors acknowledge that using a smaller pre-trained model like T5-small for Task 5 might have limited their performance compared to larger models like RoBERTa-Large.
  • Data Augmentation Effectiveness: While data augmentation showed promise, the impact of "false inferred" data was less pronounced, suggesting a need for more refined augmentation strategies.
  • Future Work: The authors plan to investigate further how large language models' outputs, like those from ChatGPT, can positively impact downstream classification tasks. They also aim to explore more sophisticated data augmentation techniques.

Summary updated

BibTeX
@inproceedings{ta2024thangdlu,
  author = {Thang Ta and Abu Rahman and Lotfollah Najjar and Alexander Gelbukh},
  title = {{ThangDLU at \#SMM4H 2024: Encoder-decoder models for classifying text data on social disorders in children and adolescents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure