Experimenting with Transformer-based and Large Language Models for Classifying Effects of Outdoor Spaces on Social Anxiety in Social Media Data

Falwah Alhamed, Julia Ive, Lucia SpeciaACL 2024

TL;DR

This paper explores how social media discussions about outdoor spaces relate to social anxiety disorder (SAD). Using transformer-based and large language models (LLMs), they developed a two-stage pipeline classifier that achieved the best performance in classifying posts, outperforming baseline methods.

What Problem Does This Paper Address?

Social anxiety disorder (SAD) is a prevalent condition, and while general research suggests that spending time in nature can be beneficial for well-being, its specific impact on individuals with SAD is not well understood. Social media platforms like Reddit have become spaces where people with SAD connect and share experiences. This study aims to bridge this knowledge gap by analyzing social media posts to understand the relationship between discussions of outdoor spaces and social anxiety.

Methods

The researchers experimented with several transformer-based models (BERT, RoBERTa, Mental-BERT) and Large Language Models (LLMs) for a four-class classification task. They explored three experimental setups:

  1. Binary Classification: Distinguishing between "Related" (outdoor space discussion relevant to user experience) and "Unrelated" (metaphorical or unclear references) posts.
  2. Multi-Class Classification within Related: Further classifying "Related" posts into "positive," "neutral," or "negative" impacts on well-being.
  3. Four-Class Classification: A direct classification into all four categories.

LLMs were used in a zero-shot learning approach, while transformer models were fine-tuned. Based on initial experiments, they developed three submission methods: a direct four-class classifier, a two-stage pipeline (binary classification followed by multi-class), and a multi-task learning model.

Datasets

The study utilized a dataset provided by the organizers of the SMM4H (Social Media Mining for Health Applications) 2024 challenge. This dataset originates from the r/socialanxiety subreddit on Reddit and contains 3,000 annotated social media posts. The training set comprised 1,800 posts, each labeled with one of four categories: positive, neutral, negative, or unrelated, based on the user's sentiment towards mentioned nature-related spaces or activities.

Key Insights and Ideas

  • Transformer models show promise: Transformer-based models, particularly RoBERTa and Mental-BERT, demonstrated strong performance in classifying social media posts related to outdoor spaces and social anxiety.
  • LLMs struggled with zero-shot: Despite their general capabilities, LLMs performed poorly in this specific zero-shot classification task, suggesting that fine-tuning on domain-specific data is crucial.
  • A staged approach is effective: A two-stage pipeline, where an initial binary classification separates relevant from irrelevant posts, followed by a more nuanced multi-class classification, yielded superior results. This suggests that breaking down the problem can improve accuracy.
  • Mental-BERT excels at relevance: Mental-BERT was identified as the most effective model for the initial binary classification of posts into "Related" and "Unrelated" categories.

Key Results

The evaluation metric was macro-averaged F1-score across all four classes.

  • Best Model Exploration: Mental-BERT was the most effective for binary classification ("Related" vs. "Unrelated"). RoBERTa achieved the best performance for multi-class classification within the "Related" category and for the overall four-class classification task.
  • Submission Performance: The two-stage pipeline approach, which used Mental-BERT for the initial binary classification and RoBERTa for the subsequent three-class classification, achieved the highest performance among the submitted systems, exceeding baseline performance. The F1 score for this submission was 0.545.

Limitations and Future Work

  • LLM Performance: The paper notes the disappointing performance of LLMs in their zero-shot setup, indicating a need for further investigation into prompt engineering or fine-tuning strategies for these models in this domain.
  • Dataset Size: While the dataset was annotated, the authors do not explicitly discuss limitations related to dataset size or potential biases.
  • Future Work: The authors suggest that their pipeline approach achieved the highest performance, implying that further refinement of such staged methods could lead to even better results. They also implicitly suggest exploring more robust fine-tuning strategies for LLMs.

Summary updated

BibTeX
@inproceedings{alhamed2024experimenting,
  author = {Falwah Alhamed and Julia Ive and Lucia Specia},
  title = {{Experimenting with Transformer-based and Large Language Models for Classifying Effects of Outdoor Spaces on Social Anxiety in Social Media Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure