IDIAP Submission@LT-EDI-ACL2022: Detecting Signs of Depression from Social Media Text

Muskaan Singh, Petr MotlicekACL 2022

TL;DR

This paper presents a system for detecting signs of depression from social media text, achieving 3rd place in the DepSign-LT-EDI@ACL-2022 challenge. The approach leverages an ensemble of fine-tuned BERT, RoBERTa, and XLNet models, demonstrating the effectiveness of combining powerful pre-trained language models for this sensitive task.

What Problem Does This Paper Address?

Depression is a significant mental health issue that often goes untreated, especially in individuals from lower socioeconomic backgrounds. Early detection is crucial for timely intervention and to mitigate severe consequences. Social media platforms offer a rich source of personal expression, where individuals often share their feelings and emotions. However, accurately identifying subtle signs of depression within this vast and often informal text presents a considerable challenge. This paper tackles the problem of automatically detecting depression from social media posts, classifying them into three categories: "not depressed," "moderately depressed," and "severely depressed."

Methods

The core of the proposed system is an ensemble voting classifier that combines the predictions of three state-of-the-art pre-trained language models: BERT, RoBERTa, and XLNet.

  1. Pre-processing: Tweets are cleaned by removing stop words, emojis, and punctuation using the NLTK library.
  2. Feature Extraction: Sentences are tokenized, and tokens are mapped to word IDs. Special tokens [CLS] and [SEP] are added, and sentences are padded or truncated to a maximum length of 30 tokens. Attention masks are generated to differentiate actual tokens from padding.
  3. Model Architecture: Each of the three models (BERT, RoBERTa, XLNet) is fine-tuned on the social media dataset. These models are known for their ability to understand context and nuances in language.
  4. Ensemble Voting: The predictions from the fine-tuned BERT, RoBERTa, and XLNet models are combined using a voting mechanism. The label with the highest number of votes across the models is selected as the final prediction.

Datasets

The primary dataset used in this work is the DepSign-LT-EDI dataset. This dataset contains social media comments annotated with three labels: "not depressed," "moderately depressed," and "severely depressed." The paper provides a detailed distribution of this dataset across training, development, and test sets.

Key Insights and Ideas

  • Ensemble Power: Combining predictions from multiple powerful pre-trained language models (BERT, RoBERTa, XLNet) significantly improves performance compared to individual models. This highlights the benefit of leveraging diverse linguistic understanding capabilities.
  • Fine-tuning for Specific Tasks: Fine-tuning large pre-trained models on domain-specific data (social media text for depression detection) is crucial for achieving high accuracy.
  • Importance of Context: The models' ability to process sequences and attention masks is key to understanding the context of social media posts, which can be informal and contain various linguistic cues.
  • Challenges in Nuance: Misclassifications can occur when medical terms are used (e.g., "TMS") or when comments contain a mix of positive and negative sentiments, indicating the model's struggle with highly nuanced or domain-specific language.

Key Results

The proposed system achieved 3rd place in the DepSign-LT-EDI@ACL-2022 challenge with an accuracy of 0.54%. The paper reports the following metrics for their submission:

  • Accuracy: 0.6253
  • Precision: 0.572
  • Recall: 0.5303
  • Macro F1-score: 0.5467

Qualitative analysis revealed that while the model correctly identified many instances of depression, it sometimes struggled with comments containing medical jargon or a mix of sentiments, leading to misclassifications. For example, a comment mentioning "TMS" for depression was misclassified as "not depressed," and another comment expressing suicidal ideation was classified as "moderate" due to the presence of mixed positive and negative phrases.

Limitations and Future Work

Limitations: * The model sometimes misclassifies comments containing medical jargon or a mix of positive and negative sentiments. * The current system focuses solely on English text.

Future Work: * Develop a multi-task learning framework to handle different types of depression and their severities. * Extend the system to detect multilingual depression speech, particularly in code-mixing scenarios.

Summary updated

BibTeX
@inproceedings{singh2022idiap,
  author = {Muskaan Singh and Petr Motlicek},
  title = {{IDIAP Submission@LT-EDI-ACL2022: Detecting Signs of Depression from Social Media Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure