scubeMSEC@LT-EDI-ACL2022: Detection of Depression using Transformer Models

Sivamanikandan S, Santhosh V, Sanjaykumar N, Jerin Mahibha C, Thenmozhi DurairajACL 2022

TL;DR

This paper tackles the challenge of automatically detecting depression from social media text using transformer models. The authors adapted and evaluated three transformer architectures (DistilBERT, RoBERTa, and ALBERT) on a specialized dataset, achieving a Macro F1-score of 0.457 with RoBERTa, highlighting the potential of these models for mental health monitoring.

What Problem Does This Paper Address?

Social media has become an integral part of daily life, with users frequently sharing their thoughts and feelings. This rich stream of text offers a unique opportunity to identify mental health conditions like depression, which significantly impacts an individual's well-being. However, accurately detecting depression from social media posts is complex due to the nuanced and often indirect ways people express their emotional state. This paper addresses the need for robust computational methods to analyze social media content for depression detection, specifically focusing on the LT-EDI 2022 ACL shared task.

Methods

The core of this research lies in the application and evaluation of transformer-based deep learning models. The authors experimented with three prominent transformer architectures:

  • DistilBERT: A distilled, smaller, faster, and cheaper version of BERT, designed to retain most of BERT's performance with fewer parameters.
  • RoBERTa: An optimized BERT pre-training approach that modifies key hyperparameters and uses a larger dataset and batch size for improved performance.
  • ALBERT: A Lite BERT model that enhances training and results through parameter sharing, embedding matrix factorization, and inter-sentence coherence loss.

The process involved standard text pre-processing steps, including converting text to lowercase, removing stop words, URLs, numbers, and tags. The labels ("Not Depressed," "Moderate," "Severe") were converted into numerical categories for model input. The models were trained on the provided dataset and evaluated using standard metrics.

Datasets

The study utilized a dataset provided by the organizers of the LT-EDI 2022 ACL shared task, specifically designed for detecting signs of depression from social media text. This dataset was in a Tab Separated Values (tsv) format and contained three columns: posting ID, text data, and label.

  • Training Data: 8,891 instances.
  • Validation Data: 4,496 instances.
  • Test Data: 3,245 instances.

The dataset was imbalanced, with a significantly higher number of "Moderate" depression instances compared to "Not Depressed" and "Severe."

Key Insights and Ideas

  • Transformer Models Show Promise: The study confirms that transformer models, particularly RoBERTa, are effective in capturing the nuances of social media text for depression detection, outperforming simpler machine learning approaches.
  • Data Imbalance is a Challenge: The inherent imbalance in the dataset (more "Moderate" cases) significantly impacted the model's performance, especially for the "Severe" category, leading to more false positives and negatives.
  • Subtlety of "Severe" Depression: The authors noted that posts categorized as "Severe" often lack direct depression markers, making them harder to classify accurately. This suggests that context and indirect linguistic cues are crucial.
  • Efficiency of DistilBERT: While RoBERTa achieved the best results, DistilBERT offered a good balance of performance and efficiency, being a smaller and faster model.

Key Results

The RoBERTa model achieved the best performance among the tested transformers, yielding:

  • Accuracy: 0.510
  • Macro F1-Score: 0.457
  • Macro Recall: 0.519
  • Macro Precision: 0.461

This performance placed the team at rank 23 on the leader board. DistilBERT and ALBERT performed with Macro F1-scores of 0.337 and 0.387, respectively. The classification report highlighted a particularly low F1-score (0.30) for the "Severe" category, underscoring the challenges in identifying extreme cases.

Limitations and Future Work

The authors acknowledge that their RoBERTa model's F1-score of 0.457 is lower than the top-performing model (0.583), indicating room for improvement. Key limitations and future directions include:

  • Addressing Data Imbalance: The paper suggests using up-sampling and down-sampling techniques to create a more balanced dataset, which is expected to improve model performance.
  • Identifying Depression Markers: Future work should focus on more effectively identifying specific depression markers and incorporating contextual information from social media posts.
  • Fine-tuning Transformer Models: Further fine-tuning of transformer models, potentially with more domain-specific data or advanced techniques, could lead to better accuracy.
  • Multi-modal Approaches: While not explored in this paper, integrating other modalities (e.g., images, user behavior) could offer a more comprehensive understanding of mental health.

Summary updated

BibTeX
@inproceedings{s2022scubemsec,
  author = {Sivamanikandan S and Santhosh V and Sanjaykumar N and Jerin Mahibha C and Thenmozhi Durairaj},
  title = {{scubeMSEC@LT-EDI-ACL2022: Detection of Depression using Transformer Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure