KADO@LT-EDI-ACL2022: BERT-based Ensembles for Detecting Signs of Depression from Social Media Text

Morteza Janatdoust, Fatemeh Ehsani-Besheli, Hossein ZeinaliACL 2022

TL;DR

This paper tackles the crucial task of automatically detecting signs of depression from social media text, a challenging problem due to the nuanced and often indirect nature of such expressions. The authors propose an ensemble model combining several BERT-based architectures (BERT, ALBERT, DistilBERT, RoBERTa) to achieve improved accuracy, outperforming individual models and ranking 5th in the DepSign-LT-EDI@ACL-2022 shared task.

What Problem Does This Paper Address?

Depression is a serious mental illness with significant public health implications, including a link to suicide. Early detection can lead to better treatment outcomes. While traditional diagnostic methods exist, they can be time-consuming and inaccessible. Social media platforms, however, have become a rich source of user-generated text, offering a potential avenue for early, large-scale detection of depressive symptoms. The challenge lies in accurately interpreting the subtle linguistic cues of depression within this vast and often informal data. This paper specifically addresses the task of classifying social media posts into three categories: "not depressed," "moderately depressed," and "severely depressed."

Methods

The core of this work is an ensemble learning strategy. The authors leverage several pre-trained BERT-based language models:

  • BERT (Bidirectional Encoder Representations from Transformers): A foundational deep transformer model that learns bidirectional representations of natural language.
  • ALBERT (A Lite BERT): A parameter-reduced version of BERT designed to decrease memory consumption and increase training speed.
  • DistilBERT: A smaller, faster, and lighter Transformer model distilled from BERT, aiming for improved inference speed while retaining most of BERT's performance.
  • RoBERTa (A Robustly optimized BERT Pretraining Approach): An optimized pre-training strategy for BERT that uses more data and a longer training duration, along with dynamic masking and removal of the Next Sentence Prediction (NSP) task, to achieve better performance.

These individual models are fine-tuned on the social media depression detection task. The final prediction is made using a majority voting mechanism, where the outputs of the individual models are combined to determine the most frequent prediction. This approach aims to capitalize on the diverse strengths and weaknesses of each model.

Datasets

The primary dataset used in this paper is the DepSign-LT-EDI@ACL-2022 shared task dataset. This dataset consists of social media comments in English, labeled with three categories: "not depressed," "moderately depressed," and "severely depressed." The dataset is divided into training, development, and test sets. The authors note that the class distribution in both the training and development sets is imbalanced.

  • Training set: 8,891 instances
  • Development set: 4,496 instances
  • Test set: 3,245 instances

Key Insights and Ideas

  • Ensembling for Robustness: The paper strongly suggests that combining multiple BERT-based models through ensembling can lead to more robust and accurate depression detection than any single model alone. This is attributed to the fact that different models may make different types of errors, and ensembling helps to mitigate these individual shortcomings.
  • Leveraging Diverse BERT Architectures: The choice to include models like ALBERT and DistilBERT alongside BERT and RoBERTa highlights an interest in balancing performance with computational efficiency, a critical consideration for real-world deployment.
  • Imbalanced Data Challenge: The authors explicitly acknowledge the imbalanced nature of the dataset, which is a common challenge in classification tasks and can significantly impact model performance.

Key Results

The proposed ensemble model achieved competitive results on the DepSign task:

  • Accuracy: 61%
  • F1 Score: 54%

The ensemble model outperformed all individual BERT-based models across all considered metrics (Accuracy, Recall, Precision, Weighted F1-score, and Macro F1-score). The paper states that this performance ranked the proposed model 5th on the leaderboard for the DepSign task, among more than 30 submissions.

Limitations and Future Work

Limitations:

  • The paper acknowledges the imbalanced nature of the dataset, which can affect the performance of classification models.
  • While the ensemble model achieved good results, the F1 score of 54% and accuracy of 61% indicate there is still significant room for improvement in accurately detecting depression from social media text.

Future Work:

The authors suggest several promising directions for future research:

  • Exploring other language models: Investigating newer or different types of language models beyond BERT.
  • Developing other ensemble strategies: Experimenting with alternative methods for combining model predictions.
  • Incorporating additional input sources: Utilizing other types of data, such as related dictionaries, NLP tools, or even multimodal information (e.g., images, user behavior), to enhance detection accuracy.

Summary updated

BibTeX
@inproceedings{janatdoust2022kado,
  author = {Morteza Janatdoust and Fatemeh Ehsani-Besheli and Hossein Zeinali},
  title = {{KADO@LT-EDI-ACL2022: BERT-based Ensembles for Detecting Signs of Depression from Social Media Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure