SSN@LT-EDI-ACL2022: Transfer Learning using BERT for Detecting Signs of Depression from Social Media Texts

Adarsh S, Betina AntonyACL 2022

TL;DR

This paper explores using BERT, a powerful language model, for detecting signs of depression in social media text. By fine-tuning a pre-trained BERT model, the researchers achieved a competitive F1-score, demonstrating the effectiveness of transfer learning for this sensitive task.

What Problem Does This Paper Address?

Mental health, particularly depression, is a significant global concern. Early detection of depression is crucial for timely treatment and prevention of severe outcomes. Social media platforms have become spaces where individuals feel more comfortable discussing mental health issues due to anonymity. However, analyzing this vast amount of unstructured text to identify depression signs and their severity presents a considerable challenge. This paper aims to develop and evaluate a model that can effectively classify social media text for depression severity.

Methods

The core of this research lies in transfer learning using BERT (Bidirectional Encoder Representations from Transformers). Specifically, the authors employed a pre-trained BERT-small model from TensorFlow Hub, originally trained on Wikipedia. This model was then fine-tuned for the specific task of depression detection on social media data.

The fine-tuning process involved several key steps:

  • Pre-processing: Raw social media text was cleaned, including removing duplicate entries and stop words to reduce noise and improve model efficiency. Sentences were then padded or truncated to a maximum length of 512 tokens, which is BERT's standard input size.
  • Tokenization: Words were converted into numerical tokens using a TensorFlow tokenizer with a vocabulary size of 1024.
  • BERT Encoding: The pre-trained BERT model processed these tokens to generate contextualized vector representations.
  • Dropout: A dropout layer was used to normalize weights and prevent overfitting.
  • Classifier: A final dense layer was added to map the BERT output to the three depression severity classes: "Not depression," "Moderate," and "Severe."

The authors also compared their BERT-based approach against two other models: a basic 3-layer embedded model and a Bidirectional LSTM model.

Datasets

The dataset used in this study comprised social media text scraped from platforms like Twitter and Reddit. It was divided into training, development, and testing sets. The paper notes that the initial train and development sets contained duplicates, which were removed. The authors then performed an 80:20 split of the combined train and development sets for their final training and development splits. The exact source and nature of the "social media texts" beyond this general description are not detailed.

Key Insights and Ideas

  • BERT's contextual understanding is key: The paper highlights BERT's strength in understanding context, even when dealing with subtle language cues or contrasting terms. This is demonstrated by the BERT model's ability to correctly classify a sentence as "moderate" depression when it contained words like "depressed" and "failure" within a broader context, whereas other models labeled it as "severe."
  • Transfer learning is effective for specialized tasks: Fine-tuning a pre-trained BERT model, originally trained on general text, proved to be a successful strategy for the specialized task of depression detection on social media. This suggests that BERT's learned linguistic features are transferable and adaptable.
  • Subtlety in language matters: The authors acknowledge that the absence of directly related depression keywords can impact model performance. This was illustrated with an example where a sentence hinting at self-harm was classified as "moderate" due to a lack of explicit depression-related terms, underscoring the challenge of interpreting nuanced language.

Key Results

The fine-tuned BERT model achieved an F1-score of 63.8%, which the authors state was "reasonably better than the other competing models." While the paper doesn't explicitly detail the F1-scores of the other models in the main text, Table 5 provides a comparative evaluation:

  • Fine-tuned BERT: Accuracy: 0.636, Macro F1-score: 0.531, Weighted F1-score: 0.638
  • Bidirectional LSTM: Accuracy: 0.610, Macro F1-score: 0.466, Weighted F1-score: 0.610
  • Embedding Model: Accuracy: 0.560, Macro F1-score: 0.432, Weighted F1-score: 0.574

The BERT model also showed superior performance in terms of accuracy and weighted F1-score compared to the other models.

Limitations and Future Work

The authors acknowledge that the model's performance can be impacted by the absence of directly depression-related words in the text. They suggest that future work could focus on enhancing the model's ability to understand superficial and unclear words by better interpreting the context and redistributing weights within the encoder layer. This could involve exploring more advanced techniques for handling nuanced language and implicit sentiment.

Summary updated

BibTeX
@inproceedings{s2022ssn,
  author = {Adarsh S and Betina Antony},
  title = {{SSN@LT-EDI-ACL2022: Transfer Learning using BERT for Detecting Signs of Depression from Social Media Texts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure