UMUTeam@LT-EDI-ACL2022: Detecting Signs of Depression from text

José García-Díaz, Rafael Valencia-GarcíaACL 2022

TL;DR

This paper presents UMUTeam's approach to detecting signs of depression in social media text for the LT-EDI-ACL 2022 shared task. They achieved 6th place by combining linguistic features with sentence embeddings from BERT and RoBERTa, using a knowledge integration strategy.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically identifying signs of depression in short social media posts. This is a crucial area for mental health support, but distinguishing between moderate, severe, or no signs of depression in text is difficult due to the nuances of language and the often imbalanced nature of datasets. The LT-EDI-ACL 2022 shared task specifically aimed to push the boundaries of NLP tools in this domain.

Methods

UMUTeam's pipeline involves several steps:

  1. Preprocessing: Text is cleaned by removing punctuation, spaces, and emojis.
  2. Feature Extraction: Four types of features are extracted:
  3. Linguistic Features (LF): These include stylometric features (word/sentence length, Type-Token ratio), Part-of-Speech tags, and analysis of emojis and social network jargon. The authors highlight their interpretability and past success in author analysis.
  4. Sentence Embeddings: * FastText (SE): Non-contextual embeddings.
  5. BERT (BF) and RoBERTa (RF): Contextual embeddings from distilled models for efficiency.
  6. Model Training: Various neural networks are trained, including individual networks for each feature set and a combined network using all features. Hyperparameter tuning is performed using Ray Tune and Tree of Parzen Estimators (TPE).
  7. Ensembling: Two ensemble strategies are explored: one based on the mode of predictions and another based on averaging probabilities.
  8. Knowledge Integration: This strategy is used to combine features, which the authors favor due to its past success in similar competitions.

Datasets

The primary dataset used is from the LT-EDI-ACL 2022 shared task, which consists of social media posts in English labeled for moderate, severe, or no signs of depression. The dataset is split into training, validation, and testing sets. The authors note that the dataset is imbalanced, with a majority of instances indicating moderate signs of depression.

Key Insights and Ideas

  • Linguistic Features Still Matter: Despite the rise of powerful deep learning models, traditional linguistic features still offer valuable, interpretable information for depression detection.
  • Knowledge Integration Outperforms Simple Ensembling: The authors found that their "knowledge integration" strategy yielded better results than simple ensemble methods (mode or averaging predictions) on the validation set.
  • Distilled Models are Efficient: Using distilled versions of BERT and RoBERTa allowed for efficient training while retaining good performance.
  • Imbalance is a Challenge: The dataset's imbalance, with a heavy skew towards "moderate" depression, presents a significant challenge that models must overcome.

Key Results

  • Individual Feature Set Performance: BERT embeddings (BF) achieved the highest F1-score (77.27%) among individual feature sets on the validation split, closely followed by RoBERTa (RF) at 76.91%.
  • Knowledge Integration Improvement: The knowledge integration strategy boosted the macro F1-score to 77.90% on the validation set.
  • Ensemble (Mean) Performance: Averaging probabilities in an ensemble achieved the highest macro F1-score of 78.69% on the validation set.
  • Official Competition Result: UMUTeam secured 6th position out of 31 teams in the official leader board, achieving a macro F1-score of 53.82.

Limitations and Future Work

  • Performance Gap: The authors acknowledge a performance gap between their validation results (79.90% macro F1) and their official test results (53.82% macro F1), suggesting that the chosen strategy might not have generalized as well as hoped on the unseen test data.
  • Figurative Language: The authors plan to incorporate linguistic features from figurative language, as described in prior work, to further improve their models.
  • English Adaptation: They intend to continue adapting their methods to English, building on their experience with this shared task.

Summary updated

BibTeX
@inproceedings{garciadiaz2022umuteam,
  author = {José García-Díaz and Rafael Valencia-García},
  title = {{UMUTeam@LT-EDI-ACL2022: Detecting Signs of Depression from text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure