Climate and Weather: Inspecting Depression Detection via Emotion Recognition

Wen Wu, Mengyue Wu, Kai Yu

TL;DR

This paper proposes a novel approach to depression detection by leveraging knowledge from emotion recognition. By pre-training a model on emotion datasets and then transferring its learned features, they achieve improved depression detection performance and stability, especially when fusing emotion with audio and text modalities. The work also sheds light on the complex relationship between transient emotions and long-term depressive states.

What Problem Does This Paper Address?

Detecting depression automatically is a critical but challenging task. Existing methods often struggle with data sparsity, as clinical data for depression is difficult to acquire and annotate. This paper is motivated by psychological research suggesting a strong link between a person's mood (like depression) and their expressed and perceived emotions. The authors hypothesize that by understanding how we recognize emotions, we can build better depression detection systems. They aim to bridge the gap between the long-term, stable nature of depression (the "climate") and the fleeting, transient nature of emotions (the "weather"), and explore how this relationship can be exploited for improved detection.

Methods

The core of this work is a knowledge transfer approach. First, an emotion recognition model is trained using a multimodal architecture that includes both audio (Log Mel filterbank features) and text (BERT embeddings) branches. This model is pre-trained on large emotion datasets.

Once pre-trained, the emotion recognition model's learned representations (specifically, the features at the input of its fusion layer) are extracted. These extracted emotion features are then treated as a new modality.

For depression detection, these emotion features are then fused with traditional audio and text features. The authors explore different configurations:

  1. Unimodal Emotion Detection: Using only the extracted emotion features for depression detection.
  2. Multimodal Depression Detection: Fusing the extracted emotion features with audio and text features.

The pre-trained emotion model's parameters are frozen during the depression detection phase, acting as a feature extractor. For the depression detection task itself, a bi-directional LSTM is used to pool segment-level emotion features and produce session-level predictions.

Datasets

  • Emotion Recognition Pre-training:
    • IEMOCAP: A multimodal dyadic conversational dataset with discrete and continuous emotion labels.
  • CMU-MOSEI: A large multimodal sentiment and emotion analysis dataset derived from movie review videos, featuring sentiment scores.
  • Depression Detection Evaluation:
    • DAIC-WOZ: A benchmark dataset for depression detection, consisting of semi-structured clinical interviews.

Key Insights and Ideas

  • Emotion as a Bridge: Transferring knowledge from emotion recognition significantly improves depression detection performance and, importantly, increases training stability, which is crucial for sparse datasets.
  • Climate vs. Weather Analogy: The paper effectively uses the "climate and weather" analogy to explain the relationship between long-term depression and transient emotions, highlighting that while related, they are not identical.
  • Modality Nuances: The study reveals that audio-based emotion features alone can be less effective for depression detection than text-based ones, suggesting that certain emotional cues might be more indicative of depression than others.
  • Complexity of Expression: Depressed individuals don't always express negative emotions; they can exhibit a range of emotions, sometimes even appearing happy or calm, making detection complex. This highlights the need for nuanced multimodal analysis.
  • Emotion Features Enhance Stability: Using extracted emotion features as a modality consistently reduces the standard deviation of results across different random seeds, indicating greater robustness.

Key Results

  • Improved Depression Detection: The proposed emotion transfer method, when fused with audio and text, achieved a F1 score of 0.869 on the DAIC-WOZ dataset.
  • Enhanced Stability: Emotion features significantly reduced the standard deviation of results compared to using direct audio or text features alone.
  • MOSEI Advantage: Features extracted from models pre-trained on the larger CMU-MOSEI dataset generally yielded better depression detection results than those from IEMOCAP, likely due to its size and naturalistic emotional expressions.
  • Emotional Analysis: Analysis of depressed data showed that while depressed samples tend to exhibit higher scores for anger and sadness, healthy samples lean towards neutral and happy emotions. However, the fused model often predicted "happy" as the dominant emotion, underscoring the complexity.

Limitations and Future Work

  • Data Sparsity: While the proposed method improves stability, the inherent sparsity of depression datasets remains a challenge.
  • Emotion Feature Discarding: The paper notes that direct audio features sometimes outperform audio-based emotion features, suggesting that some valuable information for depression detection might be lost during the emotion feature extraction process.
  • Future Work: The authors suggest further investigating the interaction between emotion and depression, exploring how specific emotion regulation strategies influence expression, and potentially developing more sophisticated fusion mechanisms. The analysis of how depressed individuals' emotions are perceived by others is also highlighted as a promising avenue.

Summary updated

Abstract

Automatic depression detection has attracted increasing amount of attention but remains a challenging task. Psychological research suggests that depressive mood is closely related with emotion expression and perception, which motivates the investigation of whether knowledge of emotion recognition can be transferred for depression detection. This paper uses pretrained features extracted from the emotion recognition model for depression detection, further fuses emotion modality with audio and text to form multimodal depression detection. The proposed emotion transfer improves depression detection performance on DAIC-WOZ as well as increases the training stability. The analysis of how the emotion expressed by depressed individuals is further perceived provides clues for further understanding of the relationship between depression and emotion.
BibTeX
@inproceedings{wu2022climate,
  author = {Wen Wu and Mengyue Wu and Kai Yu},
  title = {{Climate and Weather: Inspecting Depression Detection via Emotion Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746634},
}
Figure