TL;DR
This paper presents a multimodal system for detecting depression using speech and facial landmark features. By exploring various feature extraction techniques and temporal modeling approaches, the authors demonstrate that combining audio and visual cues, particularly with a focus on longer temporal contexts, significantly improves depression detection accuracy compared to unimodal or frame-level methods.
What Problem Does This Paper Address?
Depression is a serious mental health disorder with significant societal and economic impacts. Objective diagnosis is challenging, often relying on subjective assessments and questionnaires, leading to potential misdiagnosis. This paper aims to address this by developing an automated system for depression detection using audiovisual cues, specifically focusing on the AVEC 2016 Depression Classification Challenge (DCC). The challenge lies in effectively extracting and fusing information from speech and facial expressions to accurately classify individuals as depressed or not depressed, especially given the subtle and slow-varying nature of depressive behaviors.
Methods
The authors propose a multimodal system that leverages both audio and video data. For audio, they explore several feature sets, including baseline features, Teager energy cepstral coefficients (TECCs), and i-vectors. A key innovation is the use of window-level features with a large temporal context (10-second windows) to capture the slow-varying nature of depression, contrasting with traditional frame-level analysis. They also investigate phoneme rate and duration as features.
For video, they utilize facial landmark data. Their approach involves extracting features from these landmarks, including polynomial parameterization to capture temporal variations in facial expressions rather than just static positions. They also incorporate geometrical features derived from landmark positions. Similar to audio, they apply a 10-second windowing approach for video feature extraction.
The system employs a late fusion strategy, combining the decisions or confidence scores from individual modalities (audio and video) to make a final prediction. They experiment with different fusion techniques, including majority voting, score summation, and a logical AND operation.
Datasets
The primary dataset used is the Distress Analysis Interview Corpus - Wizard of Oz (DAIC-WOZ). This dataset consists of interviews conducted by a virtual human, annotated with self-reported PHQ-8 scores for depression severity and binarized depression ratings. The challenge focuses on classifying depression (depressed vs. not depressed) from audio and video modalities.
Key Insights and Ideas
- Temporal Context is Crucial: Depression is a slow-varying phenomenon. Analyzing features over longer temporal windows (e.g., 10 seconds) significantly outperforms frame-level analysis, as it better captures the sustained nature of depressive behaviors.
- Multimodal Fusion Outperforms Unimodal: Combining information from both speech and facial expressions leads to more robust and accurate depression detection than relying on a single modality.
- Novel Audio Features Shine: Teager energy cepstral coefficients (TECCs) and i-vector modeling show promise in capturing discriminative information from speech for depression detection.
- Facial Dynamics Matter: Polynomial parameterization of facial landmarks, which captures the temporal evolution of expressions, proves more effective than static landmark positions or simple velocity/acceleration features.
- Geometrical Features Add Value: Features derived from distances and areas between facial landmarks provide complementary information for depression detection.
Key Results
- Audio Modality: The i-vector modeling approach, particularly with MFCC features and a G-PLDA classifier, achieved the best performance among audio-only systems, showing significant improvements over the baseline. TECC features also outperformed standard baseline features.
- Video Modality: Polynomial parameterization of facial landmarks, combined with geometrical features (termed VideoFeatSet2), achieved the best performance among video-only systems, outperforming baseline systems. Feature selection was crucial for this set to avoid overfitting.
- Multimodal Fusion: While late fusion of individual modality decisions did not consistently improve performance over the best unimodal system, a logical AND operation between the best audio and video predictions yielded results comparable to the best video modality performance.
Limitations and Future Work
- Data Mismatch: The authors note that performance on the test dataset was poorer than on the development set, suggesting a potential data mismatch.
- Fusion Inconsistency: The decline in performance with some late fusion strategies is attributed to potential inconsistencies in confidence scores across modalities.
- Future Directions:
- Investigate class-specific Universal Background Models (UBMs) for i-vector extraction.
- Explore speaker-level normalization techniques to reduce inter-speaker variability.
- Examine Vocal Tract Length Normalization (VTLN) for MFCC features.
- Utilize TECC features as front-ends for i-vectors.
- Apply i-vector modeling to video features.
- Explore more sophisticated classifiers beyond SVM and G-PLDA.
Summary updated