Depression Speech Recognition With a Three-Dimensional Convolutional Network

Hongbo Wang, Yu Liu, Xiaoxiao Zhen, Xuyan TuFrontiers in Human Neuroscience 2021 · DOI

Abstract

Depression has become one of the main afflictions that threaten people's mental health. However, the current traditional diagnosis methods have certain limitations, so it is necessary to find a method of objective evaluation of depression based on intelligent technology to assist in the early diagnosis and treatment of patients. Because the abnormal speech features of patients with depression are related to their mental state to some extent, it is valuable to use speech acoustic features as objective indicators for the diagnosis of depression. In order to solve the problem of the complexity of speech in depression and the limited performance of traditional feature extraction methods for speech signals, this article suggests a Three-Dimensional Convolutional filter bank with Highway Networks and Bidirectional GRU (Gated Recurrent Unit) with an Attention mechanism (in short 3D-CBHGA), which includes two key strategies. (1) The three-dimensional feature extraction of the speech signal can timely realize the expression ability of those depression signals. (2) Based on the attention mechanism in the GRU network, the frame-level vector is weighted to get the hidden emotion vector by self-learning. Experiments show that the proposed 3D-CBHGA can well establish mapping from speech signals to depression-related features and improve the accuracy of depression detection in speech signals.

Introduction

Depression is a global health crisis affecting over 350 million people, yet it remains plagued by low recognition and treatment rates. Current diagnosis relies heavily on subjective questionnaire surveys and clinician judgment, which can be influenced by a patient’s willingness to cooperate or a doctor’s specific expertise.

Speech offers a non-invasive, objective "biomarker." Depressed individuals often exhibit characteristic vocal patterns: slower speaking rates, lower intonation, more frequent pauses, and reduced energy variance. However, capturing these nuances is difficult. Traditional feature extraction (like MFCCs) often misses high-level semantic information, and standard neural networks often struggle with the "noise" of daily conversation or the varying importance of different parts of a sentence.

Overview of the depression detection pipeline based on speech signals.

The 3D-CBHGA Methodology

The authors propose the 3D-CBHGA model (Three-Dimensional Convolutional filter bank with Highway Networks and Bidirectional GRU with Attention). The architecture consists of several sophisticated stages designed to map speech signals to depression-related features.

1. 3D Feature Extraction

Instead of using a flat 2D spectrogram, the researchers created a 3D input by combining 40-dimensional Filter bank (Fbank) features with their first-order (Δ) and second-order (ΔΔ) derivatives. This allows the model to "see" the dynamic changes in speech over time, significantly improving the expression of complex depression signals.

2. Multi-channel Convolutional Layer

Inspired by N-gram processing in text, this layer uses kernels of different sizes (e.g., 2, 3, and 4 frames) to capture local correlations in the speech signal at multiple scales. This multi-scale approach ensures that both short-term and slightly longer-term acoustic patterns are captured.

Multi-channel convolution process using various kernel sizes to extract feature maps.

3. Highway Networks and Bi-GRU with Attention

To prevent the "vanishing gradient" problem common in deep networks, Highway layers use "transform" and "carry" gates to regulate information flow. Following this, a Bidirectional Gated Recurrent Unit (Bi-GRU) captures long-term temporal dependencies from both past and future contexts. Crucially, an Attention Mechanism weights the importance of specific frames, ensuring the model focuses on emotionally significant segments while ignoring irrelevant noise.

Experimental Setup

The study utilized the DAIC-WOZ (Distress Analysis Interview Corpus - Wizard of Oz) dataset, containing 189 interactions with a virtual interviewer. Participants were classified as depressed or non-depressed based on PHQ-8 scores. The researchers tested the model on both the original dialogue (DAIC-ori) and a version where silent segments were removed (DAIC-mute-removed) using Voice Activity Detection.

Key Results and Insights

The 3D-CBHGA model outperformed traditional baselines (SVM, Random Forest) and simpler deep learning models across all metrics:

  • Accuracy: On the DAIC-mute-removed dataset, the proposed model reached 77.14% accuracy, compared to 68.57% for SVM.
  • F1-Score: The F1-score reached 0.636, representing a 33.8% improvement over the standard SVM baseline.
  • The Impact of Silence: Removing silent fragments boosted performance significantly. While pauses are a symptom of depression, in raw audio, they often act as interference that dilutes vocal features.
  • Selective Attention: The attention mechanism proved vital for mapping long conversations to mental health states by self-learning which frames carry the most diagnostic weight.

Conclusion and Future Directions

While the 3D-CBHGA model shows great promise, challenges remain regarding data imbalance and the diagnostic overlap between depression and other comorbid disorders like bipolar disorder. Future research will focus on refining the model's specificity and exploring its utility across different languages and cultures to enhance global applicability.

Summary updated

Abstract

Depression has become one of the main afflictions that threaten people's mental health. However, the current traditional diagnosis methods have certain limitations, so it is necessary to find a method of objective evaluation of depression based on intelligent technology to assist in the early diagnosis and treatment of patients. Because the abnormal speech features of patients with depression are related to their mental state to some extent, it is valuable to use speech acoustic features as objective indicators for the diagnosis of depression. In order to solve the problem of the complexity of speech in depression and the limited performance of traditional feature extraction methods for speech signals, this article suggests a Three-Dimensional Convolutional filter bank with Highway Networks and Bidirectional GRU (Gated Recurrent Unit) with an Attention mechanism (in short 3D-CBHGA), which includes two key strategies. (1) The three-dimensional feature extraction of the speech signal can timely realize the expression ability of those depression signals. (2) Based on the attention mechanism in the GRU network, the frame-level vector is weighted to get the hidden emotion vector by self-learning. Experiments show that the proposed 3D-CBHGA can well establish mapping from speech signals to depression-related features and improve the accuracy of depression detection in speech signals.
BibTeX
@inproceedings{wang2021depression,
  author = {Hongbo Wang and Yu Liu and Xiaoxiao Zhen and Xuyan Tu},
  title = {{Depression Speech Recognition With a Three-Dimensional Convolutional Network}},
  booktitle = {Frontiers in Human Neuroscience},
  year = {2021},
  doi = {10.3389/fnhum.2021.713823},
}
Figure