A New Network Structure for Speech Emotion Recognition Research

Chunsheng Xu, Yunqing Liu, Wenjun Song, Zonglin Liang, Xing ChenSensors 2024 · DOI

TL;DR

This paper introduces a novel deep learning architecture for Speech Emotion Recognition (SER) that combines Bidirectional Gated Recurrent Units (Bi-GRU) with a multi-head attention mechanism. This approach effectively captures both local and global temporal dependencies in speech, leading to improved accuracy in identifying emotions compared to existing methods.

What Problem Does This Paper Address?

Speech Emotion Recognition (SER) is a crucial component of human-computer interaction, aiming to enable machines to understand human emotions conveyed through speech. A significant challenge in SER is effectively extracting and integrating emotional information, which is often distributed across different temporal scales within speech signals. Existing methods struggle to comprehensively capture both short-term acoustic variations and long-term emotional context. This paper addresses this by proposing a network structure that can better balance local and global feature extraction, aiming to overcome the limitations of dispersed emotional information and redundant data that can hinder accuracy and stability.

Methods

The core of this paper is a novel network architecture for SER that integrates two key components:

  • Bidirectional Gated Recurrent Unit (Bi-GRU): This recurrent neural network layer is used to learn long-term correlations and contextual feature information from the speech signal. By processing the sequence in both forward and backward directions, Bi-GRU captures dependencies across the entire utterance, providing a richer understanding of temporal dynamics.
  • Multi-Head Attention Mechanism: This mechanism is employed to allow the model to focus on the most relevant features related to emotions. By attending to different parts of the input sequence simultaneously, it helps the model prioritize emotionally salient information, effectively filtering out noise and irrelevant details.

The proposed model first uses CNN layers to capture local associations from spectrograms, followed by Bi-GRU layers for long-term correlations. The multi-head attention layer then refines the focus on emotion-related features, and a softmax layer outputs the predicted emotion.

Datasets

The researchers evaluated their proposed model on two widely used SER corpora:

  • IEMOCAP (Interactive Emotional Dyadic Motion Capture Database): This dataset is known for its multimodal nature, including audio and video recordings, and features both scripted and spontaneous dyadic interactions.
  • Emo-DB (Emotional Database): This German corpus contains acted emotional speech across various categories.

Additionally, the model was applied to speech sentiment analysis tasks on the following datasets to assess its generalization capabilities:

  • CH-SIMS (Chinese Multimodal Sentiment Analysis Dataset): A Chinese multimodal sentiment analysis dataset.
  • MOSI (Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos): A dataset for multimodal sentiment analysis.

Key Insights and Ideas

  • Synergistic Power of Bi-GRU and Multi-Head Attention: The combination of Bi-GRU for temporal context and multi-head attention for feature focus proves to be more effective than using either component in isolation or traditional deep learning models. This synergy allows for a more nuanced understanding of emotional cues.
  • Spectrograms as Effective Input: The paper reinforces the idea that spectrograms are a valuable representation for SER, as they retain rich frequency domain information crucial for emotion detection.
  • Balancing Local and Global Features: The architecture successfully addresses the challenge of integrating local acoustic features with global emotional context, a key limitation in previous SER models.
  • Impact of Head Number in Attention: The study observed that the number of attention heads influences performance, with an optimal range (e.g., 8 or 16 heads) leading to the best results, suggesting a trade-off between model complexity and its ability to learn fine-grained emotional nuances.

Key Results

The proposed Bi-GRU with multi-head attention model achieved state-of-the-art or highly competitive results on the evaluated datasets:

  • IEMOCAP: Achieved an unweighted accuracy (UA) of 75.04%.
  • Emo-DB: Achieved an unweighted accuracy (UA) of 88.93%.

These results represent significant improvements over baseline models and other commonly used deep learning methods. For instance, the model showed a 10.82% improvement over a previous study on IEMOCAP and outperformed a Bi-GRU baseline by 4.40% on IEMOCAP and 5.71% on Emo-DB. The model also demonstrated good generalization performance on speech sentiment analysis tasks.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Specific Emotion Recognition: While overall accuracy is improved, the recognition of certain emotions (e.g., "calm") did not see significant gains, possibly due to their less distinct acoustic features and the current feature fusion methods. Future work could explore more advanced feature fusion techniques.
  • Multimodal Integration: The paper primarily focuses on speech. Combining speech with other modalities like text is identified as a promising direction to significantly enhance emotion recognition performance, aligning with the development of artificial intelligence.
  • Cross-Cultural and Multilingual SER: Recognizing emotions across different languages and cultures is highlighted as a future development trend, acknowledging that cultural differences can influence emotional expression.
  • Computational Cost: The paper notes that training complex models, especially for long input sequences, can be computationally expensive. Future research may explore methods to reduce this cost.
  • Locality-Sensitive Hashing (LSH) Attention: The authors plan to investigate LSH attention to potentially reduce performance loss in the multi-head attention layer.

Summary updated

BibTeX
@article{xu2024new,
  author = {Chunsheng Xu and Yunqing Liu and Wenjun Song and Zonglin Liang and Xing Chen},
  title = {{A New Network Structure for Speech Emotion Recognition Research}},
  journal = {Sensors},
  year = {2024},
  doi = {10.3390/s24051429},
}
Figure