Speech Emotion Recognition with Heterogeneous Feature Unification of Deep Neural Network

Wei Jiang, Zheng Wang, Jesse S. Jin, Xian-Feng Han, Chunguang LiSensors 2019 · DOI

TL;DR

This paper tackles the challenge of speech emotion recognition (SER) by proposing a novel deep neural network architecture that effectively unifies heterogeneous acoustic features. By addressing the inherent differences and potential redundancies in various feature types, their method achieves improved SER performance, outperforming existing state-of-the-art approaches.

What Problem Does This Paper Address?

Recognizing human emotions from speech is crucial for more natural and effective human-computer interaction. However, this task is inherently difficult because the acoustic signals that convey emotion are complex and can be represented by a wide variety of features. These features, often extracted from different aspects of speech (e.g., prosody, spectral characteristics, voice quality), can be "heterogeneous" – meaning they have distinct distributions and may contain redundant or even conflicting information. Simply combining these disparate features often leads to suboptimal performance in emotion recognition models. The core problem this paper addresses is how to effectively integrate these heterogeneous acoustic features to extract truly discriminative representations for accurate SER.

Methods

The proposed solution is a hybrid deep neural network architecture consisting of three main modules:

  1. Features Extraction Module: This module extracts a diverse set of acoustic features. It includes traditional low-level features like IS10, MFCCs, and eGemaps, as well as high-level features derived from pre-trained deep learning models like SoundNet and VGGish. This ensures a rich and varied set of initial representations.
  2. Heterogeneous Unification Module: This is the core innovation. It comprises multiple "branch networks," each designed to process a specific type of heterogeneous feature. These branch networks employ unsupervised learning techniques, specifically autoencoders (including basic autoencoders, denoising autoencoders, and an improved Shared-Hidden-Layer Autoencoder - SHLA), to learn abstract, high-level representations. The goal is to convert the diverse, heterogeneous feature spaces into a unified, uniform representation space, effectively mitigating the heterogeneity problem.
  3. Fusion Network Module: A four-layer deep neural network acts as a fusion framework. It takes the unified features from the unification module, concatenates them, and learns the associations between these unified features. This module captures higher-level interactions to generate a final, discriminative feature representation.

Finally, a Support Vector Machine (SVM) is used as the classifier for the recognition task.

Datasets

The primary dataset used for training and evaluation is the Interactive Emotional Dyadic Motion Capture Database (IEMOCAP). This is a well-established, multimodal corpus containing audio, visual, and text data from ten actors engaged in dyadic interactions. For this study, only the audio data was utilized. The paper notes that the dataset is organized into five sessions and includes approximately 12 hours of recordings. Emotions are labeled by human annotators. The paper focuses on four emotional categories: angry, happy, neutral, and sad.

Key Insights and Ideas

  • Heterogeneity is a major hurdle: The paper strongly emphasizes that the diverse nature of acoustic features is a significant challenge for SER, and simply concatenating them is insufficient.
  • Unsupervised learning for unification: Using autoencoders within branch networks is a clever way to learn robust, unified representations from heterogeneous inputs without relying solely on labeled data for this stage.
  • Deep learning for feature extraction and fusion: The architecture leverages the power of deep neural networks both for extracting high-level features (SoundNet, VGGish) and for learning complex associations between unified features during fusion.
  • SHLA's effectiveness: The improved Shared-Hidden-Layer Autoencoder (SHLA) variant shows particular promise in the heterogeneous unification module, contributing to better overall performance.

Key Results

The proposed architecture achieved a total accuracy of 64% on the IEMOCAP dataset, which the authors state is an improvement over existing state-of-the-art approaches.

  • Classifier Comparison: SVM outperformed KNN, LR, and Random Forest, achieving the best performance among the tested classifiers.
  • Feature Comparison: Deep learning-based bottleneck features (SoundNet, VGGish) significantly outperformed traditional low-level features (IS10, MFCCs, eGemaps) in per-class accuracy. VGGish bottleneck features achieved the highest single-feature accuracy (48%).
  • Ablation Studies:
    • Removing the heterogeneous unification module (ours-H) resulted in the worst performance (51% total accuracy), highlighting its critical role.
  • Removing the fusion network module (ours-F) still showed good performance (59%), indicating the unification module's strength, but the fusion module further boosted accuracy.
  • Using DAE as branch networks yielded 63% accuracy, while the proposed ours+shla method achieved the best result of 64%.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the Neutral emotion category is harder to classify due to its inherent ambiguity.
  • The classification results for Angry and Sad categories were not as strong as Happy, potentially due to smaller sample sizes.
  • The study focused solely on audio data, not exploring multimodal fusion (audio-visual-text).
  • Future Work:
    • The authors plan to conduct more experiments on other public benchmark databases to further validate their approach.
  • A significant future direction is to extend the architecture to handle multimodal features for even more robust emotion recognition.

Summary updated

Abstract

Automatic speech emotion recognition is a challenging task due to the gap between acoustic features and human emotions, which rely strongly on the discriminative acoustic features extracted for a given recognition task. We propose a novel deep neural architecture to extract the informative feature representations from the heterogeneous acoustic feature groups which may contain redundant and unrelated information leading to low emotion recognition performance in this work. After obtaining the informative features, a fusion network is trained to jointly learn the discriminative acoustic feature representation and a Support Vector Machine (SVM) is used as the final classifier for recognition task. Experimental results on the IEMOCAP dataset demonstrate that the proposed architecture improved the recognition performance, achieving accuracy of 64% compared to existing state-of-the-art approaches.
BibTeX
@inproceedings{jiang2019speech,
  author = {Wei Jiang and Zheng Wang and Jesse S. Jin and Xian-Feng Han and Chunguang Li},
  title = {{Speech Emotion Recognition with Heterogeneous Feature Unification of Deep Neural Network}},
  booktitle = {Sensors},
  year = {2019},
  doi = {10.3390/s19122730},
}
Figure