TL;DR
This paper introduces CTL-MTNet, a novel deep learning framework that simultaneously tackles both single-corpus and cross-corpus Speech Emotion Recognition (SER). By combining a Capsule Network with self-attention (CPAC) for robust feature extraction and a Corpus Adaptation Adversarial Module (CAAM) for domain invariance, CTL-MTNet achieves state-of-the-art performance on challenging SER tasks.
What Problem Does This Paper Address?
Speech Emotion Recognition (SER) aims to identify emotions from spoken language, a crucial aspect of human-computer interaction. A significant challenge in SER is the domain shift problem: models trained on one dataset (source corpus) often perform poorly on data from a different dataset (target corpus). This is due to variations in speakers, recording environments, and even languages. Existing methods often struggle to generalize across these different data distributions, especially when aiming to perform well on both single-corpus (within-dataset) and cross-corpus (across-dataset) tasks simultaneously. This paper addresses this by proposing a unified framework capable of handling both scenarios effectively.
Methods
The core of the proposed solution is the CTL-MTNet (CapsNet and Transfer Learning-based Mixed Task Net). It comprises two main novel components:
- CPAC (Convolution-Pooling and Attention-based CapsNet): This module is designed for the single-corpus SER task. It leverages a Capsule Network (CapsNet) architecture, enhanced with a self-attention mechanism. The self-attention guides the CapsNet to focus on the most informative features within different capsules, leading to more discriminative high-level representations. This is built upon a CNN-Pooling feature extractor.
- CAAM (Corpus Adaptation Adversarial Module): This module is specifically designed to tackle the cross-corpus SER challenge. It combines the CPAC feature extractor with a Margin Disparity Discrepancy (MDD) loss function. CAAM aims to learn domain-invariant emotion representations by minimizing the discrepancy between source and target corpus features, effectively bridging the domain gap. It uses an adversarial approach to align these features.
The overall architecture integrates these components to process MFCC features extracted from audio data.
Datasets
The paper evaluates CTL-MTNet on four well-known SER datasets in different languages:
- CASIA
- EMODB
- SAVEE
- RAVDESS
These datasets are used for both single-corpus and cross-corpus evaluations. No new datasets were introduced.
Key Insights and Ideas
- Unified Framework for Dual Tasks: The most significant contribution is the development of a single model capable of excelling in both single-corpus and cross-corpus SER. This is a novel approach that simplifies the SER landscape by avoiding the need for separate models for different scenarios.
- Attention-Guided Capsules: Incorporating self-attention into CapsNet (CPAC) is crucial for focusing on salient features. This allows the model to dynamically weigh the importance of different capsule representations, leading to better discrimination of emotional states.
- Domain Invariance through Adversarial Learning and MDD: The CAAM module, particularly with the MDD loss, effectively addresses the domain shift problem. By forcing the model to learn representations that are indistinguishable across different corpora, it achieves better generalization.
- Capsule Networks for Emotion Representation: The paper further validates the effectiveness of Capsule Networks in capturing complex spatial relationships within speech features for emotion recognition, building on prior work.
Key Results
- Single-Corpus Performance: CTL-MTNet (specifically the CPAC component) achieved state-of-the-art performance on the single-corpus SER tasks across all four datasets, outperforming existing methods. For example, it achieved a WAR of 92.75% on CASIA and 94.97% on EMODB.
- Cross-Corpus Performance: The CTL-MTNet demonstrated significant improvements in cross-corpus SER compared to established methods like CDAN, DANN, and NMFTSL. It achieved an average UAR of 39.90% and WAR of 41.57%, showing substantial gains (e.g., +8.24% UAR and +10.96% WAR over CDAN).
- Ablation Studies: Experiments confirmed the importance of each component. CPAC (with both CNN-Pooling and Self-Attention) significantly outperformed simpler CapsNet variants or models without attention. The CAAM module with MDD was essential for cross-corpus performance gains.
- Visualization: t-SNE visualizations showed that CPAC creates clearer class separations in feature space for single-corpus tasks, while CAAM effectively aligns features from different corpora, reducing inter-corpus distance while maintaining inter-class separation.
Limitations and Future Work
- Cross-Corpus Class Separation: While CAAM improves feature alignment, the paper notes that the separation between classes in the target corpus is not always as clear as desired, which can lead to reduced accuracy in some cross-corpus scenarios.
- Future Work: The authors suggest exploring further improvements in feature alignment for cross-corpus tasks to enhance class separability in target domains. They also implicitly point towards the potential for applying this unified framework to other related audio processing tasks that suffer from domain shift.
Summary updated