CTL-MTNet: A Novel CapsNet and Transfer Learning-Based Mixed Task Net for Single-Corpus and Cross-Corpus Speech Emotion Recognition

Xin-Cheng Wen, Jiaxin Ye, Yan Luo, Yong Xu, Xuan-Ze Wang, Chang-Li Wu, Kun-Hong Liu

TL;DR

This paper introduces CTL-MTNet, a novel framework that simultaneously tackles both single-corpus and cross-corpus Speech Emotion Recognition (SER). By combining a Capsule Network with attention mechanisms (CPAC) for robust feature extraction and an adversarial module (CAAM) for domain adaptation, CTL-MTNet achieves state-of-the-art performance on both tasks, bridging the gap between different speech datasets.

What Problem Does This Paper Address?

Speech Emotion Recognition (SER) is crucial for human-computer interaction, aiming to understand emotions conveyed through speech. A significant challenge in SER is the "domain gap": models trained on one dataset (source corpus) often perform poorly on another dataset (target corpus) due to variations in speakers, languages, and recording environments. This is particularly problematic for cross-corpus SER, where the goal is to generalize emotion recognition across diverse datasets. Existing methods often focus on either single-corpus or cross-corpus tasks, and many rely on complex, hand-crafted features that are sensitive to training data. This paper addresses the need for a unified framework that can effectively handle both scenarios simultaneously and extract robust, domain-invariant emotional features.

Methods

The proposed CTL-MTNet is a two-pronged approach:

  • CPAC (Convolution-Pooling and Attention CapsNet): For the single-corpus SER task, CPAC enhances a Capsule Network (CapsNet). It integrates a self-attention mechanism into the CapsNet architecture. This attention mechanism guides the network to focus on the most informative features within different capsules, leading to more discriminative high-level representations. The process starts with CNN-Pooling for initial feature extraction from Mel-Frequency Cepstral Coefficients (MFCCs), followed by a PrimaryCaps layer, the self-attention mechanism, and finally DigitCaps to produce robust features.
  • CAAM (Corpus Adaptation Adversarial Module): For the cross-corpus SER task, CAAM leverages CPAC for feature extraction and introduces adversarial learning with Margin Disparity Discrepancy (MDD) loss. The goal is to learn domain-invariant emotion representations. CAAM uses a gradient reversal layer to train a domain discriminator, encouraging the feature extractor (CPAC) to produce representations that are indistinguishable across source and target corpora, while still preserving emotional discriminability. This adversarial process helps bridge the gap between different datasets.

Datasets

The paper evaluates CTL-MTNet on four well-known SER datasets:

  • Institute of Automation of Chinese Academy of Sciences (CASIA)
  • Berlin Emotional dataset (EmoDB)
  • Surrey Audio-Visual Expressed Emotion dataset (SAVEE)
  • Ryerson Audio-Visual dataset of Emotional Speech and Song (RAVDESS)

These datasets vary in language and characteristics, making them suitable for testing both single-corpus and cross-corpus performance.

Key Insights and Ideas

  • Unified Framework: The most significant insight is the development of a single model (CTL-MTNet) capable of excelling in both single-corpus and cross-corpus SER. This eliminates the need for separate models for different scenarios.
  • Attention for Capsule Networks: Integrating self-attention into CapsNet (CPAC) is shown to be highly effective in making capsules focus on crucial emotional features, thereby improving discriminative power.
  • Adversarial Learning for Domain Invariance: CAAM's use of adversarial training with MDD loss effectively learns domain-invariant representations, crucial for overcoming the challenges of cross-corpus SER. This suggests that adversarial techniques are a powerful tool for bridging domain gaps in speech emotion recognition.
  • Capsule Networks' Strength: The paper reinforces the utility of Capsule Networks in capturing complex spatial relationships within speech features, which is beneficial for SER.

Key Results

  • Single-Corpus Performance: CTL-MTNet achieved state-of-the-art results on the single-corpus task, outperforming existing methods across the evaluated datasets. For instance, it achieved a Weighted Average Recall (WAR) of 92.75% on CASIA and 94.97% on EMODB.
  • Cross-Corpus Performance: In cross-corpus experiments, CTL-MTNet demonstrated significant improvements. It outperformed methods like CDAN, DANN, and NMFTSL, achieving average UAR and WAR scores of 39.90% and 41.57%, respectively. This represents substantial gains, such as +8.24% UAR and +10.96% WAR compared to CDAN.
  • Ablation Studies: Ablation studies confirmed the effectiveness of individual components. The CPAC module, in particular, showed a relative improvement of +12.58% on WAR compared to a baseline without attention, and replacing LSTM with CapsNet improved scores by +12.78%. The CAAM module with MDD also provided significant gains (+11.96% UAR and +14.99% WAR) over a baseline without domain adaptation.
  • Visualization: t-SNE visualizations confirmed that CPAC creates clearer class boundaries for single-corpus tasks, while CAAM effectively aligns features from different corpora, reducing inter-corpus distances significantly.

Limitations and Future Work

  • Cross-Corpus Separation: While CAAM significantly reduces the distance between corpora, the paper notes that the separation between different emotion categories in the target corpus is not always clear enough, which can lead to reduced accuracy. This suggests that further refinement in balancing domain invariance and class separability is needed.
  • Future Directions: The authors suggest that future work could explore more advanced adversarial techniques or different domain adaptation strategies to further improve cross-corpus performance. Additionally, investigating the model's robustness to even more diverse and challenging cross-lingual or cross-accent scenarios could be a valuable direction.

Summary updated

Abstract

Speech Emotion Recognition (SER) has become a growing focus of research in human-computer interaction. An essential challenge in SER is to extract common attributes from different speakers or languages, especially when a specific source corpus has to be trained to recognize the unknown data coming from another speech corpus. To address this challenge, a Capsule Network (CapsNet) and Transfer Learning based Mixed Task Net (CTL-MTNet) are proposed to deal with both the single-corpus and cross-corpus SER tasks simultaneously in this paper. For the single-corpus task, the combination of Convolution-Pooling and Attention CapsNet module (CPAC) is designed by embedding the self-attention mechanism to the CapsNet, guiding the module to focus on the important features that can be fed into different capsules. The extracted high-level features by CPAC provide sufficient discriminative ability. Furthermore, to handle the cross-corpus task, CTL-MTNet employs a Corpus Adaptation Adversarial Module (CAAM) by combining CPAC with Margin Disparity Discrepancy (MDD), which can learn the domain-invariant emotion representations through extracting the strong emotion commonness. Experiments including ablation studies and visualizations on both single- and cross-corpus tasks using four well-known SER datasets in different languages are conducted for performance evaluation and comparison. The results indicate that in both tasks the CTL-MTNet showed better performance in all cases compared to a number of state-of-the-art methods. The source code and the supplementary materials are available at: https://github.com/MLDMXM2017/CTLMTNet.
BibTeX
@inproceedings{wen2022ctl,
  author = {Xin-Cheng Wen and Jiaxin Ye and Yan Luo and Yong Xu and Xuan-Ze Wang and Chang-Li Wu and Kun-Hong Liu},
  title = {{CTL-MTNet: A Novel CapsNet and Transfer Learning-Based Mixed Task Net for Single-Corpus and Cross-Corpus Speech Emotion Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=XCDLh6nExP},
}
Figure