CTL-MTNet: A Novel CapsNet and Transfer Learning-Based Mixed Task Net for the Single-Corpus and Cross-Corpus Speech Emotion Recognition

Xin-Cheng Wen, Jiaxin Ye, Yan Luo, Yong Xu, Xuan-Ze Wang, Chang-Li Wu, Kun-Hong Liu

TL;DR

This paper introduces CTL-MTNet, a novel deep learning framework that simultaneously tackles both single-corpus and cross-corpus Speech Emotion Recognition (SER). By combining a Capsule Network with self-attention (CPAC) for robust feature extraction and a Corpus Adaptation Adversarial Module (CAAM) for domain invariance, CTL-MTNet achieves state-of-the-art performance on challenging SER tasks.

What Problem Does This Paper Address?

Speech Emotion Recognition (SER) aims to identify emotions from spoken language, a crucial aspect of human-computer interaction. A significant challenge in SER is the domain shift problem: models trained on one dataset (source corpus) often perform poorly on data from a different dataset (target corpus). This is due to variations in speakers, recording environments, and even languages. Existing methods often struggle to generalize across these different data distributions, especially when aiming to perform well on both single-corpus (within-dataset) and cross-corpus (across-dataset) tasks simultaneously. This paper addresses this by proposing a unified framework capable of handling both scenarios effectively.

Methods

The core of the proposed solution is the CTL-MTNet (CapsNet and Transfer Learning-based Mixed Task Net). It comprises two main novel components:

  • CPAC (Convolution-Pooling and Attention-based CapsNet): This module is designed for the single-corpus SER task. It leverages a Capsule Network (CapsNet) architecture, enhanced with a self-attention mechanism. The self-attention guides the CapsNet to focus on the most informative features within different capsules, leading to more discriminative high-level representations. This is built upon a CNN-Pooling feature extractor.
  • CAAM (Corpus Adaptation Adversarial Module): This module is specifically designed to tackle the cross-corpus SER challenge. It combines the CPAC feature extractor with a Margin Disparity Discrepancy (MDD) loss function. CAAM aims to learn domain-invariant emotion representations by minimizing the discrepancy between source and target corpus features, effectively bridging the domain gap. It uses an adversarial approach to align these features.

The overall architecture integrates these components to process MFCC features extracted from audio data.

Datasets

The paper evaluates CTL-MTNet on four well-known SER datasets in different languages:

  • CASIA
  • EMODB
  • SAVEE
  • RAVDESS

These datasets are used for both single-corpus and cross-corpus evaluations. No new datasets were introduced.

Key Insights and Ideas

  • Unified Framework for Dual Tasks: The most significant contribution is the development of a single model capable of excelling in both single-corpus and cross-corpus SER. This is a novel approach that simplifies the SER landscape by avoiding the need for separate models for different scenarios.
  • Attention-Guided Capsules: Incorporating self-attention into CapsNet (CPAC) is crucial for focusing on salient features. This allows the model to dynamically weigh the importance of different capsule representations, leading to better discrimination of emotional states.
  • Domain Invariance through Adversarial Learning and MDD: The CAAM module, particularly with the MDD loss, effectively addresses the domain shift problem. By forcing the model to learn representations that are indistinguishable across different corpora, it achieves better generalization.
  • Capsule Networks for Emotion Representation: The paper further validates the effectiveness of Capsule Networks in capturing complex spatial relationships within speech features for emotion recognition, building on prior work.

Key Results

  • Single-Corpus Performance: CTL-MTNet (specifically the CPAC component) achieved state-of-the-art performance on the single-corpus SER tasks across all four datasets, outperforming existing methods. For example, it achieved a WAR of 92.75% on CASIA and 94.97% on EMODB.
  • Cross-Corpus Performance: The CTL-MTNet demonstrated significant improvements in cross-corpus SER compared to established methods like CDAN, DANN, and NMFTSL. It achieved an average UAR of 39.90% and WAR of 41.57%, showing substantial gains (e.g., +8.24% UAR and +10.96% WAR over CDAN).
  • Ablation Studies: Experiments confirmed the importance of each component. CPAC (with both CNN-Pooling and Self-Attention) significantly outperformed simpler CapsNet variants or models without attention. The CAAM module with MDD was essential for cross-corpus performance gains.
  • Visualization: t-SNE visualizations showed that CPAC creates clearer class separations in feature space for single-corpus tasks, while CAAM effectively aligns features from different corpora, reducing inter-corpus distance while maintaining inter-class separation.

Limitations and Future Work

  • Cross-Corpus Class Separation: While CAAM improves feature alignment, the paper notes that the separation between classes in the target corpus is not always as clear as desired, which can lead to reduced accuracy in some cross-corpus scenarios.
  • Future Work: The authors suggest exploring further improvements in feature alignment for cross-corpus tasks to enhance class separability in target domains. They also implicitly point towards the potential for applying this unified framework to other related audio processing tasks that suffer from domain shift.

Summary updated

Abstract

Speech Emotion Recognition (SER) has become a growing focus of research in human-computer interaction. An essential challenge in SER is to extract common attributes from different speakers or languages, especially when a specific source corpus has to be trained to recognize the unknown data coming from another speech corpus. To address this challenge, a Capsule Network (CapsNet) and Transfer Learning based Mixed Task Net (CTLMTNet) are proposed to deal with both the singlecorpus and cross-corpus SER tasks simultaneously in this paper. For the single-corpus task, the combination of Convolution-Pooling and Attention CapsNet module CPAC) is designed by embedding the self-attention mechanism to the CapsNet, guiding the module to focus on the important features that can be fed into different capsules. The extracted high-level features by CPAC provide sufficient discriminative ability. Furthermore, to handle the cross-corpus task, CTL-MTNet employs a Corpus Adaptation Adversarial Module (CAAM) by combining CPAC with Margin Disparity Discrepancy (MDD), which can learn the domain-invariant emotion representations through extracting the strong emotion commonness. Experiments including ablation studies and visualizations on both singleand cross-corpus tasks using four well-known SER datasets in different languages are conducted for performance evaluation and comparison. The results indicate that in both tasks the CTL-MTNet showed better performance in all cases compared to a number of state-of-the-art methods. The source code and the supplementary materials are available at: https://github.com/MLDMXM2017/CTLMTNet
BibTeX
@inproceedings{wen2022ctl,
  author = {Xin-Cheng Wen and Jiaxin Ye and Yan Luo and Yong Xu and Xuan-Ze Wang and Chang-Li Wu and Kun-Hong Liu},
  title = {{CTL-MTNet: A Novel CapsNet and Transfer Learning-Based Mixed Task Net for the Single-Corpus and Cross-Corpus Speech Emotion Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=XCDLh6nExP},
}
Figure