Transferable Positive/Negative Speech Emotion Recognition via Class-wise Adversarial Domain Adaptation

Hao Zhou, Ke Chen

Abstract

Speech emotion recognition (SER) is a critical component for developing human-like intelligent agents. However, the scarcity of emotional data often necessitates leveraging source corpora to improve performance on target domains. The primary challenge in this transfer is domain shift—differences in recording conditions, speakers, and emotion types. This paper proposes Class-wise Adversarial Domain Adaptation (CADA) to mitigate these shifts by aligning specific emotion classes across domains. Our results on the EMODB and Aibo corpora demonstrate that CADA significantly outperforms existing methods, particularly in few-shot scenarios with extremely limited target data.

Introduction: The Challenge of Domain Shift in SER

The effectiveness of SER models is frequently hampered by domain shift. When a model trained on one dataset (e.g., adult actors in a studio) is applied to another (e.g., children in a spontaneous interaction), performance drops due to high intra-class variability. Acoustic features of "anger" or "happiness" vary wildly across different demographics and recording environments.

Traditional domain adaptation techniques often attempt to align the global distribution of data, which can be insufficient for the nuanced task of emotion recognition. Furthermore, many real-world applications lack the large unlabeled datasets required by standard unsupervised methods. CADA addresses this by focusing on class-by-class alignment in a supervised or semi-supervised setting.

Methodology: Class-wise Adversarial Domain Adaptation (CADA)

CADA addresses domain shift by performing alignment at the class level rather than the global level. The architecture consists of a feature encoder ($G_e$) and a predictor ($G_p$).

The CADA architecture featuring an encoder and predictor, with gradient flows for supervised and adversarial training stages.

The training process is divided into two main stages:

  1. Supervised Learning: The model is initially trained on both source and target data. To facilitate class-wise alignment, the data is re-labeled into four distinct categories: (Source-Class 1, Source-Class 2, Target-Class 1, Target-Class 2). This allows the predictor to learn boundaries between domains and classes simultaneously.
  2. Adversarial Alignment: In this stage, the predictor's weights are frozen. The encoder is then trained to "confuse" the predictor by mapping source features into the target feature space for each corresponding class. For example, the encoder is optimized to make "Source-Class 1" features indistinguishable from "Target-Class 1" features to the predictor. This forces the model to learn domain-invariant emotional representations.

Experimental Setup

The proposed method was evaluated using three distinct German speech datasets:

  • EMODB: Contains acted emotional speech from 10 adults.
  • Aibo (Ohm and Mont subsets): Features spontaneous emotional speech from children interacting with a robot.

The transition from EMODB (acted/adults) to Aibo (spontaneous/children) represents a significant domain gap, testing the robustness of the transfer learning approach.

Key Results and Discussion

Performance was evaluated using Unweighted Accuracy (UA %), which accounts for class imbalances in the datasets. CADA was compared against standard fine-tuning and the state-of-the-art Few-shot Adversarial Domain Adaptation (FADA).

CADA demonstrated superior performance, especially in "few-shot" scenarios where only 2 to 12 labeled examples per class were available in the target domain.

Unweighted Accuracy (UA %) for few-shot domain adaptation from Aibo-Ohm (source) to EMODB (target).

As shown in the table above, when transferring from Aibo-Ohm to EMODB with only 2 labeled examples, CADA achieved a UA of 58%, outperforming fine-tuning (55%) and FADA (54%). As the number of target examples increased to 12, CADA's performance rose to 64%, while FADA remained stagnant at 55%. This indicates that CADA is more efficient at utilizing limited target data to bridge the domain gap by forcing the encoder to find a feature space where emotions from different domains overlap perfectly.

Conclusion

Class-wise Adversarial Domain Adaptation (CADA) provides a robust framework for speech emotion recognition across disparate corpora. By focusing on class-specific alignment, the method effectively handles high intra-class variability and domain shift. Future work will explore extending CADA to multi-class emotion recognition and developing unsupervised versions of the algorithm to eliminate the need for target labels entirely.

Summary updated

Abstract

Speech emotion recognition plays an important role in building more intelligent and human-like agents. Due to the difficulty of collecting speech emotional data, an increasingly popular solution is leveraging a related and rich source corpus to help address the target corpus. However, domain shift between the corpora poses a serious challenge, making domain shift adaptation difficult to function even on the recognition of positive/negative emotions. In this work, we propose class-wise adversarial domain adaptation to address this challenge by reducing the shift for all classes between different corpora. Experiments on the well-known corpora EMODB and Aibo demonstrate that our method is effective even when only a very limited number of target labeled examples are provided.
BibTeX
@inproceedings{zhou2018transferable,
  author = {Hao Zhou and Ke Chen},
  title = {{Transferable Positive/Negative Speech Emotion Recognition via Class-wise Adversarial Domain Adaptation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
}
Figure