Abstract
Speech emotion recognition (SER) is a critical component for developing human-like intelligent agents. However, the scarcity of emotional data often necessitates leveraging source corpora to improve performance on target domains. The primary challenge in this transfer is domain shift—differences in recording conditions, speakers, and emotion types. This paper proposes Class-wise Adversarial Domain Adaptation (CADA) to mitigate these shifts by aligning specific emotion classes across domains. Our results on the EMODB and Aibo corpora demonstrate that CADA significantly outperforms existing methods, particularly in few-shot scenarios with extremely limited target data.
Introduction: The Challenge of Domain Shift in SER
The effectiveness of SER models is frequently hampered by domain shift. When a model trained on one dataset (e.g., adult actors in a studio) is applied to another (e.g., children in a spontaneous interaction), performance drops due to high intra-class variability. Acoustic features of "anger" or "happiness" vary wildly across different demographics and recording environments.
Traditional domain adaptation techniques often attempt to align the global distribution of data, which can be insufficient for the nuanced task of emotion recognition. Furthermore, many real-world applications lack the large unlabeled datasets required by standard unsupervised methods. CADA addresses this by focusing on class-by-class alignment in a supervised or semi-supervised setting.
Methodology: Class-wise Adversarial Domain Adaptation (CADA)
CADA addresses domain shift by performing alignment at the class level rather than the global level. The architecture consists of a feature encoder ($G_e$) and a predictor ($G_p$).
The training process is divided into two main stages:
- Supervised Learning: The model is initially trained on both source and target data. To facilitate class-wise alignment, the data is re-labeled into four distinct categories: (Source-Class 1, Source-Class 2, Target-Class 1, Target-Class 2). This allows the predictor to learn boundaries between domains and classes simultaneously.
- Adversarial Alignment: In this stage, the predictor's weights are frozen. The encoder is then trained to "confuse" the predictor by mapping source features into the target feature space for each corresponding class. For example, the encoder is optimized to make "Source-Class 1" features indistinguishable from "Target-Class 1" features to the predictor. This forces the model to learn domain-invariant emotional representations.
Experimental Setup
The proposed method was evaluated using three distinct German speech datasets:
- EMODB: Contains acted emotional speech from 10 adults.
- Aibo (Ohm and Mont subsets): Features spontaneous emotional speech from children interacting with a robot.
The transition from EMODB (acted/adults) to Aibo (spontaneous/children) represents a significant domain gap, testing the robustness of the transfer learning approach.
Key Results and Discussion
Performance was evaluated using Unweighted Accuracy (UA %), which accounts for class imbalances in the datasets. CADA was compared against standard fine-tuning and the state-of-the-art Few-shot Adversarial Domain Adaptation (FADA).
CADA demonstrated superior performance, especially in "few-shot" scenarios where only 2 to 12 labeled examples per class were available in the target domain.
As shown in the table above, when transferring from Aibo-Ohm to EMODB with only 2 labeled examples, CADA achieved a UA of 58%, outperforming fine-tuning (55%) and FADA (54%). As the number of target examples increased to 12, CADA's performance rose to 64%, while FADA remained stagnant at 55%. This indicates that CADA is more efficient at utilizing limited target data to bridge the domain gap by forcing the encoder to find a feature space where emotions from different domains overlap perfectly.
Conclusion
Class-wise Adversarial Domain Adaptation (CADA) provides a robust framework for speech emotion recognition across disparate corpora. By focusing on class-specific alignment, the method effectively handles high intra-class variability and domain shift. Future work will explore extending CADA to multi-class emotion recognition and developing unsupervised versions of the algorithm to eliminate the need for target labels entirely.
Summary updated