Abstract
Large speech emotion recognition datasets are hard to obtain, and small datasets may contain biases. Deep-net-based classifiers, in turn, are prone to exploit those biases and find shortcuts such as speaker characteristics. These shortcuts usually harm a model's ability to generalize. To address this challenge, we propose a gradient-based adversary learning framework that learns a speech emotion recognition task while normalizing speaker characteristics from the feature representation. We demonstrate the efficacy of our method on both speaker-independent and speaker-dependent settings and obtain new state-of-the-art results on the challenging IEMOCAP dataset.
Introduction
Speech Emotion Recognition (SER) is a critical task in human-computer interaction, yet it faces significant hurdles due to the scarcity of high-quality labeled data. Small datasets often suffer from "spurious correlations," where models learn to associate specific speaker identities with certain emotions rather than identifying universal acoustic emotional cues. For instance, if a specific speaker in a training set is frequently angry, a model might rely on that speaker's unique pitch or accent as a shortcut for "anger." This reliance on speaker-specific traits hinders the model's ability to generalize to new, unseen speakers. To mitigate this, we propose a method for Speaker Normalization, aimed at disentangling emotional content from speaker identity.
Methodology
Our approach leverages the Upstream-Downstream architecture common in self-supervised learning (SSL). We utilize HuBERT as the upstream model to generate rich, high-dimensional speech representations. To achieve speaker normalization, we attach two distinct downstream heads to the upstream model: an Emotion Recognition head and a Speaker Identification (SID) head.
The core innovation lies in the Adversarial Gradient Negation process. During training, the model is optimized to perform well on the emotion recognition task. Simultaneously, for the SID task, the gradients flowing back into the upstream model are negated. This adversarial setup forces the upstream model to produce "speaker-agnostic" features; it is essentially penalized if its features contain enough information to identify the speaker. We explored two implementation strategies: Speaker Normalization Projector (SNP), which uses a non-linear bottleneck layer, and Train All Parameters (TAP), which involves full fine-tuning of the HuBERT model.
Experimental Setup
We evaluated our framework using the IEMOCAP dataset, the standard benchmark for SER, which includes scripted and improvised emotional performances. For the adversarial speaker identification task, we utilized VoxCeleb, a large-scale dataset containing over 100,000 utterances from 1,251 speakers. A key advantage of this framework is its flexibility: the data used for emotion recognition does not need to overlap with the data used for speaker identification.
Results and Discussion
Our results demonstrate that forcing the model to "forget" speaker identity significantly improves emotion recognition performance. By reducing SID accuracy, the model is compelled to learn more robust emotional features. This is particularly evident in low-resource scenarios, where the model is less likely to overfit to the limited number of speakers available in small training sets.
As shown in the table above, our method achieved new State-of-the-Art (SOTA) results on IEMOCAP. In the speaker-independent (5-fold cross-validation) setting, the HuBERT Large + TAP configuration reached 74.2% weighted accuracy, a 2.3% absolute improvement over previous benchmarks. In speaker-dependent scenarios, we achieved 81.0% accuracy. Furthermore, our method showed superior scaling and robustness when training with very few samples per class.
Conclusion
By incorporating adversarial speaker normalization, we have developed a more robust framework for Speech Emotion Recognition that generalizes better to new speakers. While fine-tuning large models like HuBERT is computationally intensive, the performance gains—especially in data-constrained environments—are substantial. Future work will explore extending this normalization framework to other potential biases, such as gender, age, or environmental noise, to further enhance the reliability of SER systems in real-world applications.
Summary updated