Speaker Normalization for Self-supervised Speech Emotion Recognition

Itai Gat, Hagai Aronowitz, Weizhong Zhu, Edmilson da Silva Morais, Ron Hoory

Abstract

Large speech emotion recognition datasets are hard to obtain, and small datasets may contain biases. Deep-net-based classifiers, in turn, are prone to exploit those biases and find shortcuts such as speaker characteristics. These shortcuts usually harm a model's ability to generalize. To address this challenge, we propose a gradient-based adversary learning framework that learns a speech emotion recognition task while normalizing speaker characteristics from the feature representation. We demonstrate the efficacy of our method on both speaker-independent and speaker-dependent settings and obtain new state-of-the-art results on the challenging IEMOCAP dataset.

Introduction

Speech Emotion Recognition (SER) is a critical task in human-computer interaction, yet it faces significant hurdles due to the scarcity of high-quality labeled data. Small datasets often suffer from "spurious correlations," where models learn to associate specific speaker identities with certain emotions rather than identifying universal acoustic emotional cues. For instance, if a specific speaker in a training set is frequently angry, a model might rely on that speaker's unique pitch or accent as a shortcut for "anger." This reliance on speaker-specific traits hinders the model's ability to generalize to new, unseen speakers. To mitigate this, we propose a method for Speaker Normalization, aimed at disentangling emotional content from speaker identity.

Methodology

Our approach leverages the Upstream-Downstream architecture common in self-supervised learning (SSL). We utilize HuBERT as the upstream model to generate rich, high-dimensional speech representations. To achieve speaker normalization, we attach two distinct downstream heads to the upstream model: an Emotion Recognition head and a Speaker Identification (SID) head.

The proposed adversarial learning framework utilizes gradient negation to strip speaker-specific information from the upstream model's representations.

The core innovation lies in the Adversarial Gradient Negation process. During training, the model is optimized to perform well on the emotion recognition task. Simultaneously, for the SID task, the gradients flowing back into the upstream model are negated. This adversarial setup forces the upstream model to produce "speaker-agnostic" features; it is essentially penalized if its features contain enough information to identify the speaker. We explored two implementation strategies: Speaker Normalization Projector (SNP), which uses a non-linear bottleneck layer, and Train All Parameters (TAP), which involves full fine-tuning of the HuBERT model.

Experimental Setup

We evaluated our framework using the IEMOCAP dataset, the standard benchmark for SER, which includes scripted and improvised emotional performances. For the adversarial speaker identification task, we utilized VoxCeleb, a large-scale dataset containing over 100,000 utterances from 1,251 speakers. A key advantage of this framework is its flexibility: the data used for emotion recognition does not need to overlap with the data used for speaker identification.

Results and Discussion

Our results demonstrate that forcing the model to "forget" speaker identity significantly improves emotion recognition performance. By reducing SID accuracy, the model is compelled to learn more robust emotional features. This is particularly evident in low-resource scenarios, where the model is less likely to overfit to the limited number of speakers available in small training sets.

Comparison of the proposed method against existing state-of-the-art models on the IEMOCAP dataset, showing significant gains in both 5-fold and Speaker-Dependent (SD) settings.

As shown in the table above, our method achieved new State-of-the-Art (SOTA) results on IEMOCAP. In the speaker-independent (5-fold cross-validation) setting, the HuBERT Large + TAP configuration reached 74.2% weighted accuracy, a 2.3% absolute improvement over previous benchmarks. In speaker-dependent scenarios, we achieved 81.0% accuracy. Furthermore, our method showed superior scaling and robustness when training with very few samples per class.

Conclusion

By incorporating adversarial speaker normalization, we have developed a more robust framework for Speech Emotion Recognition that generalizes better to new speakers. While fine-tuning large models like HuBERT is computationally intensive, the performance gains—especially in data-constrained environments—are substantial. Future work will explore extending this normalization framework to other potential biases, such as gender, age, or environmental noise, to further enhance the reliability of SER systems in real-world applications.

Summary updated

Abstract

Large speech emotion recognition datasets are hard to obtain, and small datasets may contain biases. Deep-net-based classifiers, in turn, are prone to exploit those biases and find shortcuts such as speaker characteristics. These shortcuts usually harm a model's ability to generalize. To address this challenge, we propose a gradient-based adversary learning framework that learns a speech emotion recognition task while normalizing speaker characteristics from the feature representation. We demonstrate the efficacy of our method on both speaker-independent and speaker-dependent settings and obtain new state-of-the-art results on the challenging IEMOCAP dataset.
BibTeX
@inproceedings{gat2022speaker,
  author = {Itai Gat and Hagai Aronowitz and Weizhong Zhu and Edmilson da Silva Morais and Ron Hoory},
  title = {{Speaker Normalization for Self-supervised Speech Emotion Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=rfIm_dqB6W},
}
Figure