Speaker Normalization for Self-Supervised Speech Emotion Recognition

Itai Gat, Hagai Aronowitz, Weizhong Zhu, Edmilson da Silva Morais, Ron Hoory

TL;DR

Researchers at IBM Research AI have developed a way to make Speech Emotion Recognition (SER) more robust by forcing models to "forget" who is speaking. By using a gradient-based adversarial learning framework, they strip speaker-specific characteristics from self-supervised speech representations, achieving new state-of-the-art results on the IEMOCAP benchmark, especially in challenging speaker-independent and low-resource scenarios.

What Problem Does This Paper Address?

Speech Emotion Recognition is notoriously difficult because high-quality labeled datasets are small and often contain "spurious correlations." For example, if a specific speaker in a dataset is frequently angry, a deep learning model might learn to associate that person’s unique voice pitch or accent with "anger" rather than learning the actual acoustic cues of the emotion.

These "shortcuts" allow the model to perform well on training data but cause it to fail miserably when encountering a new speaker (generalization). The authors aim to solve this by performing Speaker Normalization: disentangling the emotional content of a voice from the identity of the speaker.

Methods

The authors build upon the popular Upstream-Downstream architecture used in self-supervised learning (SSL).

  1. The Upstream Model: They use HuBERT, a massive model pre-trained on unlabeled speech, to generate rich feature representations.
  2. The Downstream Tasks: They attach two separate "heads" to the upstream model:
  3. An Emotion Recognition head (the primary goal).
  4. A Speaker Identification (SID) head (the adversary).
  5. Adversarial Gradient Negation: This is the core innovation. During training, the model learns to recognize emotions normally. However, for the Speaker ID task, the authors negate the gradients flowing back into the upstream model.

Essentially, the model is told: "Update your features to be great at recognizing emotions, but if those features help you identify the speaker, change them so you become worse at it." This forces the upstream model to produce "speaker-agnostic" features.

The authors tested two training strategies:

  • SNP (Speaker Normalization Projector): Adding a small non-linear layer between the upstream and downstream models to act as a "gate."
  • TAP (Train All Parameters): Fine-tuning the entire massive HuBERT model alongside the downstream tasks.

Datasets

  • IEMOCAP: The primary benchmark for Speech Emotion Recognition. It contains recordings of actors performing scripted and improvised scenes, labeled with emotions like neutral, happy, sad, and angry.
  • VoxCeleb: A large-scale dataset used specifically for the Speaker Identification task. It contains over 100,000 utterances from 1,251 different speakers.

Crucially, the authors note that the data used for emotion recognition doesn't need to be the same as the data used for speaker identification, making the framework highly flexible.

Key Insights and Ideas

  • Identity is a Distraction: The paper confirms that speaker identity is a "shortcut" that hinders emotion recognition. When they forced the model to be bad at identifying speakers (dropping SID accuracy from 60.7% to 45.9%), the emotion recognition performance actually went up.
  • Self-Supervision + Adversarial Learning: While self-supervised models like HuBERT provide a great starting point, they still capture "undesired" traits. Adversarial training is a powerful "filter" to refine these general-purpose representations for specific sensitive tasks.
  • Robustness in the "Small Data" Regime: The method is particularly effective when training data is scarce. By preventing the model from over-fitting to the few speakers available in a small dataset, the model learns more universal emotional cues.

Key Results

The proposed method achieved new State-of-the-Art (SOTA) results on the IEMOCAP dataset:

  • Speaker-Independent (5-fold cross-validation): Reached 74.2% weighted accuracy, outperforming the previous SOTA by 2.3% absolute.
  • Speaker-Dependent: Reached 81.0% weighted accuracy, a 0.5% improvement over previous benchmarks.
  • Low-Resource Performance: In experiments with very few samples per class (e.g., only 100-300 samples), the "Ours (TAP)" method consistently outperformed the standard HuBERT baseline, showing much better scaling as data size increases.

Limitations and Future Work

  • Computational Overhead: Fine-tuning "All Parameters" (TAP) on models like HuBERT Large (317M+ parameters) is computationally expensive, though the authors suggest the "Projector" (SNP) method as a lighter alternative.
  • Defining "Characteristics": While the paper focuses on speaker identity, other biases (like gender, age, or background noise) could also act as shortcuts. The authors suggest their framework is general enough to normalize these other cues in the future.
  • Dataset Diversity: The study focuses heavily on IEMOCAP. Testing on more diverse, "in-the-wild" emotion datasets would further validate how well this normalization handles real-world variability.

Summary updated

Abstract

Large speech emotion recognition datasets are hard to obtain, and small datasets may contain biases. Deep-net-based classifiers, in turn, are prone to exploit those biases and find shortcuts such as speaker characteristics. These shortcuts usually harm a model’s ability to generalize. To address this challenge, we propose a gradient-based adversary learning framework that learns a speech emotion recognition task while normalizing speaker characteristics from the feature representation. We demonstrate the efficacy of our method on both speaker-independent and speaker-dependent settings and obtain new state-of-the-art results on the challenging IEMOCAP dataset.
BibTeX
@inproceedings{gat2022speaker,
  author = {Itai Gat and Hagai Aronowitz and Weizhong Zhu and Edmilson da Silva Morais and Ron Hoory},
  title = {{Speaker Normalization for Self-Supervised Speech Emotion Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=rfIm_dqB6W},
}

Loading related content...

Figure