X-Vectors Meet Emotions: A Study On Dependencies Between Emotion and Speaker Recognition

Raghavendra Pappagari, Tianzi Wang, Jesús Villalba, Nanxin Chen, Najim Dehak

TL;DR

This paper investigates the interplay between speaker recognition and speech emotion recognition (SER). They demonstrate that powerful speaker embeddings (x-vectors) can be effectively repurposed for SER via transfer learning, and conversely, that speaker verification performance is significantly impacted by the emotional state of the speaker.

What Problem Does This Paper Address?

The field of speech processing often treats speaker recognition and emotion recognition as distinct tasks. However, both rely on acoustic features like pitch and energy. This paper explores whether knowledge gained from one task can benefit the other, specifically by leveraging pre-trained speaker recognition models for SER and by examining how emotions affect speaker verification. The authors aim to bridge this gap, showing that these two seemingly different tasks are more interconnected than previously assumed.

Methods

The core of this work revolves around x-vectors, which are utterance-level embeddings learned by deep neural networks for speaker recognition. The authors employ a transfer learning approach for SER:

  1. Feature Extraction: They extract x-vectors from pre-trained speaker recognition models.
  2. SER with Linear Models: They first test the efficacy of using these extracted x-vectors with a simple linear classifier (logistic regression) for emotion recognition.
  3. Fine-tuning for SER: To further improve SER performance, they fine-tune the pre-trained x-vector models by replacing the speaker classification layer with an emotion classification layer.
  4. Speaker Verification Analysis: They also investigate the impact of emotion on speaker verification by creating trials with both same-emotion and cross-emotion utterances.

They compare two versions of pre-trained x-vector models: one trained on clean data ("ResNet-clean") and another trained with data augmentation ("ResNet-aug"). They also explore data augmentation for the SER task itself.

Datasets

The study utilizes three publicly available datasets for evaluation:

  • IEMOCAP: A multimodal dyadic conversational dataset with 10 speakers and 8 emotion classes (though the authors use a subset of 4: angry, sad, neutral, happy). Due to its size, 5-fold cross-validation with a leave-one-session-out strategy was employed.
  • MSP-Podcast: A corpus collected from podcast recordings, used for emotion classification with 5 emotion categories (angry, happy, sad, neutral, disgust). Standard training, development, and testing splits were used.
  • Crema-D: A multimodal dataset featuring 91 professional actors enacting specific emotions. For this study, 4 emotion categories were used (angry, happy, sad, neutral, discarding disgust and fear).

Key Insights and Ideas

  • X-vectors are Emotionally Aware: The study confirms that x-vectors, primarily trained for speaker identity, implicitly encode information about the speaker's emotional state. This makes them valuable features for SER.
  • Transfer Learning is Effective: Reusing speaker recognition models for SER yields significant improvements over baseline methods, demonstrating the power of transfer learning.
  • Fine-tuning Boosts SER: While pre-trained x-vectors are useful, fine-tuning the models specifically for emotion classification leads to even better SER performance.
  • Emotion Impacts Speaker Verification: Speaker verification systems are demonstrably sensitive to the emotional content of utterances. This sensitivity can lead to performance degradation, especially when enrollment and test utterances have different emotions.
  • Anger is a Challenge: The emotion "angry" consistently appears in the worst-performing cross-emotion trials for speaker verification across all datasets.

Key Results

  • SER Improvements: Fine-tuning pre-trained x-vector models resulted in substantial absolute improvements in weighted f-scores: 30.40% on IEMOCAP, 7.99% on MSP-Podcast, and 8.61% on Crema-D compared to baseline models without pre-training.
  • Augmentation's Role: Data augmentation for SER improved performance on IEMOCAP and Crema-D but showed less clear benefits on MSP-Podcast, suggesting varying robustness to noise across datasets. The unaugmented x-vector model ("ResNet-clean") was found to be less robust to noise.
  • Best SER Performance: The fine-tuned "ResNet-aug" model achieved the best SER results, reaching 70.30% on IEMOCAP (with augmentation) and 81.54% on Crema-D (with augmentation). On MSP-Podcast, the fine-tuned "ResNet-aug" model achieved 58.46% on clean training data.
  • Speaker Verification EERs: Error rates (EERs) for speaker verification were generally high, particularly on IEMOCAP due to short utterances. Cross-emotion trials consistently showed higher EERs than same-emotion trials.

Limitations and Future Work

  • Dataset Limitations: The authors acknowledge that IEMOCAP and Crema-D have limitations in terms of speaker and utterance counts, making direct comparison with other works challenging. MSP-Podcast is an ongoing effort, and comparisons with prior work on its current release are difficult.
  • Utterance Length: The short duration of utterances in datasets like IEMOCAP and MSP-Podcast contributes to higher EERs in speaker verification.
  • Future Work: The authors propose focusing on developing emotion-invariant speaker verification models as a key future direction. They hope their findings will inspire further research into the intricate relationship between speaker and emotion recognition.

Summary updated

Abstract

In this work, we explore the dependencies between speaker recognition and emotion recognition. We first show that knowledge learned for speaker recognition can be reused for emotion recognition through transfer learning. Then, we show the effect of emotion on speaker recognition. For emotion recognition, we show that using a simple linear model is enough to obtain good performance on the features extracted from pre-trained models such as the x-vector model. Then, we improve emotion recognition performance by finetuning for emotion classification. We evaluated our experiments on three different types of datasets: IEMOCAP, MSP-Podcast, and Crema-D. By fine-tuning, we obtained 30.40%, 7.99%, and 8.61% absolute improvement on IEMOCAP, MSP-Podcast, and Crema-D respectively over baseline model with no pre-training. Finally, we present results on the effect of emotion on speaker verification. We observed that speaker verification performance is prone to changes in test speaker emotions. We found that trials with angry utterances performed worst in all three datasets. We hope our analysis will initiate a new line of research in the speaker recognition community.
BibTeX
@inproceedings{pappagari2020x,
  author = {Raghavendra Pappagari and Tianzi Wang and Jesús Villalba and Nanxin Chen and Najim Dehak},
  title = {{X-Vectors Meet Emotions: A Study On Dependencies Between Emotion and Speaker Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=vLO8muJXWt},
}
Figure