Speech Emotion Recognition Using Self-Supervised Features

Edmilson da Silva Morais, Ron Hoory, Weizhong Zhu, Itai Gat, Matheus Damasceno, Hagai Aronowitz

This paper from researchers at IBM Research AI explores how self-supervised learning (SSL) models—which have already revolutionized natural language processing and speech recognition—can be specifically tuned to master the subtle art of Speech Emotion Recognition (SER).

TL;DR

The researchers introduce a modular "Upstream + Downstream" framework for emotion recognition that leverages fine-tuned self-supervised features like Wav2Vec 2.0 and huBERT. Their monomodal (audio-only) system achieves state-of-the-art results on the IEMOCAP dataset, proving that a well-tuned audio system can rival complex multimodal systems that use both speech and text.

What Problem Does This Paper Address?

Speech Emotion Recognition (SER) is notoriously difficult because human emotions are complex, subjective, and often expressed through a mix of vocal characteristics and linguistic content. Historically, the best systems have been "multimodal," requiring both the audio signal and a text transcript to perform well.

However, text isn't always available or accurate in real-world scenarios (like real-time call center analysis). Furthermore, while self-supervised models have dominated other speech tasks, their application in SER has been less explored. The authors aim to bridge this gap by creating a modular system that can easily integrate various self-supervised features and determine if audio alone can reach the performance levels of audio-plus-text systems.

Methods

The authors utilize a modular Upstream + Downstream architecture:

  • Upstream Model: A task-independent, pre-trained model (either Wav2Vec 2.0 or huBERT) that acts as a feature extractor. Crucially, the authors don't just use these models "out of the box"; they fine-tune them specifically for emotion recognition.
  • Downstream Model: A task-dependent "back-end" that classifies the features. They tested two types of aggregators to turn frame-level data into utterance-level embeddings: a simple Mean Average Pooling and the more sophisticated ECAPA-TDNN (typically used in speaker verification).
  • Checkpoint Averaging: To reduce variance and boost performance, they averaged the top 5 checkpoints of their models during training.
  • Fusion Strategies: They experimented with "Early Fusion" (combining Wav2Vec 2.0 and huBERT features before the aggregator) and "Late Fusion" (combining them after the embeddings were generated).

Datasets

The study uses the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset.

  • Content: ~12 hours of multimodal dyadic conversations.
  • Labels: Categorical emotions: "angry," "happy" (merged with "excited"), "sad," and "neutral."
  • Evaluation: 5-fold cross-validation (leave-one-session-out) to ensure the model generalizes across different speakers.

Key Insights and Ideas

  • SSL is a Game Changer: Self-supervised features (Wav2Vec 2.0/huBERT) vastly outperform traditional hand-crafted features like Filter-banks (Fbanks). The gap isn't just a few percentage points; it's a massive leap in accuracy.
  • The Power of Fine-Tuning and Averaging: Simply averaging the best checkpoints provided a significant boost (over 2% improvement in accuracy), suggesting that SER models are sensitive to training variance.
  • Early Fusion is Superior: Combining the raw features of huBERT and Wav2Vec 2.0 before they hit the aggregator (Early Fusion) yielded better results than combining them at the very end (Late Fusion).
  • Audio-Only Can Compete: The most significant finding is that their audio-only system outperformed several existing systems that use both audio and ground-truth text transcripts. This suggests that SSL models are capturing much of the "linguistic" emotional context directly from the acoustic signal.

Key Results

  • Best Performance: The system using early fusion of huBERT and Wav2Vec 2.0 with an ECAPA-TDNN aggregator achieved an Unweighted Accuracy (UACC) of 77.76%.
  • SOTA Comparison: This result sets a new benchmark for audio-only SER on IEMOCAP, significantly beating the previous audio-only SOTA of 73.30%.
  • Vs. Multimodal: Their audio-only result (77.76%) is remarkably close to the SOTA multimodal (Audio + Text) result of 78.30%, despite not having access to text transcripts.

Limitations and Future Work

The authors acknowledge that while their results are strong, they are currently limited to a single dataset (IEMOCAP). Future work includes:

  • Multitask Learning: Exploring if training the model on multiple tasks simultaneously (e.g., emotion + speaker ID) improves robustness.
  • Expanding Modalities: Integrating this SSL-based framework into a true multimodal system (Audio + Text + Vision) to see if it can push the state-of-the-art even further.
  • Unlabeled Data: Leveraging even larger amounts of unlabeled data to pre-train the acoustic features specifically for the nuances of emotional speech.

Summary updated

Abstract

Self-supervised pre-trained features have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of speech emotion recognition (SER) still need further investigation. In this paper we introduce a modular End-to-End (E2E) SER system based on an Upstream + Downstream architecture paradigm, which allows easy use/integration of a large variety of self-supervised features. Several SER experiments for predicting categorical emotion classes from the IEMOCAP dataset are performed. These experiments investigate interactions among fine-tuning of self-supervised feature models, aggregation of frame-level features into utterance-level features and back-end classification networks. The proposed monomodal speech-only based system not only achieves SOTA results, but also brings light to the possibility of powerful and well fine-tuned self-supervised acoustic features that reach results similar to the results achieved by SOTA multimodal systems using both Speech and Text modalities.
BibTeX
@inproceedings{morais2022speech,
  author = {Edmilson da Silva Morais and Ron Hoory and Weizhong Zhu and Itai Gat and Matheus Damasceno and Hagai Aronowitz},
  title = {{Speech Emotion Recognition Using Self-Supervised Features}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=TdKnZhHG0P},
}

Loading related content...

Figure