Speech Emotion Recognition using Self-Supervised Features

Edmilson da Silva Morais, Ron Hoory, Weizhong Zhu, Itai Gat, Matheus Damasceno, Hagai Aronowitz

Abstract

Self-supervised pre-trained features have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of speech emotion recognition (SER) still need further investigation. In this paper we introduce a modular End-to-End (E2E) SER system based on an Upstream + Downstream architecture paradigm, which allows easy use/integration of a large variety of self-supervised features. Several SER experiments for predicting categorical emotion classes from the IEMOCAP dataset are performed. These experiments investigate interactions among fine-tuning of self-supervised feature models, aggregation of frame-level features into utterance-level features and back-end classification networks. The proposed monomodal speech-only based system not only achieves SOTA results, but also brings light to the possibility of powerful and well finetuned self-supervised acoustic features that reach results similar to the results achieved by SOTA multimodal systems using both Speech and Text modalities.

Introduction

Speech Emotion Recognition (SER) is a complex task because human emotions are conveyed through a subtle interplay of vocal characteristics (prosody) and linguistic content. Historically, the most effective systems have been multimodal, requiring both the raw audio signal and a text transcript to achieve high accuracy. However, text transcripts are not always available or reliable in real-world applications.

Researchers at IBM Research AI have addressed this by exploring how self-supervised learning (SSL) models—which have already revolutionized speech recognition—can be specifically tuned for SER. Their goal was to determine if a modular, audio-only system could reach the performance levels of complex systems that use both speech and text.

The Upstream + Downstream Framework

The core of the proposed system is a modular Upstream + Downstream architecture. This paradigm allows for the flexible integration of different pre-trained models and classification back-ends.

The modular Upstream + Downstream architecture paradigm for SER.
  • Upstream Model: A task-independent, pre-trained model (such as Wav2Vec 2.0 or huBERT) acts as the feature extractor. A key contribution of this work is the fine-tuning of these models specifically for emotion recognition, rather than using them as static extractors.
  • Downstream Model: This task-dependent component classifies the extracted features. It includes an aggregator to convert frame-level data into a fixed-length utterance-level embedding, followed by a Linear Classifier (LC).

Methodology and Implementation

The study utilized the IEMOCAP dataset, which contains approximately 12 hours of emotional dyadic conversations. The researchers tested several configurations to optimize the pipeline:

  • Aggregators: They compared simple Mean Average Pooling with the more sophisticated ECAPA-TDNN, an architecture typically used in speaker verification that is adept at capturing temporal nuances.
  • Checkpoint Averaging: By averaging the top five checkpoints during training, the team achieved a performance boost of over 2%, indicating that SER models are highly sensitive to training variance.
  • Fusion Strategies: The study explored "Early Fusion" (combining features from different SSL models before the aggregator) and "Late Fusion" (combining them after the embeddings were generated).
The proposed modular system utilizing early fusion of Wav2Vec 2.0 and huBERT features with an ECAPA-TDNN aggregator.

Key Results and Insights

The experiments yielded several significant findings that push the boundaries of monomodal SER:

  • SSL Superiority: Self-supervised features (Wav2Vec 2.0 and huBERT) vastly outperformed traditional hand-crafted features like Filter-banks (Fbanks), representing a massive leap in accuracy rather than a marginal gain.
  • State-of-the-Art Performance: The system using early fusion of huBERT and Wav2Vec 2.0 with an ECAPA-TDNN aggregator achieved an Unweighted Accuracy (UACC) of 77.76%. This sets a new benchmark for audio-only SER on IEMOCAP, significantly beating the previous audio-only SOTA of 73.30%.
  • Audio-Only vs. Multimodal: Perhaps most importantly, the audio-only result of 77.76% is remarkably close to the current multimodal (Audio + Text) SOTA of 78.30%. This suggests that fine-tuned SSL models capture much of the emotional context that was previously thought to require linguistic text analysis.

Conclusion

This research demonstrates that a well-tuned, monomodal audio system can rival complex multimodal architectures. By leveraging the power of self-supervised features and sophisticated aggregation techniques, the proposed framework offers a robust solution for real-world SER. Future work will explore multitask learning and the integration of vision to further advance multimodal SSL performance.

Summary updated

Abstract

Self-supervised pre-trained features have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of speech emotion recognition (SER) still need further investigation. In this paper we introduce a modular End-to- End (E2E) SER system based on an Upstream + Downstream architecture paradigm, which allows easy use/integration of a large variety of self-supervised features. Several SER experiments for predicting categorical emotion classes from the IEMOCAP dataset are performed. These experiments investigate interactions among fine-tuning of self-supervised feature models, aggregation of frame-level features into utterance-level features and back-end classification networks. The proposed monomodal speechonly based system not only achieves SOTA results, but also brings light to the possibility of powerful and well finetuned self-supervised acoustic features that reach results similar to the results achieved by SOTA multimodal systems using both Speech and Text modalities.
BibTeX
@inproceedings{morais2022speech,
  author = {Edmilson da Silva Morais and Ron Hoory and Weizhong Zhu and Itai Gat and Matheus Damasceno and Hagai Aronowitz},
  title = {{Speech Emotion Recognition using Self-Supervised Features}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=TdKnZhHG0P},
}
Figure