Cross-Corpus Speech Emotion Recognition with HuBERT Self-Supervised Representation

Miguel A. Pastor, Dayana Ribas, Alfonso Ortega, Antonio Miguel, Eduardo LleidaIberSPEECH 2022 · DOI

TL;DR

This paper explores using data from multiple speech emotion datasets (cross-corpus training) to improve speech emotion recognition (SER) systems, especially when dealing with limited labeled data. They found that combining datasets, even those in different languages, significantly boosts performance, particularly when using the powerful HuBERT self-supervised representation model.

What Problem Does This Paper Address?

Speech Emotion Recognition (SER) is crucial for more natural human-machine interaction, but its performance is often hampered by a lack of large, diverse, and realistically labeled emotional speech datasets. Current deep learning models require substantial training data, and existing datasets are often acted, subjective, and primarily in English. This paper tackles the challenge of improving SER system generalization and accuracy by investigating cross-corpus training as a data augmentation strategy, specifically with the advanced HuBERT self-supervised learning model.

Methods

The researchers employed two main approaches for SER:

  • Spectral-based system: This served as a baseline, using traditional handcrafted spectral features (GeMAPS, eGeMAPS, ComParE) fed into a Support Vector Machine (SVM) classifier.
  • DNN-based system: This is the core of their investigation, utilizing the HuBERT self-supervised model for speech representations. They then experimented with three different DNN classifiers on top of HuBERT embeddings:
  • A simple pooling layer followed by a linear layer.
  • A Convolutional Neural Network (CNN) with Self-Attention.
  • A Class Token (CT) Transformer.

The key novelty lies in evaluating HuBERT's performance within a cross-corpus training paradigm, where data from multiple datasets is pooled for training.

Datasets

The study utilized the following datasets:

  • EmoDB: A German acted emotion dataset.
  • RAVDESS: An English acted emotion dataset.
  • IEMOCAP: An English acted emotion dataset, notable for its improvised dialogues and external labeling, aiming for more realism.

For consistency, all datasets were reduced to four core emotions: neutral, happiness, anger, and sadness. Audio was standardized to mono-channel and a 16 kHz sample rate.

Key Insights and Ideas

  • Cross-corpus training is a powerful data augmentation technique: Combining data from different emotion datasets, even across languages, significantly improves SER performance compared to training on a single dataset.
  • HuBERT excels in cross-corpus SER: The self-supervised HuBERT model, when used with cross-corpus training, outperforms traditional spectral features and SVM classifiers, demonstrating its ability to generalize across diverse data.
  • Language barriers are less of an issue for DNNs: The study's findings align with previous work suggesting that DNNs can leverage cross-lingual data augmentation effectively, indicating that the model can learn robust representations despite language differences.
  • Complexity of DNNs matters: While the CT-Transformer is a more complex architecture, its performance was comparable to simpler DNNs, suggesting that such models might require even larger datasets to fully realize their potential.

Key Results

  • Spectral-based system: Using the "Extended Train" condition (all datasets combined for training) consistently outperformed the "Matched Train" condition (training and testing on the same dataset). The ComParE feature set showed the most significant improvement, exceeding 3% in UAR.
  • DNN-based system (HuBERT): The HuBERT-based system consistently outperformed the spectral-based system. For the IEMOCAP dataset, the "Extended Train" condition with HuBERT achieved a UAR of 67.58% with the CT-Transformer, which is approximately 3% higher than the best SVM with ComParE. Even the simplest HuBERT-LinearLayer model showed a 1.5% absolute improvement over the spectral-based baseline.

Limitations and Future Work

  • Dataset characteristics: The authors note that further research is needed to understand which specific properties of emotional datasets make them most suitable for cross-corpus combination.
  • Model complexity vs. data: The performance of the CT-Transformer suggests that more complex models might benefit from even larger training datasets than those used in this study.
  • Exploring other SS representations: The paper suggests investigating other self-supervised models that have shown even better performance on various speech tasks, such as WavLM, for future SER applications.

Summary updated

BibTeX
@inproceedings{pastor2022cross,
  author = {Miguel A. Pastor and Dayana Ribas and Alfonso Ortega and Antonio Miguel and Eduardo Lleida},
  title = {{Cross-Corpus Speech Emotion Recognition with HuBERT Self-Supervised Representation}},
  booktitle = {IberSPEECH},
  year = {2022},
  doi = {10.21437/iberspeech.2022-16},
}
Figure