Robust Speech Emotion Recognition Under Different Encoding Conditions

Christopher Oates, Andreas Triantafyllopoulos, Ingmar Steiner, Björn W. SchullerInterspeech 2019

TL;DR

This paper investigates how audio compression affects speech emotion recognition (SER) systems. They found that while traditional SER models struggle with mismatched audio compression between training and testing, modern end-to-end deep learning approaches show remarkable robustness, making them more suitable for real-world applications where audio quality can vary.

What Problem Does This Paper Address?

Collecting large, high-quality, and naturally expressed emotional speech datasets is a significant challenge. Consequently, researchers often turn to readily available online audio, which is frequently compressed to save storage and bandwidth. This compression can degrade audio quality. The critical problem this paper addresses is understanding how this audio compression impacts the performance of SER systems, especially when the audio used for training an SER model has a different compression format or bitrate than the audio it will encounter in a real-world deployment (mismatched conditions). This is crucial as SER systems move from research labs into production environments where audio quality is often unpredictable.

Methods

The authors compare two main approaches to SER:

  • Traditional Machine Learning (openSMILE & SVM): This method involves extracting hand-crafted acoustic features using the openSMILE toolkit and then classifying these features with Support Vector Machines (SVMs). They evaluated several feature sets (ComParE, IS09_emotion, eGeMAPS, emo_large) and used SVMs with a linear kernel.
  • End-to-End Deep Learning: This approach uses convolutional and Long Short-Term Memory (LSTM) layers to jointly learn feature extraction and classification. The network is trained to predict the arousal dimension of emotion.

They tested three experimental scenarios:

  1. Matched Conditions: Training and testing on audio compressed with the same codec and bitrate.
  2. Mismatched Conditions: Training on uncompressed audio and testing on compressed audio.
  3. Multi-condition: Training on a mix of uncompressed and compressed audio, and testing on compressed audio with a different bitrate than those used in training.

Datasets

The study utilized four standard emotion datasets:

  • EMO-DB: German, acted emotions.
  • ENTERFACE: English, acted emotions.
  • Polish-Emo: Polish, acted emotions.
  • RECOLA: French, spontaneous, long-term interactions.

All datasets were resampled to 16 kHz to ensure a consistent number of bits per sample.

Key Insights and Ideas

  • Compression Doesn't Kill All Information: Even at low bitrates, compressed audio retains enough emotional information for robust SER, especially when training and testing conditions match.
  • Mismatched Compression is a Major Hurdle for Traditional Models: When SER models trained on clean, uncompressed audio encounter compressed audio (especially at lower bitrates), their performance drops significantly. This highlights a critical gap for deploying traditional SER systems in the wild.
  • End-to-End Models are Surprisingly Resilient: Deep learning models that learn features and classify end-to-end demonstrate a remarkable ability to maintain performance even under various compression conditions, suggesting they learn more fundamental, compression-invariant emotional cues.
  • Codec and Bitrate Matter (and Data Matters Too): The AAC codec generally performed worst in mismatched conditions. The Polish-EMO dataset showed particular sensitivity to the OPUS codec due to a DC offset issue.
  • Augmentation is Key for Traditional Models: For traditional SVM-based models, augmenting the training data with even a few compressed audio samples (e.g., two different bitrates) can significantly improve robustness in mismatched and multi-condition scenarios.

Key Results

  • Matched Conditions: For most feature sets and codecs, UAR (Unweighted Average Recall) remained comparable to uncompressed audio when training and testing on the same compressed format, especially for bitrates above 64 kbit/s.
  • Mismatched Conditions: Traditional models (openSMILE+SVM) showed a large drop in UAR on compressed test sets. AAC was particularly problematic, and MP3 required bitrates above 96 kbit/s for good performance.
  • Multi-condition: Augmenting training data with compressed samples significantly improved performance for traditional models, especially for AAC and low-bitrate MP3. For example, AAC performance improved by up to 35% on average.
  • End-to-End: The end-to-end approach showed only marginal differences in Concordance Correlation Coefficient (CCC) results on the RECOLA dataset under various encodings and bitrates, indicating high robustness.

Limitations and Future Work

  • Codec Specificity: The study noted that the OPUS codec's performance was significantly impacted by a DC offset in the Polish-EMO dataset, suggesting that specific audio characteristics can interact with codecs.
  • Feature Set Impact: While end-to-end models were robust, the choice of feature sets still influenced the performance of traditional models.
  • Future Work: The authors suggest that end-to-end architectures are a promising direction for robust SER in real-world, variable-quality audio environments. Further investigation into how specific audio characteristics interact with different codecs and how to further optimize multi-condition training for traditional models could be beneficial.

Summary updated

BibTeX
@inproceedings{oates2019robust,
  author = {Christopher Oates and Andreas Triantafyllopoulos and Ingmar Steiner and Björn W. Schuller},
  title = {{Robust Speech Emotion Recognition Under Different Encoding Conditions}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure