Advancing Emotion Recognition via Ensemble Learning: Integrating Speech, Context, and Text Representations

Xiaohan Shi, Jinyi Mi, Xingfeng Li, Tomoki TodaInterspeech 2025

TL;DR

This paper introduces a novel Speech-Context-Text (SCT) model that combines audio, contextual information, and transcribed text to improve speech emotion recognition (SER) in real-world scenarios. By integrating advanced techniques like Mamba for speech, layer adapters for context, and ASR correction for text, the SCT model achieved a 6th place ranking in the INTERSPEECH 2025 challenge and significant performance gains on benchmark datasets.

What Problem Does This Paper Address?

Speech Emotion Recognition (SER) aims to understand human emotions from spoken language, a crucial capability for more natural human-computer interactions. However, real-world speech is often messy, filled with background noise, overlapping speech, and other distractions that degrade performance. While previous research has focused on making individual components (like speech processing or text analysis) more robust to noise, integrating these different modalities in a noise-robust way for SER has been underexplored. This paper tackles the challenge of building a more robust SER system by effectively combining information from speech acoustics, contextual cues, and the semantic content of the speech, all while mitigating the impact of real-world noise.

Methods

The core of the proposed solution is the Speech-Context-Text (SCT) model, an ensemble learning approach that fuses representations from three distinct modules:

  • Speech-based Emotion Recognition: This module uses WavLM, a powerful self-supervised learning model pre-trained on vast amounts of speech data, to extract robust acoustic features. To further enhance its ability to model long-range dependencies and aggregate information, they incorporate a Mamba layer.
  • Context-based Emotion Recognition: Here, the Whisper model, known for its strong performance in speech recognition, is employed. To better capture contextual emotional nuances, they use a layer adapter mechanism applied to both the encoder and decoder layers of Whisper. This adapter helps refine the intermediate representations.
  • Text-based Emotion Recognition: This module leverages RoBERTa, a robustly optimized BERT pre-training approach, for text analysis. Crucially, to combat errors introduced by Automatic Speech Recognition (ASR) in noisy conditions, they apply an ASR correction method (using LanguageTool) to improve the accuracy of the transcribed text before feeding it to RoBERTa.

Finally, these three modality-specific representations are concatenated and fed into a simple FC layer followed by a SoftMax activation to produce the final emotion prediction. The model is trained using a cross-entropy loss function.

Datasets

The paper evaluates the proposed SCT model on two datasets:

  • MSP-Podcast Corpus: This dataset consists of naturalistic podcast recordings annotated with eight categorical emotion labels. The authors used the official training and validation partitions, with the validation set serving as the test set due to the unavailability of the official test set labels.
  • Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset: A widely used benchmark for affective computing, IEMOCAP contains audio-visual recordings of dyadic conversations. For this study, they focused on four categorical emotions (neutral, happiness, sadness, anger) and merged happiness and excitement into a single class, following common practice. They employed a leave-one-session-out cross-validation strategy.

Key Insights and Ideas

  • Multimodal Fusion is Key: The results strongly suggest that combining speech, context, and text representations significantly outperforms any single modality for SER, especially in naturalistic conditions.
  • Advanced Modality-Specific Enhancements Pay Off: Simply using off-the-shelf models for each modality isn't enough. The specific enhancements – Mamba for speech, layer adapters for context, and ASR correction for text – demonstrably improve the performance of each individual component.
  • ASR Correction is Crucial for Text: Noisy speech directly impacts ASR accuracy, which in turn can corrupt the emotional signal in text. The ASR correction step proves vital for extracting reliable textual emotion cues.
  • Contextual Information from Whisper is Valuable: Using Whisper's encoder and decoder outputs, refined by layer adapters, provides a rich source of contextual emotional information that complements acoustic and textual data.

Key Results

  • INTERSPEECH 2025 Challenge: The SCT model achieved 6th place in the Speech Emotion Recognition in Naturalistic Conditions Challenge, demonstrating its state-of-the-art performance.
  • Performance Gains:
    • On the MSP-Podcast dataset, the full SCT model achieved 48.64% UA and 41.54% F1, outperforming the best baseline by 12.9% UA and 7.37% F1.
  • On the IEMOCAP dataset, the SCT model achieved 79.89% UA and 79.88% F1, showing an improvement of 8.02% UA and 7.95% F1 over the best unimodal baseline.
  • Modality Contributions: The paper shows that individual modality enhancements also yield significant gains: WavLM with Mamba improved UA by 6.48% on MSP-Podcast, and layer adapters on Whisper improved UA by 1.60%. ASR correction on RoBERTa led to improvements of 0.69% UA on MSP-Podcast.

Limitations and Future Work

  • Limitations: While the paper demonstrates strong performance, the authors don't explicitly detail limitations of the SCT model itself. However, the reliance on ASR for text input inherently means that extremely poor transcription quality might still pose a challenge.
  • Future Work: The authors suggest exploring additional paralinguistic information, such as speaker identity and gender, to further enhance SER performance. This indicates a direction towards even richer multimodal fusion and more personalized emotion recognition.

Summary updated

BibTeX
@inproceedings{shi2025advancing,
  author = {Xiaohan Shi and Jinyi Mi and Xingfeng Li and Tomoki Toda},
  title = {{Advancing Emotion Recognition via Ensemble Learning: Integrating Speech, Context, and Text Representations}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure