MEDUSA: A Multimodal Deep Fusion Multi-Stage Training Framework for Speech Emotion Recognition in Naturalistic Conditions

Georgios Chatzichristodoulou, Despoina Kosmopoulou, Antonios Kritikos, Anastasia Poulopoulou, Efthymios Georgiou, Athanasios Katsamanis, Vassilis Katsouros, Alexandros Potamianos

TL;DR

MEDUSA is a state-of-the-art multimodal framework for Speech Emotion Recognition (SER) that won 1st place in the Interspeech 2025 SER in Naturalistic Conditions Challenge. By combining a novel deep transformer-based fusion architecture (DeepSER) with a robust four-stage training pipeline, it successfully navigates the dual challenges of real-world data: severe class imbalance and human annotation ambiguity.

What Problem Does This Paper Address?

Recognizing emotions from speech in the real world is incredibly messy. Unlike clean, acted datasets where actors clearly portray specific emotions, naturalistic speech (like podcasts or phone calls) introduces two massive hurdles:

  1. Emotion Ambiguity: Human emotions are subjective. One annotator might hear a clip as "happy," while another hears it as "neutral." Standard machine learning approaches force these into rigid "one-hot" labels, throwing away this valuable nuance.
  2. Class Imbalance: In everyday conversation, people are neutral or calm far more often than they are disgusted or terrified. This causes models to over-predict majority classes and ignore minority ones.

The authors introduce MEDUSA to tackle these real-world complexities head-on, moving beyond simplistic late-fusion techniques and standard hard-label classification.

Methods

MEDUSA’s architecture relies on a two-pronged strategy: a deep multimodal fusion model called DeepSER, and a four-stage training pipeline.

Figure 1: An overview of MEDUSA. Different modalities are combined on each model using DeepSER. In total, 14 differ- ent such models are trained leveraging 2 or 3 modalities as their backbone. The training process employs standard train- ing (Stage 1) on the full training dataset, balanced subset train- ing (Stage 2) and a linear meta-classifier on top for ensembling (Stage 3). Six different runs of the meta-classifier are merged into a model soup by averaging the weights (Stage 4).

1. Feature Extraction & DeepSER

Instead of relying on a single feature type, the framework extracts acoustic features using self-supervised learning (SSL) models (WavLM, HuBERT, Whisper-v3) and linguistic features by transcribing the audio via Whisper ASR and passing it to language models (RoBERTa, ModernBERT).

Rather than simply concatenating these features at the very end (late fusion), DeepSER uses a hierarchical transformer encoder setup. It fuses intermediate representations across multiple layers, allowing the model to learn complex, cross-modal relationships between how something is said and what is being said.

2. The Four-Stage Training Pipeline

  • Stage 1 (Standard Training): DeepSER models are trained on the full dataset using soft targets (the actual distribution of annotator votes) rather than hard labels. It also uses Manifold MixUp for regularization and a multitask loss that simultaneously predicts categorical emotions and continuous emotional attributes (valence, arousal, dominance).
  • Stage 2 (Balanced Training): Training continues, but switches to randomly sampled, balanced subsets of data to force the model to learn rare emotion categories.
  • Stage 3 (Meta-Classifier): An ensemble of 14 distinct DeepSER configurations is formed. A linear meta-classifier is trained on top of their posterior distributions using balanced sampling to optimally combine their strengths.
  • Stage 4 (Model Soup): To maximize robustness, the weights of meta-classifiers trained across six different random seeds are averaged together into a final "model soup."

Datasets

The framework was trained and evaluated on the MSP-Podcast corpus (specifically the version curated for the Interspeech 2025 SER Challenge). This dataset consists of audio recordings, transcriptions, speaker metadata, and detailed annotator decisions. Crucially, it includes roughly 20,000 samples that lack a clear annotator consensus—samples that traditional pipelines would typically discard, but which MEDUSA leverages via soft-target learning.

Key Insights and Ideas

  • Embrace the Disagreement: The ablation study revealed that replacing soft targets with traditional one-hot labels caused the largest performance drop (an 8.23% relative degradation in Macro-F1). Retaining the full annotator voting distribution helps the model map nuanced, realistic decision boundaries.
  • Deep Fusion Matters: Moving from DeepSER's hierarchical transformer fusion back to standard late fusion caused a 6.31% drop in performance, proving that early and mid-level cross-modal interactions are vital for capturing emotional context.
  • Ensembling via Meta-Classification: Traditional ensembling methods like majority or soft voting proved ineffective. A simple, trainable linear layer (the meta-classifier) trained on balanced data effectively learns which models to trust for specific emotion classes.

Key Results

MEDUSA secured 1st place in Task 1 (Categorical Emotion Recognition) of the Interspeech 2025 Challenge.

On the challenge's held-out test set, it achieved an exceptionally balanced performance across both majority and minority classes:

  • Macro-F1: 0.4316
  • Micro-F1: 0.4319
  • Accuracy: 0.4319

In internal testing, the full MEDUSA framework achieved a Macro-F1 of 0.472, significantly outperforming both the average single DeepSER model (0.413) and the top-performing individual DeepSER model (0.440).

Table 1: Configurations of MEDUSA’s DeepSER models and their (F1) score in our internal test-set. ID: refers to DeepSER model ID; MODEL: refers to SSL feature extractor; F1: macro- F1 score.

Limitations and Future Work

While highly effective, the authors note a few areas for improvement:

  • Unused Metadata: The current pipeline does not directly integrate speaker metadata (IDs, genders) or forced alignments into the feature space, which could provide helpful context.
  • Modality-Agnostic Augmentations: Exploring more generalized data augmentation techniques could further improve the model's ability to generalize to entirely unseen datasets.
  • Deeper Meta-Classifiers: The ensemble aggregation currently relies on a single linear layer; exploring deeper architectures or utilizing pre-logit representations could yield even better fusion results.

Summary updated

Abstract

SER is a challenging task due to the subjective nature of human emotions and their uneven representation under naturalistic conditions. We propose MEDUSA, a multimodal framework with a four-stage training pipeline, which effectively handles class imbalance and emotion ambiguity. The first two stages train an ensemble of classifiers that utilize DeepSER, a novel extension of a deep cross-modal transformer fusion mechanism from pretrained self-supervised acoustic and linguistic representations. Manifold MixUp is employed for further regularization. The last two stages optimize a trainable meta-classifier that combines the ensemble predictions. Our training approach incorporates human annotation scores as soft targets, coupled with balanced data sampling and multitask learning. MEDUSA ranked 1st in Task 1: Categorical Emotion Recognition in the Interspeech 2025: Speech Emotion Recognition in Naturalistic Conditions Challenge.
BibTeX
@inproceedings{chatzichristodoulou2025medusa,
  author = {Georgios Chatzichristodoulou and Despoina Kosmopoulou and Antonios Kritikos and Anastasia Poulopoulou and Efthymios Georgiou and Athanasios Katsamanis and Vassilis Katsouros and Alexandros Potamianos},
  title = {{MEDUSA: A Multimodal Deep Fusion Multi-Stage Training Framework for Speech Emotion Recognition in Naturalistic Conditions}},
  booktitle = {Interspeech},
  year = {2025},
  doi = {10.21437/interspeech.2025-2636},
}
Figure