TL;DR
MEDUSA is a state-of-the-art multimodal framework for Speech Emotion Recognition (SER) that won 1st place in the Interspeech 2025 SER in Naturalistic Conditions Challenge. By combining a novel deep transformer-based fusion architecture (DeepSER) with a robust four-stage training pipeline, it successfully navigates the dual challenges of real-world data: severe class imbalance and human annotation ambiguity.
What Problem Does This Paper Address?
Recognizing emotions from speech in the real world is incredibly messy. Unlike clean, acted datasets where actors clearly portray specific emotions, naturalistic speech (like podcasts or phone calls) introduces two massive hurdles:
- Emotion Ambiguity: Human emotions are subjective. One annotator might hear a clip as "happy," while another hears it as "neutral." Standard machine learning approaches force these into rigid "one-hot" labels, throwing away this valuable nuance.
- Class Imbalance: In everyday conversation, people are neutral or calm far more often than they are disgusted or terrified. This causes models to over-predict majority classes and ignore minority ones.
The authors introduce MEDUSA to tackle these real-world complexities head-on, moving beyond simplistic late-fusion techniques and standard hard-label classification.
Methods
MEDUSA’s architecture relies on a two-pronged strategy: a deep multimodal fusion model called DeepSER, and a four-stage training pipeline.
1. Feature Extraction & DeepSER
Instead of relying on a single feature type, the framework extracts acoustic features using self-supervised learning (SSL) models (WavLM, HuBERT, Whisper-v3) and linguistic features by transcribing the audio via Whisper ASR and passing it to language models (RoBERTa, ModernBERT).
Rather than simply concatenating these features at the very end (late fusion), DeepSER uses a hierarchical transformer encoder setup. It fuses intermediate representations across multiple layers, allowing the model to learn complex, cross-modal relationships between how something is said and what is being said.
2. The Four-Stage Training Pipeline
- Stage 1 (Standard Training): DeepSER models are trained on the full dataset using soft targets (the actual distribution of annotator votes) rather than hard labels. It also uses Manifold MixUp for regularization and a multitask loss that simultaneously predicts categorical emotions and continuous emotional attributes (valence, arousal, dominance).
- Stage 2 (Balanced Training): Training continues, but switches to randomly sampled, balanced subsets of data to force the model to learn rare emotion categories.
- Stage 3 (Meta-Classifier): An ensemble of 14 distinct DeepSER configurations is formed. A linear meta-classifier is trained on top of their posterior distributions using balanced sampling to optimally combine their strengths.
- Stage 4 (Model Soup): To maximize robustness, the weights of meta-classifiers trained across six different random seeds are averaged together into a final "model soup."
Datasets
The framework was trained and evaluated on the MSP-Podcast corpus (specifically the version curated for the Interspeech 2025 SER Challenge). This dataset consists of audio recordings, transcriptions, speaker metadata, and detailed annotator decisions. Crucially, it includes roughly 20,000 samples that lack a clear annotator consensus—samples that traditional pipelines would typically discard, but which MEDUSA leverages via soft-target learning.
Key Insights and Ideas
- Embrace the Disagreement: The ablation study revealed that replacing soft targets with traditional one-hot labels caused the largest performance drop (an 8.23% relative degradation in Macro-F1). Retaining the full annotator voting distribution helps the model map nuanced, realistic decision boundaries.
- Deep Fusion Matters: Moving from DeepSER's hierarchical transformer fusion back to standard late fusion caused a 6.31% drop in performance, proving that early and mid-level cross-modal interactions are vital for capturing emotional context.
- Ensembling via Meta-Classification: Traditional ensembling methods like majority or soft voting proved ineffective. A simple, trainable linear layer (the meta-classifier) trained on balanced data effectively learns which models to trust for specific emotion classes.
Key Results
MEDUSA secured 1st place in Task 1 (Categorical Emotion Recognition) of the Interspeech 2025 Challenge.
On the challenge's held-out test set, it achieved an exceptionally balanced performance across both majority and minority classes:
- Macro-F1: 0.4316
- Micro-F1: 0.4319
- Accuracy: 0.4319
In internal testing, the full MEDUSA framework achieved a Macro-F1 of 0.472, significantly outperforming both the average single DeepSER model (0.413) and the top-performing individual DeepSER model (0.440).
Limitations and Future Work
While highly effective, the authors note a few areas for improvement:
- Unused Metadata: The current pipeline does not directly integrate speaker metadata (IDs, genders) or forced alignments into the feature space, which could provide helpful context.
- Modality-Agnostic Augmentations: Exploring more generalized data augmentation techniques could further improve the model's ability to generalize to entirely unseen datasets.
- Deeper Meta-Classifiers: The ensemble aggregation currently relies on a single linear layer; exploring deeper architectures or utilizing pre-logit representations could yield even better fusion results.
Summary updated