M4SER: Multimodal, Multirepresentation, Multitask, and Multistrategy Learning for Speech Emotion Recognition

Jiajun He, Xiaohan Shi, Cheng-Hung Hu, Jinyi Mi, Xingfeng Li, Tomoki Toda

TL;DR

M4SER is a sophisticated framework for Speech Emotion Recognition (SER) that achieves state-of-the-art results by integrating audio and text while explicitly mitigating the "noise" of automatic speech recognition (ASR) errors. By combining multitask learning (error detection/correction) with adversarial and contrastive training strategies, the model learns to better disentangle and fuse modality-specific and modality-invariant features.

What Problem Does This Paper Address?

Speech Emotion Recognition is a core component of human-computer interaction, but it’s notoriously difficult. While using both audio and text (multimodal) is more effective than either alone, most systems rely on perfect, human-annotated transcripts. In real-world applications, we use ASR, which is prone to errors. These errors act as noise, degrading the performance of emotion classifiers.

Furthermore, effectively fusing audio and text is a challenge. Simple concatenation often fails to capture the subtle, fine-grained emotional dynamics present in long interactions. There is a constant tension between learning features that are unique to a modality (e.g., pitch in audio) and features that are shared across modalities (e.g., the semantic meaning of a word).

Methods

The authors propose M4SER, which stands for:

  1. Multimodal: It uses both audio (processed via HuBERT) and text (processed via BERT).
  2. Multirepresentation: It employs a Cross-modal Encoder (CME) to extract Modality-Specific Representations (MSR) and a Hybrid-modal Attention (HMA) generator to extract Modality-Invariant Representations (MIR).
  3. Multitask: The model is trained on three tasks simultaneously: the primary Emotion Recognition (ER) task, plus two auxiliary tasks—ASR Error Detection (AED) and ASR Error Correction (AEC). This forces the model to understand and "fix" the text it's reading.
  4. Multistrategy:
    • Adversarial Learning: A modality discriminator tries to guess if a feature came from audio or text. The MIR generator is trained to "deceive" this discriminator, ensuring the shared features are truly modality-agnostic.
  5. Label-based Contrastive Learning (LCL): This strategy forces the model to cluster representations of the same emotion more tightly in the feature space while pushing different emotions further apart.

Datasets

The model was evaluated on two gold-standard benchmarks:

  • IEMOCAP: Roughly 12 hours of scripted and improvised speech categorized into four emotions (Neutral, Angry, Happy/Excited, Sad).
  • MELD: Over 13,000 samples from the TV show Friends, featuring seven emotion labels (Neutral, Happy, Fear, Sad, Disgust, Angry, Surprise). This dataset is particularly challenging due to its "in-the-wild" nature and more complex emotional categories.

Key Insights and Ideas

  • The "ASR Advantage": Surprisingly, the authors found that their model sometimes performs better using ASR-generated text than ground-truth (GT) human transcripts. They hypothesize that ASR errors—like simplifications or specific misrecognitions—might actually align better with the emotional "vibe" or acoustic patterns of the speaker than a perfectly clean transcript.
  • Fine-Grained Adversarial Training: Unlike previous works that apply adversarial filters at a global level, M4SER uses a frame-level discriminator. This allows the model to align audio and text at a much more granular temporal level, which is crucial for capturing fleeting emotional cues.
  • Synergy of Tasks: The multitask approach creates a "virtuous cycle." By learning to detect and correct ASR errors, the model develops a more robust internal representation of the text, which directly feeds into more accurate emotion classification.

Key Results

M4SER achieved new state-of-the-art (SOTA) performance on both datasets:

  • IEMOCAP: Reached 79.2% Weighted Accuracy (WA) and 80.1% Unweighted Accuracy (UA), outperforming previous SOTA models like MAF-DCT and MGAT.
  • MELD: Achieved 66.5% Accuracy and a 66.0% Weighted F1-score, showing significant robustness in complex, multi-speaker scenarios.
  • Ablation Studies: Removing any of the "M4" components (the auxiliary tasks, the adversarial strategy, or the contrastive learning) resulted in a measurable drop in performance, validating the holistic design of the framework.

Limitations and Future Work

While M4SER is powerful, it is computationally heavy. The inclusion of GAN-based adversarial training and multiple large pre-trained backbones (HuBERT and BERT) increases training time significantly (up to 21.5 hours in their setup).

The authors suggest two main paths for future work:

  1. Adding the Visual Modality: Expanding the framework to include facial expressions and gestures to create a truly tri-modal system.
  2. LLM Integration: Investigating whether even more powerful Large Language Models (LLMs) could replace the current ASR error correction module to provide even deeper semantic coherence.

Summary updated

Abstract

Multimodal speech emotion recognition (SER) has emerged as pivotal for improving human-machine interaction. Researchers are increasingly leveraging both speech and textual information obtained through automatic speech recognition (ASR) to comprehensively recognize emotional states from speakers. Although this approach reduces reliance on human-annotated text data, ASR errors possibly degrade emotion recognition performance. To address this challenge, in our previous work, we introduced two auxiliary tasks, namely, ASR error detection and ASR error correction, and we proposed a novel multimodal fusion (MF) method for learning modality-specific and modality-invariant representations across different modalities. Building on this foundation, in this paper, we introduce two additional training strategies. First, we propose an adversarial network to enhance the diversity of modality-specific representations. Second, we introduce a label-based contrastive learning strategy to better capture emotional features. We refer to our proposed method as M4SER and validate its superiority over state-of-the-art methods through extensive experiments using IEMOCAP and MELD datasets.
BibTeX
@misc{he2025m,
  author = {Jiajun He and Xiaohan Shi and Cheng-Hung Hu and Jinyi Mi and Xingfeng Li and Tomoki Toda},
  title = {{M4SER: Multimodal, Multirepresentation, Multitask, and Multistrategy Learning for Speech Emotion Recognition}},
  year = {2025},
  eprint = {2509.18706},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.18706},
}
Figure