Abstract
Multimodal Emotion Recognition (MER) often suffers from English-centric bias and cultural nuances in non-Western contexts. We present IndoMER, the first comprehensive Indonesian MER benchmark, and OmniMER, a framework that leverages auxiliary-task adaptation to improve grounding in low-resource settings. By forcing the model to identify modality-specific emotional cues before fusion, OmniMER achieves significant performance gains, particularly in minority emotion classes, and demonstrates robust cross-lingual generalization.
Introduction
Multimodal Emotion Recognition (MER) aims to understand human affect by integrating text, audio, and visual signals. While the field has advanced rapidly, it suffers from a massive "English-centric" bias. Languages like Indonesian, spoken by over 200 million people, lack high-quality annotated datasets.
Beyond simple data scarcity, Indonesian presents unique cultural challenges. Concepts like Rukun (social harmony) and Sopan Santun (politeness) often lead speakers to use neutral or indirect language while expressing stronger emotions through tone or facial expressions. Furthermore, natural discourse is plagued by long-tailed distributions where "neutral" states dominate, making it difficult for models to learn rare emotions like "fear" or "disgust."
The IndoMER Dataset
To address the data gap, the authors introduced IndoMER, a benchmark featuring 1,944 video segments from 203 speakers across 13 themes, including politics, cooking, and blogging. It includes temporally aligned text, audio, and visual annotations for seven emotion categories.
As shown in the distribution above, the dataset reflects the real-world challenge of imbalanced classes, where neutral samples significantly outnumber specific emotional categories like surprise or fear.
The OmniMER Framework
The authors propose OmniMER, built upon the Qwen2.5-Omni backbone. Instead of simple fusion, they implement a strategy of Auxiliary-task Adaptation. Before final fusion, the model must complete three modality-specific auxiliary tasks: extracting emotional keywords from text, analyzing facial expressions from video, and identifying prosodic cues (pitch, loudness) from audio.
To train these without manual labels, a consistency-check mechanism was used: the LLM generated its own explanations, which were kept if the predicted sentiment matched the ground truth. The framework was optimized using Parameter-Efficient Fine-Tuning (LoRA) with two scheduling strategies: Multi-stage optimization (grounding first, then classification) and Hybrid mixed-instruction (simultaneous learning).
Key Results and Insights
OmniMER demonstrated massive gains over the base Qwen2.5-Omni model on the IndoMER dataset:
- Sentiment Classification: Achieved a 0.582 Macro-F1, outperforming the baseline by 7.6 points.
- Emotion Recognition: Achieved a 0.454 Macro-F1, a staggering 22.1-point improvement.
- Minority Classes: The multi-stage approach was particularly effective for rare emotions, achieving a perfect 1.000 F1 score on the "Surprise" class, which the base model failed to recognize entirely.
The research highlights that grounding prevents overfitting by forcing the model to identify why a modality reflects an emotion, reducing reliance on spurious correlations common in small datasets. Furthermore, the auxiliary tasks allow the model to "see through" neutral text by weighing audio and visual cues more effectively in culturally nuanced contexts.
Conclusion
OmniMER proves to be a robust method for LLM adaptation in MER, showing strong generalization even on Chinese datasets (CH-SIMS). While the IndoMER dataset is a significant first step, future work will explore scaling these auxiliary tasks across different architectures and extending the framework to other underrepresented languages.
Summary updated