OmniMER: Indonesian Multimodal Emotion Recognition via Auxiliary-Enhanced LLM Adaptation

Xueming Yan, Boyan Xu, Yaochu Jin, Lixian Xiao, Wenlong Ye, Runyang Cai, Zeqi Zheng, Jingfa Liu, Aimin YangarXiv 2025 · DOI · arXiv

Abstract

Multimodal Emotion Recognition (MER) often suffers from English-centric bias and cultural nuances in non-Western contexts. We present IndoMER, the first comprehensive Indonesian MER benchmark, and OmniMER, a framework that leverages auxiliary-task adaptation to improve grounding in low-resource settings. By forcing the model to identify modality-specific emotional cues before fusion, OmniMER achieves significant performance gains, particularly in minority emotion classes, and demonstrates robust cross-lingual generalization.

Introduction

Multimodal Emotion Recognition (MER) aims to understand human affect by integrating text, audio, and visual signals. While the field has advanced rapidly, it suffers from a massive "English-centric" bias. Languages like Indonesian, spoken by over 200 million people, lack high-quality annotated datasets.

Beyond simple data scarcity, Indonesian presents unique cultural challenges. Concepts like Rukun (social harmony) and Sopan Santun (politeness) often lead speakers to use neutral or indirect language while expressing stronger emotions through tone or facial expressions. Furthermore, natural discourse is plagued by long-tailed distributions where "neutral" states dominate, making it difficult for models to learn rare emotions like "fear" or "disgust."

The IndoMER Dataset

To address the data gap, the authors introduced IndoMER, a benchmark featuring 1,944 video segments from 203 speakers across 13 themes, including politics, cooking, and blogging. It includes temporally aligned text, audio, and visual annotations for seven emotion categories.

Long-tail distribution of emotion categories in the IndoMER dataset.

As shown in the distribution above, the dataset reflects the real-world challenge of imbalanced classes, where neutral samples significantly outnumber specific emotional categories like surprise or fear.

The OmniMER Framework

The authors propose OmniMER, built upon the Qwen2.5-Omni backbone. Instead of simple fusion, they implement a strategy of Auxiliary-task Adaptation. Before final fusion, the model must complete three modality-specific auxiliary tasks: extracting emotional keywords from text, analyzing facial expressions from video, and identifying prosodic cues (pitch, loudness) from audio.

OmniMER architecture featuring auxiliary-task adaptation and optimization strategies.

To train these without manual labels, a consistency-check mechanism was used: the LLM generated its own explanations, which were kept if the predicted sentiment matched the ground truth. The framework was optimized using Parameter-Efficient Fine-Tuning (LoRA) with two scheduling strategies: Multi-stage optimization (grounding first, then classification) and Hybrid mixed-instruction (simultaneous learning).

Key Results and Insights

OmniMER demonstrated massive gains over the base Qwen2.5-Omni model on the IndoMER dataset:

  • Sentiment Classification: Achieved a 0.582 Macro-F1, outperforming the baseline by 7.6 points.
  • Emotion Recognition: Achieved a 0.454 Macro-F1, a staggering 22.1-point improvement.
  • Minority Classes: The multi-stage approach was particularly effective for rare emotions, achieving a perfect 1.000 F1 score on the "Surprise" class, which the base model failed to recognize entirely.

The research highlights that grounding prevents overfitting by forcing the model to identify why a modality reflects an emotion, reducing reliance on spurious correlations common in small datasets. Furthermore, the auxiliary tasks allow the model to "see through" neutral text by weighing audio and visual cues more effectively in culturally nuanced contexts.

Conclusion

OmniMER proves to be a robust method for LLM adaptation in MER, showing strong generalization even on Chinese datasets (CH-SIMS). While the IndoMER dataset is a significant first step, future work will explore scaling these auxiliary tasks across different architectures and extending the framework to other underrepresented languages.

Summary updated

Abstract

Indonesian, spoken by over 200 million people, remains underserved in multimodal emotion recognition research despite its dominant presence on Southeast Asian social media platforms. We introduce IndoMER, the first multimodal emotion recognition benchmark for Indonesian, comprising 1,944 video segments from 203 speakers with temporally aligned text, audio, and visual annotations across seven emotion categories. The dataset exhibits realistic challenges including cross-modal inconsistency and long-tailed class distributions shaped by Indonesian cultural communication norms. To address these challenges, we propose OmniMER, a multimodal adaptation framework built upon Qwen2.5-Omni that enhances emotion recognition through three auxiliary modality-specific perception tasks: emotion keyword extraction for text, facial expression analysis for video, and prosody analysis for audio. These auxiliary tasks help the model identify emotion-relevant cues in each modality before fusion, reducing reliance on spurious correlations in low-resource settings. Experiments on IndoMER show that OmniMER achieves 0.582 Macro-F1 on sentiment classification and 0.454 on emotion recognition, outperforming the base model by 7.6 and 22.1 absolute points respectively. Cross-lingual evaluation on the Chinese CH-SIMS dataset further demonstrates the generalizability of the proposed framework. The dataset and code are publicly available. https://github.com/yanxm01/INDOMER
BibTeX
@misc{yan2025omnimer,
  author = {Xueming Yan and Boyan Xu and Yaochu Jin and Lixian Xiao and Wenlong Ye and Runyang Cai and Zeqi Zheng and Jingfa Liu and Aimin Yang},
  title = {{OmniMER: Indonesian Multimodal Emotion Recognition via Auxiliary-Enhanced LLM Adaptation}},
  year = {2025},
  eprint = {2512.19379},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.19379},
}
Figure