MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models

Fan Zhang, Zebang Cheng, Chong Deng, Haoxuan Li, Zheng Lian, Qian Chen, Huadai Liu, Wen Wang, Yifan Zhang, Renrui Zhang, Ziyu Guo, Zhihong Zhu, Hao Wu, Haixin Wang, Yefeng Zheng, Xiaojiang Peng, Xian Wu, Kun Wang, Xiangang Li, Jieping Ye, Pheng-Ann Heng

Abstract

Recent advances in multimodal large language models (MLLMs) have catalyzed transformative progress in affective computing, enabling models to exhibit emergent emotional intelligence. Despite substantial methodological progress, current emotional benchmarks remain limited, as it is still unknown: (a) the generalization abilities of MLLMs across distinct scenarios, and (b) their reasoning capabilities to identify the triggering factors behind emotional states. To bridge these gaps, we present MME-Emotion, a systematic benchmark that assesses both emotional understanding and reasoning capabilities of MLLMs, enjoying scalable capacity, diverse settings, and unified protocols. As the largest emotional intelligence benchmark for MLLMs, MME-Emotion contains over 6,000 curated video clips with task-specific questioning-answering (QA) pairs, spanning broad scenarios to formulate eight emotional tasks. It further incorporates a holistic evaluation suite with hybrid metrics for emotion recognition and reasoning, analyzed through a multi-agent system framework. Through a rigorous evaluation of 20 advanced MLLMs, we uncover both their strengths and limitations, yielding several key insights: (1) Current MLLMs exhibit unsatisfactory emotional intelligence, with the best-performing model achieving only 39.3% recognition score and 56.0% Chain-of-Thought (CoT) score on our benchmark. (2) Generalist models (e.g., Gemini-2.5-Pro) derive emotional intelligence from generalized multimodal understanding capabilities, while specialist models (e.g., R1-Omni) can achieve comparable performance through domain-specific post-training adaptation.

Introduction

While Multimodal Large Language Models (MLLMs) have shown incredible progress in describing images or solving math problems, their "affective computing" capabilities—understanding how humans feel and why—remain under-explored. Existing emotional benchmarks often suffer from narrow scopes, focusing on dramatic movie clips while failing to test generalization to controlled lab settings or noisy environments. Furthermore, most benchmarks prioritize simple recognition ("What is the emotion?") over reasoning ("Why is this the emotion?").

To address these limitations, we introduce MME-Emotion, a massive new benchmark designed to evaluate the Emotional Intelligence (EI) of MLLMs across 6,500 video-based tasks. Our study reveals a sobering reality: even the most advanced models struggle with human emotions, with the top performer barely cracking a 39% recognition score.

The MME-Emotion Benchmark

MME-Emotion is the largest benchmark of its kind, characterized by its scale and diversity. It spans 27 distinct scenarios, from vlogs and movies to lab-controlled interviews, ensuring models are tested against a wide variety of emotional expressions.

Figure 3: Overview of the MME-Emotion dataset, showing the diversity of scenarios and the distribution of samples and video lengths across eight core tasks.

The benchmark covers eight specific tasks, including Emotion Recognition (in Lab, Wild, and Noisy settings), Fine-grained and Multi-label Recognition, Sentiment Analysis, and Intent Recognition (e.g., determining if a speaker is "consoling" or "questioning").

Evaluation Framework

We move beyond simple accuracy by introducing a systematic evaluation framework that utilizes a multi-agent system to avoid the bias of human grading.

  • Multi-Agent System: A pipeline of specialized agents (Step, Audio, and Judge) breaks down model responses into logical steps and compares them against ground-truth multimodal clues.
  • Unified Metrics: We propose three core metrics: the Recognition Score (Rec-S) for accuracy, the Reasoning Score (Rea-S) for the validity of intermediate steps, and the Chain-of-Thought Score (CoT-S) for a holistic view of emotional "thinking."

Results and Analysis

Our evaluation of 20 advanced MLLMs highlights a significant "low ceiling" in current emotional intelligence. The average recognition score across all models was a lowly 29.4%.

Figure 1: Radar charts illustrating the performance of top models across Recognition, Reasoning, and CoT scores for various emotional tasks.

Key Insights

  • Reasoning-Recognition Synergy: There is a strong positive correlation between a model's ability to reason and its final accuracy. Models that "think step-by-step" are more likely to arrive at the correct emotional state.
  • Generalists vs. Specialists: While generalists like Gemini-2.5-Pro lead the pack, "specialist" models (like Audio-Reasoner or R1-Omni) that undergo domain-specific post-training can achieve comparable results with smaller parameter counts.
  • The Multimodal Fusion Gap: Many models fail because they cannot effectively integrate audio and video. For instance, a model might identify a smiling face visually but miss a sarcastic, angry tone in the audio.
  • The "Lab" Difficulty: Models perform significantly worse on controlled laboratory data compared to "in-the-wild" movie clips. This suggests current models are over-reliant on the dramatic, exaggerated emotions found in cinematic media and struggle with subtle, real-world expressions.

Conclusion

MME-Emotion serves as a foundational benchmark for advancing the emotional intelligence of MLLMs. By highlighting the current limitations in multimodal fusion and subtle emotion recognition, we hope to guide the development of a new generation of models capable of high-fidelity emotional reasoning. Future work will expand the benchmark to include multilingual nuances and more granular difficulty grading.

Summary updated

Abstract

Recent advances in multimodal large language models (MLLMs) have catalyzed transformative progress in affective computing, enabling models to exhibit emergent emotional intelligence. Despite substantial methodological progress, current emotional benchmarks remain limited, as it is still unknown: (a) the generalization abilities of MLLMs across distinct scenarios, and (b) their reasoning capabilities to identify the triggering factors behind emotional states. To bridge these gaps, we present \textbf{MME-Emotion}, a systematic benchmark that assesses both emotional understanding and reasoning capabilities of MLLMs, enjoying \textit{scalable capacity}, \textit{diverse settings}, and \textit{unified protocols}. As the largest emotional intelligence benchmark for MLLMs, MME-Emotion contains over 6,000 curated video clips with task-specific questioning-answering (QA) pairs, spanning broad scenarios to formulate eight emotional tasks. It further incorporates a holistic evaluation suite with hybrid metrics for emotion recognition and reasoning, analyzed through a multi-agent system framework. Through a rigorous evaluation of 20 advanced MLLMs, we uncover both their strengths and limitations, yielding several key insights: \ding{182} Current MLLMs exhibit unsatisfactory emotional intelligence, with the best-performing model achieving only $39.3\%$ recognition score and $56.0\%$ Chain-of-Thought (CoT) score on our benchmark. \ding{183} Generalist models (\emph{e.g.}, Gemini-2.5-Pro) derive emotional intelligence from generalized multimodal understanding capabilities, while specialist models (\emph{e.g.}, R1-Omni) can achieve comparable performance through domain-specific post-training adaptation. By introducing MME-Emotion, we hope that it can serve as a foundation for advancing MLLMs' emotional intelligence in the future.
BibTeX
@inproceedings{zhang2026mme,
  author = {Fan Zhang and Zebang Cheng and Chong Deng and Haoxuan Li and Zheng Lian and Qian Chen and Huadai Liu and Wen Wang and Yifan Zhang and Renrui Zhang and Ziyu Guo and Zhihong Zhu and Hao Wu and Haixin Wang and Yefeng Zheng and Xiaojiang Peng and Xian Wu and Kun Wang and Xiangang Li and Jieping Ye and Pheng-Ann Heng},
  title = {{MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=oSX9aenbea},
}
Figure