MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models

Fan Zhang, Zebang Cheng, Chong Deng, Haoxuan Li, Zheng Lian, Qian Chen, Huadai Liu, Wen Wang, Yi-Fan Zhang, Renrui Zhang, Ziyu Guo, Zhihong Zhu, Hao Wu, Haixin Wang, Yefeng Zheng, Xiaojiang Peng, Xian Wu, Kun Wang, Xiangang Li, Jieping Ye, Pheng-Ann Heng

TL;DR

MME-Emotion is a new large-scale benchmark featuring over 6,000 video clips designed to evaluate both the emotional recognition and reasoning capabilities of Multimodal Large Language Models (MLLMs). The evaluation reveals that even top-tier models like Gemini-2.5-Pro struggle significantly with emotional intelligence, highlighting a massive gap between general multimodal understanding and true affective reasoning.

What Problem Does This Paper Address?

While Multimodal Large Language Models (MLLMs) excel at general tasks like visual QA and image captioning, their emotional intelligence—the ability to perceive, integrate, and reason about human emotions—remains a major blind spot.

Existing emotional evaluation benchmarks suffer from two primary limitations:

  1. Inadequate scenario coverage: They fail to test how well models generalize across diverse, real-world situations.
  2. Lack of reasoning evaluation: They focus almost entirely on recognition (e.g., mapping a face to a "sad" label) rather than reasoning (e.g., explaining why the person is sad based on visual and acoustic cues).

To bridge these gaps, the authors introduce MME-Emotion, a comprehensive benchmark designed to test both the understanding and the underlying reasoning of MLLMs across a wide variety of emotional contexts.

Methods

Evaluating open-ended reasoning without relying on expensive, subjective human annotation is notoriously difficult. To solve this, the authors developed an annotation-free, multi-agent system framework that uses an MLLM-as-a-judge strategy:

  1. Step Extraction: When an MLLM answers an emotional question, a unimodal Step Agent (powered by GPT-4.1) breaks down the model's response into distinct reasoning steps, isolating the final prediction.
  2. Modality Splitting: To ensure the judge has full context, an Audio Agent (Qwen2-Audio) extracts explicit acoustic clues from the video clip, while the visual frames are extracted directly.
  3. The Judge: A Judge Agent (GPT-4o) ingests the visual frames, audio clues, ground-truth labels, and the model's extracted reasoning steps.

The judge scores the model using three unified metrics:

  • Recognition Score (Rec-S): Accuracy of the final emotional label prediction.
  • Reasoning Score (Rea-S): Average correctness across all intermediate reasoning steps.
  • Chain-of-Thought Score (CoT-S): A balanced, weighted combination of both recognition and reasoning.
Illustration of the multi-agent evaluation strategy

Datasets

The authors curated MME-Emotion, which stands as the largest emotional intelligence benchmark for MLLMs to date.

  • Scale: 6,500 video clips paired with task-specific question-answering (QA) pairs.
  • Diversity: Covers 27 distinct scenario types (e.g., movies, interviews, vlogs, talk shows) across 8 emotional tasks: laboratory emotion recognition, in-the-wild recognition, recognition under noise, fine-grained recognition, multi-label recognition, sentiment analysis, fine-grained sentiment analysis, and intent recognition.
  • Data Integrity: All samples were resampled exclusively from the test splits of various public datasets (such as IEMOCAP, MELD, and MAFW) to strictly prevent data leakage during model pre-training.
Figure 1: Overview of MME-Emotion Statistics. Left: Task Types. MME-Emotion encompasses eight emotional tasks across 27 distinct scenario types, enabling fine-grained analysis of diverse video contexts. Right: Data Distributions. MME-Emotion features balanced distributions of question volume and video duration, facilitating comprehensive evaluation of temporal understanding.

Key Insights and Ideas

  • Current MLLMs lack emotional depth: Even the best-performing models achieve low recognition scores, proving that emotional intelligence does not automatically emerge at a high level from standard multimodal training.
  • Two paths to emotional competence: Generalist models (like Gemini-2.5-Pro) derive their emotional capabilities from massive scale and generalized understanding. Conversely, smaller specialist models (like R1-Omni or Audio-Reasoner) can achieve comparable performance through targeted post-training adaptation techniques like Supervised Fine-Tuning (SFT) and Reinforcement Learning.
  • Reasoning drives accuracy: The authors observed a strong positive correlation between the number of reasoning steps a model takes and its final accuracy. Encouraging models to generate a Chain-of-Thought (CoT) directly improves their emotional recognition.
  • The omnimodal bottleneck: Surprisingly, "omnimodal" models that process video, audio, and text simultaneously often underperformed compared to models processing fewer modalities. This indicates that current architectures struggle to effectively fuse redundant or conflicting emotional cues across different senses.

Key Results

The authors conducted a rigorous evaluation of 20 advanced MLLMs under zero-shot settings:

  • Top Performer: Gemini-2.5-Pro achieved the highest overall performance, but still only reached a 39.3% Recognition Score and a 56.0% CoT Score.
  • Open-Source Standout: Audio-Reasoner (a 7B model processing only audio and text) achieved a remarkably competitive CoT Score of 54.8%, beating out several larger vision-language models.
  • Benchmark Average: Across all 20 models, the average performance was quite low: 29.4% for recognition and 49.5% for reasoning, underscoring the challenging nature of the benchmark.
Figure 2: Performance Comparison of Leading MLLMs on MME-Emotion. Our evaluation suite assesses MLLMs using three unified metrics (Rec-S, Rea-S, and CoT-S) across eight emotional tasks.

Limitations and Future Work

The authors note two primary limitations of the current benchmark:

  1. Lack of difficulty tiers: The benchmark does not categorize samples by complexity (e.g., separating obvious emotional expressions from subtle, micro-expressions). Future iterations could benefit from a "primary school to Olympiad" style difficulty rating.
  2. Linguistic constraints: Although the underlying video data spans multiple languages, the benchmark does not currently evaluate how linguistic and cultural contexts impact an MLLM's emotional reasoning.

Future work will focus on addressing these multi-modal fusion bottlenecks and developing more robust, emotion-specialized architectures.

Summary updated

Abstract

Recent advances in multimodal large language models (MLLMs) have catalyzed transformative progress in affective computing, enabling models to exhibit emergent emotional intelligence. Despite substantial methodological progress, current emotional benchmarks remain limited, as it is still unknown: (a) the generalization abilities of MLLMs across distinct scenarios, and (b) their reasoning capabilities to identify the triggering factors behind emotional states. To bridge these gaps, we present \textbf{MME-Emotion}, a systematic benchmark that assesses both emotional understanding and reasoning capabilities of MLLMs, enjoying \textit{scalable capacity}, \textit{diverse settings}, and \textit{unified protocols}. As the largest emotional intelligence benchmark for MLLMs, MME-Emotion contains over 6,000 curated video clips with task-specific questioning-answering (QA) pairs, spanning broad scenarios to formulate eight emotional tasks. It further incorporates a holistic evaluation suite with hybrid metrics for emotion recognition and reasoning, analyzed through a multi-agent system framework. Through a rigorous evaluation of 20 advanced MLLMs, we uncover both their strengths and limitations, yielding several key insights: \ding{182} Current MLLMs exhibit unsatisfactory emotional intelligence, with the best-performing model achieving only $39.3\%$ recognition score and $56.0\%$ Chain-of-Thought (CoT) score on our benchmark. \ding{183} Generalist models (\emph{e.g.}, Gemini-2.5-Pro) derive emotional intelligence from generalized multimodal understanding capabilities, while specialist models (\emph{e.g.}, R1-Omni) can achieve comparable performance through domain-specific post-training adaptation. By introducing MME-Emotion, we hope that it can serve as a foundation for advancing MLLMs' emotional intelligence in the future.
BibTeX
@inproceedings{zhang2026mme,
  author = {Fan Zhang and Zebang Cheng and Chong Deng and Haoxuan Li and Zheng Lian and Qian Chen and Huadai Liu and Wen Wang and Yi-Fan Zhang and Renrui Zhang and Ziyu Guo and Zhihong Zhu and Hao Wu and Haixin Wang and Yefeng Zheng and Xiaojiang Peng and Xian Wu and Kun Wang and Xiangang Li and Jieping Ye and Pheng-Ann Heng},
  title = {{MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=oSX9aenbea},
}
Figure