Abstract
Recent advances in multimodal large language models (MLLMs) have catalyzed transformative progress in affective computing, enabling models to exhibit emergent emotional intelligence. Despite substantial methodological progress, current emotional benchmarks remain limited, as it is still unknown: (a) the generalization abilities of MLLMs across distinct scenarios, and (b) their reasoning capabilities to identify the triggering factors behind emotional states. To bridge these gaps, we present MME-Emotion, a systematic benchmark that assesses both emotional understanding and reasoning capabilities of MLLMs, enjoying scalable capacity, diverse settings, and unified protocols. As the largest emotional intelligence benchmark for MLLMs, MME-Emotion contains over 6,000 curated video clips with task-specific questioning-answering (QA) pairs, spanning broad scenarios to formulate eight emotional tasks. It further incorporates a holistic evaluation suite with hybrid metrics for emotion recognition and reasoning, analyzed through a multi-agent system framework. Through a rigorous evaluation of 20 advanced MLLMs, we uncover both their strengths and limitations, yielding several key insights: (1) Current MLLMs exhibit unsatisfactory emotional intelligence, with the best-performing model achieving only 39.3% recognition score and 56.0% Chain-of-Thought (CoT) score on our benchmark. (2) Generalist models (e.g., Gemini-2.5-Pro) derive emotional intelligence from generalized multimodal understanding capabilities, while specialist models (e.g., R1-Omni) can achieve comparable performance through domain-specific post-training adaptation.
Introduction
While Multimodal Large Language Models (MLLMs) have shown incredible progress in describing images or solving math problems, their "affective computing" capabilities—understanding how humans feel and why—remain under-explored. Existing emotional benchmarks often suffer from narrow scopes, focusing on dramatic movie clips while failing to test generalization to controlled lab settings or noisy environments. Furthermore, most benchmarks prioritize simple recognition ("What is the emotion?") over reasoning ("Why is this the emotion?").
To address these limitations, we introduce MME-Emotion, a massive new benchmark designed to evaluate the Emotional Intelligence (EI) of MLLMs across 6,500 video-based tasks. Our study reveals a sobering reality: even the most advanced models struggle with human emotions, with the top performer barely cracking a 39% recognition score.
The MME-Emotion Benchmark
MME-Emotion is the largest benchmark of its kind, characterized by its scale and diversity. It spans 27 distinct scenarios, from vlogs and movies to lab-controlled interviews, ensuring models are tested against a wide variety of emotional expressions.
The benchmark covers eight specific tasks, including Emotion Recognition (in Lab, Wild, and Noisy settings), Fine-grained and Multi-label Recognition, Sentiment Analysis, and Intent Recognition (e.g., determining if a speaker is "consoling" or "questioning").
Evaluation Framework
We move beyond simple accuracy by introducing a systematic evaluation framework that utilizes a multi-agent system to avoid the bias of human grading.
- Multi-Agent System: A pipeline of specialized agents (Step, Audio, and Judge) breaks down model responses into logical steps and compares them against ground-truth multimodal clues.
- Unified Metrics: We propose three core metrics: the Recognition Score (Rec-S) for accuracy, the Reasoning Score (Rea-S) for the validity of intermediate steps, and the Chain-of-Thought Score (CoT-S) for a holistic view of emotional "thinking."
Results and Analysis
Our evaluation of 20 advanced MLLMs highlights a significant "low ceiling" in current emotional intelligence. The average recognition score across all models was a lowly 29.4%.
Key Insights
- Reasoning-Recognition Synergy: There is a strong positive correlation between a model's ability to reason and its final accuracy. Models that "think step-by-step" are more likely to arrive at the correct emotional state.
- Generalists vs. Specialists: While generalists like Gemini-2.5-Pro lead the pack, "specialist" models (like Audio-Reasoner or R1-Omni) that undergo domain-specific post-training can achieve comparable results with smaller parameter counts.
- The Multimodal Fusion Gap: Many models fail because they cannot effectively integrate audio and video. For instance, a model might identify a smiling face visually but miss a sarcastic, angry tone in the audio.
- The "Lab" Difficulty: Models perform significantly worse on controlled laboratory data compared to "in-the-wild" movie clips. This suggests current models are over-reliant on the dramatic, exaggerated emotions found in cinematic media and struggle with subtle, real-world expressions.
Conclusion
MME-Emotion serves as a foundational benchmark for advancing the emotional intelligence of MLLMs. By highlighting the current limitations in multimodal fusion and subtle emotion recognition, we hope to guide the development of a new generation of models capable of high-fidelity emotional reasoning. Future work will expand the benchmark to include multilingual nuances and more granular difficulty grading.
Summary updated