TL;DR
MME-Emotion is a new large-scale benchmark featuring over 6,000 video clips designed to evaluate both the emotional recognition and reasoning capabilities of Multimodal Large Language Models (MLLMs). The evaluation reveals that even top-tier models like Gemini-2.5-Pro struggle significantly with emotional intelligence, highlighting a massive gap between general multimodal understanding and true affective reasoning.
What Problem Does This Paper Address?
While Multimodal Large Language Models (MLLMs) excel at general tasks like visual QA and image captioning, their emotional intelligence—the ability to perceive, integrate, and reason about human emotions—remains a major blind spot.
Existing emotional evaluation benchmarks suffer from two primary limitations:
- Inadequate scenario coverage: They fail to test how well models generalize across diverse, real-world situations.
- Lack of reasoning evaluation: They focus almost entirely on recognition (e.g., mapping a face to a "sad" label) rather than reasoning (e.g., explaining why the person is sad based on visual and acoustic cues).
To bridge these gaps, the authors introduce MME-Emotion, a comprehensive benchmark designed to test both the understanding and the underlying reasoning of MLLMs across a wide variety of emotional contexts.
Methods
Evaluating open-ended reasoning without relying on expensive, subjective human annotation is notoriously difficult. To solve this, the authors developed an annotation-free, multi-agent system framework that uses an MLLM-as-a-judge strategy:
- Step Extraction: When an MLLM answers an emotional question, a unimodal Step Agent (powered by GPT-4.1) breaks down the model's response into distinct reasoning steps, isolating the final prediction.
- Modality Splitting: To ensure the judge has full context, an Audio Agent (Qwen2-Audio) extracts explicit acoustic clues from the video clip, while the visual frames are extracted directly.
- The Judge: A Judge Agent (GPT-4o) ingests the visual frames, audio clues, ground-truth labels, and the model's extracted reasoning steps.
The judge scores the model using three unified metrics:
- Recognition Score (Rec-S): Accuracy of the final emotional label prediction.
- Reasoning Score (Rea-S): Average correctness across all intermediate reasoning steps.
- Chain-of-Thought Score (CoT-S): A balanced, weighted combination of both recognition and reasoning.
Datasets
The authors curated MME-Emotion, which stands as the largest emotional intelligence benchmark for MLLMs to date.
- Scale: 6,500 video clips paired with task-specific question-answering (QA) pairs.
- Diversity: Covers 27 distinct scenario types (e.g., movies, interviews, vlogs, talk shows) across 8 emotional tasks: laboratory emotion recognition, in-the-wild recognition, recognition under noise, fine-grained recognition, multi-label recognition, sentiment analysis, fine-grained sentiment analysis, and intent recognition.
- Data Integrity: All samples were resampled exclusively from the test splits of various public datasets (such as IEMOCAP, MELD, and MAFW) to strictly prevent data leakage during model pre-training.
Key Insights and Ideas
- Current MLLMs lack emotional depth: Even the best-performing models achieve low recognition scores, proving that emotional intelligence does not automatically emerge at a high level from standard multimodal training.
- Two paths to emotional competence: Generalist models (like Gemini-2.5-Pro) derive their emotional capabilities from massive scale and generalized understanding. Conversely, smaller specialist models (like R1-Omni or Audio-Reasoner) can achieve comparable performance through targeted post-training adaptation techniques like Supervised Fine-Tuning (SFT) and Reinforcement Learning.
- Reasoning drives accuracy: The authors observed a strong positive correlation between the number of reasoning steps a model takes and its final accuracy. Encouraging models to generate a Chain-of-Thought (CoT) directly improves their emotional recognition.
- The omnimodal bottleneck: Surprisingly, "omnimodal" models that process video, audio, and text simultaneously often underperformed compared to models processing fewer modalities. This indicates that current architectures struggle to effectively fuse redundant or conflicting emotional cues across different senses.
Key Results
The authors conducted a rigorous evaluation of 20 advanced MLLMs under zero-shot settings:
- Top Performer: Gemini-2.5-Pro achieved the highest overall performance, but still only reached a 39.3% Recognition Score and a 56.0% CoT Score.
- Open-Source Standout: Audio-Reasoner (a 7B model processing only audio and text) achieved a remarkably competitive CoT Score of 54.8%, beating out several larger vision-language models.
- Benchmark Average: Across all 20 models, the average performance was quite low: 29.4% for recognition and 49.5% for reasoning, underscoring the challenging nature of the benchmark.
Limitations and Future Work
The authors note two primary limitations of the current benchmark:
- Lack of difficulty tiers: The benchmark does not categorize samples by complexity (e.g., separating obvious emotional expressions from subtle, micro-expressions). Future iterations could benefit from a "primary school to Olympiad" style difficulty rating.
- Linguistic constraints: Although the underlying video data spans multiple languages, the benchmark does not currently evaluate how linguistic and cultural contexts impact an MLLM's emotional reasoning.
Future work will focus on addressing these multi-modal fusion bottlenecks and developing more robust, emotion-specialized architectures.
Summary updated