TL;DR
This paper introduces HumanSense, a hierarchical benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on human-centered perception, implicit understanding, and empathetic interaction. The authors reveal a massive performance gap between current state-of-the-art models and humans, and demonstrate that an omni-modal, multi-stage reinforcement learning approach—along with structured prompt engineering—can significantly unlock a model's human-centric reasoning capabilities.
What Problem Does This Paper Address?
While MLLMs have advanced rapidly in general video and text understanding, they still struggle to act as true human companions capable of social support and emotional connection. Existing benchmarks typically evaluate models on objective, low-level tasks (e.g., identifying objects or counting actions) rather than the subtle nuances of human interaction, such as decoding implicit intentions, recognizing emotional rapport, or planning empathetic responses.
To bridge this gap, the authors address two core challenges:
- The lack of a fine-grained evaluation framework for human-centered multimodal scenarios.
- The deficiency of current MLLMs in high-level, long-context reasoning, which prevents them from generating context-aware, rational feedback.
Methods
The authors tackle these challenges with a two-pronged approach:
1. Multi-Stage Omni-Modal Reinforcement Learning
To build a reasoning-focused omni-modal model (HumanSense-Omni-Reasoning), the authors apply Group Relative Policy Optimization (GRPO). Because training a model on video, audio, and text simultaneously can be unstable, they introduce a modality-progressive training pipeline:
- Stage 1 (Vision-Centric): Trains the model using pure video frames and textual QA pairs to anchor visual reasoning.
- Stage 2 (Audio-Centric): Introduces audio-only QA pairs to develop auditory comprehension.
- Stage 3 (Omni-Modal): Combines complete video-audio QA pairs to reinforce joint audio-visual reasoning.
The model is rewarded based on both structural formatting (using <think> and <answer> tags) and absolute accuracy.
2. Training-Free Prompt Enhancement
Observing that successful RL-trained models follow a highly consistent thought pattern, the authors designed a prompt template to stimulate this behavior in non-reasoning models without additional training. The prompt forces the model to explicitly identify three elements before drawing a conclusion:
- Key Characteristics: Notable features, actions, or behaviors.
- Emotion: Expressed or inferred emotional states.
- Context: Relevant dialogue or spoken words.
Datasets
The HumanSense benchmark consists of 3,291 video-based questions and 591 audio-based questions (3,882 total) structured into a four-tier pyramid of difficulty:
The benchmark repurposes and annotates several open-source datasets:
- L1: Surface Perception: CelebV-HQ (Appearance, Emotion Recognition), AVA (Action Recognition), Jester (Gesture Recognition).
- L2: Complex Perception: ActivityNet (Complex Motion), UCF-Crime100 (Abnormal Classification), TAO (Human Counting with ReID), RealTalk (Talking Person Recognition).
- L3: Contextual Understanding: The Skin Deep (Familiarity, Rapport, and Relation Recognition), SEUMLD (Lie Detection), Telecom Fraud Texts (Fraud Recognition).
- L4: Feedback Strategy: RealTalk (Emo Strategy for facial feedback), Emotional First Aid dataset (Psychological Chat for multi-turn dialogue responses).
For human baseline comparison, the authors curated HumanSense (tiny), a subset of 450 randomly sampled questions.
Key Insights and Ideas
- The Perception-Reasoning Bottleneck: The primary limitation of current MLLMs isn't low-level perception (seeing and hearing), but rather high-level, long-context reasoning. Models can often see an expression but fail to plan an appropriate response based on the broader social context.
- Audio is Essential for Empathy: Visual data alone is insufficient for high-level interaction tasks. Modality ablation studies showed that adding raw audio significantly boosts performance on tasks like rapport and relationship recognition, outperforming text-only transcriptions (ASR).
- Consistent Thought Patterns Can Be Prompted: Human-centric reasoning follows a predictable pipeline: Perceive Characteristics + Emotion + Context $\rightarrow$ Think $\rightarrow$ Respond. Mimicking this structure via prompt engineering yields immediate performance gains for standard MLLMs.
Key Results
The authors evaluated leading proprietary and open-source models (including GPT-4o, InternVL3-8B, and Qwen2.5-Omni-7B) and uncovered several striking findings:
- The Human-AI Gap: Human evaluators achieved an average accuracy of 87.5%, outperforming the best-tested zero-shot model by a massive 29.7% margin.
- RL Training Breakthrough: The multi-stage RL framework drastically improved Qwen2.5-Omni-7B's performance on complex interactive tasks. For instance, its accuracy on the Psychological Chat (PC) task jumped from a baseline of 39.9% to 61.9% after stage-3 training.
- Prompting Success: The training-free Prompt Enhancement (PE) strategy boosted the baseline model's Psychological Chat performance from 39.9% to 43.6% and improved familiarity recognition tasks without changing model weights.
Limitations and Future Work
While the paper demonstrates clear pathways to improving empathetic AI, some limitations remain:
- API Constraints: Proprietary models like GPT-4o could not be fully evaluated on audio-native tasks due to API limitations that lacked direct audio input support at the time of testing.
- Static Evaluation of Dynamic Interaction: Although the benchmark evaluates response planning, it relies on single-choice questions derived from real-world records rather than live, open-ended, multi-turn human-in-the-loop interactions.
The authors note that future work should focus on expanding these reasoning capabilities toward fully generative, real-time anthropomorphic expressions (integrating speech synthesis and talking heads) to achieve truly human-like AI companions.
Summary updated