HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs

Zheng Qin, Ruobing Zheng, Yabing Wang, Tianqi Li, Yi Yuan, Jingdong Chen, Le Wang

TL;DR

This paper introduces HumanSense, a hierarchical benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on human-centered perception, implicit understanding, and empathetic interaction. The authors reveal a massive performance gap between current state-of-the-art models and humans, and demonstrate that an omni-modal, multi-stage reinforcement learning approach—along with structured prompt engineering—can significantly unlock a model's human-centric reasoning capabilities.

What Problem Does This Paper Address?

While MLLMs have advanced rapidly in general video and text understanding, they still struggle to act as true human companions capable of social support and emotional connection. Existing benchmarks typically evaluate models on objective, low-level tasks (e.g., identifying objects or counting actions) rather than the subtle nuances of human interaction, such as decoding implicit intentions, recognizing emotional rapport, or planning empathetic responses.

To bridge this gap, the authors address two core challenges:

  1. The lack of a fine-grained evaluation framework for human-centered multimodal scenarios.
  2. The deficiency of current MLLMs in high-level, long-context reasoning, which prevents them from generating context-aware, rational feedback.

Methods

The authors tackle these challenges with a two-pronged approach:

1. Multi-Stage Omni-Modal Reinforcement Learning

To build a reasoning-focused omni-modal model (HumanSense-Omni-Reasoning), the authors apply Group Relative Policy Optimization (GRPO). Because training a model on video, audio, and text simultaneously can be unstable, they introduce a modality-progressive training pipeline:

  • Stage 1 (Vision-Centric): Trains the model using pure video frames and textual QA pairs to anchor visual reasoning.
  • Stage 2 (Audio-Centric): Introduces audio-only QA pairs to develop auditory comprehension.
  • Stage 3 (Omni-Modal): Combines complete video-audio QA pairs to reinforce joint audio-visual reasoning.

The model is rewarded based on both structural formatting (using <think> and <answer> tags) and absolute accuracy.

2. Training-Free Prompt Enhancement

Observing that successful RL-trained models follow a highly consistent thought pattern, the authors designed a prompt template to stimulate this behavior in non-reasoning models without additional training. The prompt forces the model to explicitly identify three elements before drawing a conclusion:

  • Key Characteristics: Notable features, actions, or behaviors.
  • Emotion: Expressed or inferred emotional states.
  • Context: Relevant dialogue or spoken words.

Datasets

The HumanSense benchmark consists of 3,291 video-based questions and 591 audio-based questions (3,882 total) structured into a four-tier pyramid of difficulty:

Figure 1: HumanSense benchmark is structured hierarchically to evaluate the end-to-end, human-centric process of perception, understanding, and feedback through a series of demonstrated tasks and questions.

The benchmark repurposes and annotates several open-source datasets:

  • L1: Surface Perception: CelebV-HQ (Appearance, Emotion Recognition), AVA (Action Recognition), Jester (Gesture Recognition).
  • L2: Complex Perception: ActivityNet (Complex Motion), UCF-Crime100 (Abnormal Classification), TAO (Human Counting with ReID), RealTalk (Talking Person Recognition).
  • L3: Contextual Understanding: The Skin Deep (Familiarity, Rapport, and Relation Recognition), SEUMLD (Lie Detection), Telecom Fraud Texts (Fraud Recognition).
  • L4: Feedback Strategy: RealTalk (Emo Strategy for facial feedback), Emotional First Aid dataset (Psychological Chat for multi-turn dialogue responses).

For human baseline comparison, the authors curated HumanSense (tiny), a subset of 450 randomly sampled questions.

Key Insights and Ideas

  • The Perception-Reasoning Bottleneck: The primary limitation of current MLLMs isn't low-level perception (seeing and hearing), but rather high-level, long-context reasoning. Models can often see an expression but fail to plan an appropriate response based on the broader social context.
  • Audio is Essential for Empathy: Visual data alone is insufficient for high-level interaction tasks. Modality ablation studies showed that adding raw audio significantly boosts performance on tasks like rapport and relationship recognition, outperforming text-only transcriptions (ASR).
  • Consistent Thought Patterns Can Be Prompted: Human-centric reasoning follows a predictable pipeline: Perceive Characteristics + Emotion + Context $\rightarrow$ Think $\rightarrow$ Respond. Mimicking this structure via prompt engineering yields immediate performance gains for standard MLLMs.

Key Results

The authors evaluated leading proprietary and open-source models (including GPT-4o, InternVL3-8B, and Qwen2.5-Omni-7B) and uncovered several striking findings:

  • The Human-AI Gap: Human evaluators achieved an average accuracy of 87.5%, outperforming the best-tested zero-shot model by a massive 29.7% margin.
Figure 3: Performance Radar Charts on HumanSense (mini). The results include human-level performance with several state-of-the-art multimodal models.
  • RL Training Breakthrough: The multi-stage RL framework drastically improved Qwen2.5-Omni-7B's performance on complex interactive tasks. For instance, its accuracy on the Psychological Chat (PC) task jumped from a baseline of 39.9% to 61.9% after stage-3 training.
  • Prompting Success: The training-free Prompt Enhancement (PE) strategy boosted the baseline model's Psychological Chat performance from 39.9% to 43.6% and improved familiarity recognition tasks without changing model weights.

Limitations and Future Work

While the paper demonstrates clear pathways to improving empathetic AI, some limitations remain:

  • API Constraints: Proprietary models like GPT-4o could not be fully evaluated on audio-native tasks due to API limitations that lacked direct audio input support at the time of testing.
  • Static Evaluation of Dynamic Interaction: Although the benchmark evaluates response planning, it relies on single-choice questions derived from real-world records rather than live, open-ended, multi-turn human-in-the-loop interactions.

The authors note that future work should focus on expanding these reasoning capabilities toward fully generative, real-time anthropomorphic expressions (integrating speech synthesis and talking heads) to achieve truly human-like AI companions.

Summary updated

Abstract

While Multimodal Large Language Models (MLLMs) show immense promise for achieving truly human-like interactions, progress is hindered by the lack of fine-grained evaluation frameworks for human-centered scenarios, encompassing both the understanding of complex human intentions and the provision of empathetic, context-aware responses. Here we introduce HumanSense, a comprehensive benchmark designed to evaluate the human-centered perception and interaction capabilities of MLLMs, with a particular focus on deep understanding of extended multimodal contexts and the formulation of rational feedback. Our evaluation reveals that leading MLLMs still have considerable room for improvement, particularly for advanced interaction-oriented tasks. Supplementing visual input with audio and text information yields substantial improvements, and Omni-modal models show advantages on these tasks.Furthermore, grounded in the observation that appropriate feedback stems from a contextual analysis of the interlocutor's needs and emotions, we posit that reasoning ability serves as the key to unlocking it. We devise a multi-stage, modality-progressive reinforcement learning approach, resulting in HumanSense-Omni-Reasoning, which substantially enhances performance on higher-level understanding and interactive tasks. Additionally, we observe that successful reasoning processes appear to exhibit consistent thought patterns. By designing corresponding prompts, we also enhance the performance of non-reasoning models in a training-free manner.Project page: \textcolor{brightpink}{https://digital-avatar.github.io/ai/HumanSense/}
BibTeX
@inproceedings{qin2026humansense,
  author = {Zheng Qin and Ruobing Zheng and Yabing Wang and Tianqi Li and Yi Yuan and Jingdong Chen and Le Wang},
  title = {{HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i30.39685},
}
Figure