HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs

Ruobing Zheng

TL;DR

This paper introduces HumanSense, a novel benchmark for evaluating multimodal large language models (MLLMs) on human-centered interaction tasks, addressing the current lack of fine-grained assessment in this area. They demonstrate that reasoning capabilities, particularly in omni-modal models that integrate vision, audio, and text, are crucial for empathetic and context-aware responses, leading to significant performance improvements.

What Problem Does This Paper Address?

The field of AI is moving towards more human-like interactions, with Multimodal Large Language Models (MLLMs) showing great promise. However, current evaluation frameworks for these models are insufficient for assessing their ability to truly understand complex human intentions and provide empathetic, context-aware responses. This gap hinders the development of AI companions that can offer social support and emotional connection, moving beyond simple task execution. The paper aims to bridge this gap by creating a comprehensive benchmark that specifically targets these human-centered capabilities.

Methods

The core of this work is the HumanSense benchmark, a hierarchical evaluation framework structured into four tiers: Perception (L1 & L2), Understanding (L3), and Response (L4). This structure ensures a systematic evaluation, starting from basic sensory input processing to complex interactive decision-making.

To improve MLLM performance, the authors propose a multi-stage, omni-modal reinforcement learning approach. This method progressively trains an omni-modal model (Qwen2.5-Omni-7B) by exposing it to different modalities (video, audio, text) in stages, enhancing its reasoning capabilities. They also introduce a training-free prompt enhancement technique, inspired by observed reasoning patterns, to boost the performance of existing MLLMs without additional training.

Datasets

The HumanSense benchmark is constructed using data from existing open-source datasets and YouTube videos, including:

  • CelebV-HQ: For person appearance and emotion recognition.
  • AVA (Atomic Visual Actions): For action recognition.
  • Jester: For gesture recognition.
  • ActivityNet: For complex motion understanding.
  • UCF-Crime100: For abnormal classification.
  • TAO: For human counting with ReID.
  • RealTalk: For talking person recognition and emotion strategy.
  • The Skin Deep: For relation, familiarity, and rapport recognition.
  • SEUMLD: For lie detection.
  • Telecom Fraud Texts: For fraud recognition.
  • Emotional First Aid dataset: For psychological chat.

The paper also mentions a smaller, curated evaluation set called HumanSense (tiny) for establishing a human baseline.

Key Insights and Ideas

  • Reasoning is Key: The paper strongly posits that reasoning ability is the critical factor for unlocking empathetic and context-aware responses in MLLMs, especially when considering the interlocutor's needs and emotions.
  • Omni-modal Advantage: Omni-modal models (integrating vision, audio, and text) significantly outperform visual-only models, particularly in higher-level understanding and interaction tasks.
  • Audio's Crucial Role: While vision is foundational, audio input provides substantial improvements, especially for tasks requiring nuanced understanding of human interaction.
  • Consistent Reasoning Patterns: Successful reasoning processes in MLLMs tend to follow a predictable pattern: perceiving characteristics, emotion, and context, followed by thinking and then responding. This observation led to the prompt enhancement technique.
  • Current MLLMs Lag Behind Humans: Despite advancements, there's a significant performance gap between state-of-the-art MLLMs and human-level performance, especially in complex interaction tasks.

Key Results

  • Human Performance Benchmark: Human evaluators achieved an average accuracy of 87.5% on the HumanSense benchmark.
  • Model Performance: The best-performing model (GPT-40) achieved an average accuracy of 57.8%, indicating a substantial gap.
  • Omni-modal Superiority: Omni-modal models, particularly Qwen2.5-Omni, showed marked advantages over visual-only LLMs in higher-level tasks.
  • Prompt Enhancement Effectiveness: The proposed prompt enhancement technique improved performance on high-level tasks without requiring additional training.
  • Reinforcement Learning Gains: The multi-stage reinforcement learning approach significantly enhanced the reasoning and interactive capabilities of the omni-modal model.

Limitations and Future Work

  • Performance Gap: A significant performance gap still exists between current MLLMs and human-level performance, particularly in complex L3-L4 tasks.
  • Reasoning Bottleneck: For current leading omni-models, the primary bottleneck appears to be in high-level, long-context reasoning rather than low-level perception.
  • Future Work: The authors aim to inspire the community to explore the potential of omni-MLLMs for improving human-centered AI interactions and to further advance this emerging direction. They also provide their benchmark and code to facilitate broader evaluation.

Summary updated

Abstract

While Multimodal Large Language Models (MLLMs) show immense promise for achieving truly human-like interactions, progress is hindered by the lack of fine-grained evaluation frameworks for human-centered scenarios, encompassing both the understanding of complex human intentions and the provision of empathetic, context-aware responses. Here we introduce HumanSense, a comprehensive benchmark designed to evaluate the human-centered perception and interaction capabilities of MLLMs, with a particular focus on deep understanding of extended multimodal contexts and the formulation of rational feedback. Our evaluation reveals that leading MLLMs still have considerable room for improvement, particularly for advanced interaction-oriented tasks. Supplementing visual input with audio and text information yields substantial improvements, and Omni-modal models show advantages on these tasks. Furthermore, we argue that appropriate feedback stems from a contextual analysis of the interlocutor's needs and emotions, with reasoning ability serving as the key to unlocking it. Accordingly, we employ a multi-stage, modality-progressive reinforcement learning to enhance the reasoning abilities of an Omni model, achieving substantial gains on evaluation results. Additionally, we observe that successful reasoning processes exhibit highly consistent thought patterns. By designing corresponding prompts, we also enhance the performance of non-reasoning models in a training-free manner.
BibTeX
@inproceedings{zheng2026humansense,
  author = {Ruobing Zheng},
  title = {{HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i30.39685},
}
Figure