HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes

Keliang Li, Hongze Shen, Hao Shi, Ruibing Hou, Hong Chang, Jie Huang, Chenghao Jia, Wen Wang, Yiling Wu, Dongmei Jiang, Shiguang Shan, Xilin Chen

TL;DR

HumanPCR is a new hierarchical evaluation suite designed to test Multimodal Large Language Models (MLLMs) on fine-grained human-centric visual tasks across three levels: Perception, Comprehension, and Reasoning. The benchmark reveals a stark reality: while advanced models excel at basic perception, they severely struggle with complex human-centric reasoning because they rely too heavily on explicit question cues rather than proactively extracting implicit visual evidence.

What Problem Does This Paper Address?

While MLLMs have advanced rapidly, their ability to truly understand humans in complex, dynamic, real-world scenarios remains flawed. Existing benchmarks are either too narrow (focusing solely on faces or egocentric actions) or lack the depth required for fine-grained human-centric evaluation (such as tracking gaze estimation, body orientation, or mind modeling).

Crucially, many current video reasoning benchmarks allow for "shortcuts." Models can often guess the correct answer using single-frame matching or text-based priors without genuinely understanding the continuous, multi-step human activities taking place. The authors address this gap by introducing a benchmark that explicitly demands proactive visual evidence extraction—requiring models to uncover implicit visual cues that aren't explicitly mentioned in the prompt.

Overview of the HumanPCR evaluation suite and comparison with existing benchmarks.

Methods

The authors structure HumanPCR around a three-level hierarchical taxonomy:

  1. Human-P (Perception): Evaluates fine-grained elements like spatial relations, body posture, hand states, and human-object contact.
  2. Human-C (Comprehension): Assesses the integration of common sense to understand multi-human sequential actions, social relations, and broader scene dynamics.
  3. Human-R (Reasoning): Evaluates complex, open-ended video reasoning across five categories: Causal Reasoning, Prediction, Counter-Factual Reasoning, Assessment, and Planning.

To build HumanPCR, the authors implemented a rigorous multi-stage quality control pipeline. They utilized LLM filtering to eliminate questions answerable without visual context, followed by human verification. For the reasoning subset (Human-R), expert meta-reviewers enforced strict criteria to ensure questions required multi-evidence integration and proactive context extraction. Furthermore, every reasoning question is augmented with human-annotated Chain-of-Thought (CoT) rationales that map directly to key visual evidence.

Datasets

The paper introduces the HumanPCR benchmark, which consists of:

  • Over 6,000 human-verified multiple-choice QA pairs split across the Perception (Human-P) and Comprehension (Human-C) levels.
  • 442 open-ended video reasoning questions for the Reasoning (Human-R) level.

The dataset spans 9 dimensions, 34 distinct tasks, and 11 diverse human-related domains, including outdoor adventures, household activities, sports, and daily spending.

Key Insights and Ideas

  • The "Query-Guided" Crutch: Error analysis reveals that current MLLMs heavily depend on explicit query cues. If the visual evidence is directly referenced in the question, models can retrieve it. However, if the task requires extracting implicit, proactive context to understand a human's intent or next move, the models frequently fail.
  • More Frames $\neq$ Better Reasoning: Simply scaling up the number of video frames or context length does not solve the reasoning bottleneck. For highly complex questions requiring 6+ pieces of visual evidence, adding more frames actually introduces noise and degrades model performance.
  • The Open-Source Reasoning Gap: Open-source models (such as the InternVL3 series) are highly competitive—and sometimes superior—at basic perception and comprehension tasks. However, they lag significantly behind proprietary models in complex reasoning, where advanced test-time thinking architectures (like OpenAI's o3) show a clear advantage.
Illustration of proactive vs. referred visual evidence shortcuts.

Key Results

  • Perception & Comprehension: Most state-of-the-art models score below 60% accuracy on Human-P and Human-C. InternVL3-78B and o4-mini led the multi-choice suite with macro-accuracies of 62.77% and 62.28%, respectively, though both showed distinct weaknesses in tracking fine-grained postures and contact regions.
  • Complex Reasoning: On the open-ended Human-R dataset, OpenAI’s o3 achieved the top score of 59.28%, followed by o4-mini at 53.39%. Meanwhile, the vast majority of open-source models struggled to break past 30% accuracy.
  • Test-Time Scaling: Implementing advanced test-time compute strategies like Best-of-N (BoN) sampling yielded steady but modest improvements (~5%), while iterative Self-Refine strategies often degraded reasoning performance.

Limitations and Future Work

The authors note a few limitations in their current framework. First, the perception and comprehension levels rely heavily on reorganizing existing academic datasets rather than entirely new, in-the-wild data. Second, evaluating open-ended reasoning answers requires an LLM judge (o3-mini), which can introduce evaluation biases.

For future work, the authors plan to expand HumanPCR into more specialized professional domains and develop objective, non-LLM-reliant evaluation protocols that preserve the benchmark's inherent reasoning difficulty.

Summary updated

Abstract

The aspiration for artificial general intelligence, fueled by the rapid progress of multimodal models, demands human-comparable performance across diverse environments. We propose HumanPCR, an evaluation suite for probing MLLMs' capacity about human-related visual contexts across three hierarchical levels: Perception, Comprehension, and Reasoning (denoted by Human-P, Human-C, and Human-R, respectively). Human-P and Human-C feature over 6,000 human-verified multiple choice questions, assessing massive tasks of 9 dimensions, including but not limited to essential skills frequently overlooked by existing benchmarks. Human-R offers a challenging manually curated video reasoning test that requires integrating multiple visual evidences, proactively extracting context beyond question cues, and applying human-like expertise. Each question includes human-annotated Chain-of-Thought (CoT) rationales with key visual evidence to support further research. Extensive evaluations on over 30 state-of-the-art models exhibit significant challenges in human-centric visual understanding, particularly in tasks involving detailed space perception, temporal understanding, and mind modeling. Moreover, analysis of Human-R reveals the struggle of models in extracting essential proactive visual evidence from diverse human scenes and their faulty reliance on query-guided retrieval. Even with advanced techniques like scaling visual contexts and test-time thinking yield only limited benefits. We hope HumanPCR and our findings will advance the development, evaluation, and human-centric application of multimodal models.
BibTeX
@inproceedings{li2026humanpcr,
  author = {Keliang Li and Hongze Shen and Hao Shi and Ruibing Hou and Hong Chang and Jie Huang and Chenghao Jia and Wen Wang and Yiling Wu and Dongmei Jiang and Shiguang Shan and Xilin Chen},
  title = {{HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=I6LUSZMJLa},
}
Figure