TL;DR
This paper introduces HVSBench, the first large-scale benchmark designed to evaluate how well Multimodal Large Language Models (MLLMs) align with human visual perception. Experiments reveal a significant gap between current MLLMs and human performance across various visual tasks, highlighting the need for more human-aligned AI development.
What Problem Does This Paper Address?
While MLLMs have shown remarkable progress in vision-language tasks, there's a fundamental question: do they see like humans? Current benchmarks often focus on task-specific performance (like image captioning or VQA) but don't deeply probe whether MLLMs process visual information in a way that mirrors the human visual system (HVS). This includes how humans identify salient objects, estimate quantities, prioritize information, and follow gaze patterns during free viewing or visual search. Existing HVS research often relies on low-level vision-only domains and outputting heatmaps, which isn't directly compatible with MLLMs that output text. This paper aims to bridge this gap by creating a benchmark that specifically tests MLLM alignment with human perceptual behaviors.
Methods
The core contribution is HVSBench, a comprehensive benchmark comprising over 85,000 multimodal question-answer pairs. It's structured around five key fields of the HVS:
- Prominence: Tests if MLLMs focus on visually salient objects like humans do.
- Subitizing: Assesses if MLLMs can quickly estimate the number of salient objects.
- Prioritizing: Evaluates if MLLMs rank object importance according to human perception.
- Free-Viewing: Models bottom-up gaze patterns in task-free scenarios.
- Searching: Captures top-down gaze behavior during goal-driven tasks.
The benchmark includes 13 question types and uses a robust, human-inspired automatic standardization and evaluation protocol to handle diverse MLLM outputs. They also propose a novel evaluation protocol that minimizes false negatives and improves robustness.
Datasets
HVSBench is constructed using data from several existing, high-quality datasets relevant to HVS research:
- SIFR dataset: Used for Prominence and Prioritizing.
- SIS10K: Used for Subitizing.
- COCO-FreeView dataset: Used for Free-Viewing.
- COCO-Search18 dataset: Used for Searching.
Key Insights and Ideas
- The "Perceptual Gap" is Real: Even state-of-the-art MLLMs perform only moderately on HVSBench, significantly underperforming human participants. This suggests that current MLLMs don't inherently "see" or process visual information in a human-like manner.
- Scanpath Prediction is a Major Hurdle: MLLMs struggle significantly with accurately predicting human gaze patterns (scanpaths) during free-viewing and searching tasks. This is a key area where human-like alignment is lacking.
- Model Size Matters, But Isn't Everything: Larger models generally perform better, indicating that scaling up can improve HVS alignment. However, even the largest models don't fully bridge the gap.
- Beyond Simple Captions: Simply incorporating human-generated captions or detailed descriptions doesn't automatically lead to better HVS alignment, especially for dynamic tasks like prioritizing and scanpath prediction.
- Field-Specific Hints Can Help (Sometimes): Providing hints tailored to specific HVS fields can improve performance on tasks like Prominence and Subitizing, but can sometimes hinder performance on Free-Viewing and Searching by introducing too much prior context.
Key Results
- Human Performance: Human participants achieved an overall accuracy of 0.7438 on HVSBench, far exceeding random guessing (0.2806).
- Best MLLM Performance: The top-performing open-source model, Qwen3-VL-235B, achieved an overall accuracy of 0.4851, while the proprietary Gemini-2.5-pro reached 0.4754.
- Specific Task Performance: While MLLMs show some progress in tasks like Searching (e.g., Qwen3-VL-235B at 0.6259 accuracy), they still lag significantly behind humans. Prioritizing tasks (like Q3) proved particularly challenging, with many models achieving near-zero accuracy.
- Qualitative Examples: The paper includes examples showing MLLMs misidentifying salient objects, failing to count correctly, and generating scanpaths that don't match human behavior.
Limitations and Future Work
- Scanpath Prediction Challenges: The paper acknowledges that scanpath prediction remains a significant challenge for current MLLMs, with a notable gap to human performance.
- Task-Specific Performance Variation: No single model excels across all HVS fields, indicating that current architectures have varying strengths and weaknesses in mimicking human perception.
- Future Work: The authors suggest that HVSBench will be a critical tool for developing the next generation of explainable and human-aligned MLLMs. Future work could focus on improving scanpath prediction, understanding task variability, and exploring how to better integrate HVS principles into MLLM architectures. They also propose using HVSBench for instruction-tuning MLLMs.
Summary updated