Introduction
While Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in high-level reasoning—such as solving complex physics problems or interpreting medical charts—a fundamental question remains: do these models actually "see" as well as a human? Current benchmarks often conflate perception (the ability to identify basic elements) with reasoning (the processing of that information).
Percept-V is a new benchmark designed to isolate basic visual perception from complex logic. By testing models on foundational tasks like counting shapes or identifying colors, the study reveals a startling "perception gap." Even advanced models struggle to match human-level accuracy, with performance plummeting as visual complexity increases.
Methodology: The Percept-V Framework
To ensure the evaluation was grounded in established cognitive science, the researchers developed Percept-V, a dataset of 6,000 programmatically generated images. These tasks are mapped to the TVPS-4 (Test of Visual Perceptual Skills) framework, which categorizes human perception into seven core skills:
- Visual Discrimination: Identifying differences and similarities.
- Visual Memory: Recalling specific traits.
- Visual Sequential Memory: Recalling sequences of objects.
- Visual Spatial Relationships: Understanding relative object positions.
- Visual Figure Ground: Isolating objects from busy backgrounds.
- Visual Form Constancy: Recognizing shapes despite changes in size or rotation.
- Visual Closure: Identifying a shape when it is only partially visible.
The images are "uncontaminated," generated by scripts to ensure they were not part of the models' training data. By varying the "problem size"—the number of objects or distractors—the authors could observe how model performance scales with visual complexity.
Key Insights: The Complexity Wall
The Scaling Collapse
MLLMs exhibit a distinct "complexity wall." While they may solve tasks involving two or three objects with high accuracy, their performance drops exponentially as more objects are added. This suggests that current models lack the robust scanning and tracking abilities inherent in human vision.
Thinking vs. Seeing
Interestingly, "thinking" models that utilize intermediate reasoning steps (like o1-mini) significantly outperformed non-thinking models. This indicates that internal chain-of-thought can partially compensate for weak raw perception, though it does not solve the underlying vision problem.
Spatial Strengths and Perceptual Weaknesses
Models performed best on "Visual Spatial Relationships" (e.g., determining if a circle is above a square), likely due to the abundance of spatial descriptions in image-captioning training data. Conversely, they struggled most with "Visual Closure" and "Figure Ground" tasks, which require isolating specific signals from noise.
Results and Performance Gap
The performance disparity between humans and AI is stark. While human participants achieved an average accuracy of 95.43%, the best-performing model reached only 55.22%. The divide between proprietary and open-source models is equally massive; top proprietary models hovered around 50-55%, while some open-source models struggled at a dismal 6.43%. In specific domains like color identification (colours_present), models were nearly blind, averaging only 2.33% accuracy.
Conclusion and Future Work
The Percept-V benchmark highlights a critical bottleneck in MLLM development. While these models are becoming better "thinkers," their foundational "vision" remains underdeveloped. Future research must determine whether these failures stem from the vision encoder (e.g., CLIP) losing information or the LLM failing to integrate that information. Addressing these fundamental perceptual blind spots is essential for creating truly robust multimodal systems.
Summary updated