Can Multimodal LLMs Solve the Basic Perception Problems of Percept-V?

Samrajnee Ghosh, Naman Agarwal, Hemanshu Garg, Chinmay Mittal, Mausam, Parag Singla

Introduction

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in high-level reasoning—such as solving complex physics problems or interpreting medical charts—a fundamental question remains: do these models actually "see" as well as a human? Current benchmarks often conflate perception (the ability to identify basic elements) with reasoning (the processing of that information).

Percept-V is a new benchmark designed to isolate basic visual perception from complex logic. By testing models on foundational tasks like counting shapes or identifying colors, the study reveals a startling "perception gap." Even advanced models struggle to match human-level accuracy, with performance plummeting as visual complexity increases.

Methodology: The Percept-V Framework

To ensure the evaluation was grounded in established cognitive science, the researchers developed Percept-V, a dataset of 6,000 programmatically generated images. These tasks are mapped to the TVPS-4 (Test of Visual Perceptual Skills) framework, which categorizes human perception into seven core skills:

  1. Visual Discrimination: Identifying differences and similarities.
  2. Visual Memory: Recalling specific traits.
  3. Visual Sequential Memory: Recalling sequences of objects.
  4. Visual Spatial Relationships: Understanding relative object positions.
  5. Visual Figure Ground: Isolating objects from busy backgrounds.
  6. Visual Form Constancy: Recognizing shapes despite changes in size or rotation.
  7. Visual Closure: Identifying a shape when it is only partially visible.
Figure 1: Overview of the Percept-V benchmark tasks and the performance gap between correct answers and GPT-4o predictions.

The images are "uncontaminated," generated by scripts to ensure they were not part of the models' training data. By varying the "problem size"—the number of objects or distractors—the authors could observe how model performance scales with visual complexity.

Key Insights: The Complexity Wall

The Scaling Collapse

MLLMs exhibit a distinct "complexity wall." While they may solve tasks involving two or three objects with high accuracy, their performance drops exponentially as more objects are added. This suggests that current models lack the robust scanning and tracking abilities inherent in human vision.

Thinking vs. Seeing

Interestingly, "thinking" models that utilize intermediate reasoning steps (like o1-mini) significantly outperformed non-thinking models. This indicates that internal chain-of-thought can partially compensate for weak raw perception, though it does not solve the underlying vision problem.

Spatial Strengths and Perceptual Weaknesses

Models performed best on "Visual Spatial Relationships" (e.g., determining if a circle is above a square), likely due to the abundance of spatial descriptions in image-captioning training data. Conversely, they struggled most with "Visual Closure" and "Figure Ground" tasks, which require isolating specific signals from noise.

Figure 2: An example of the Visual Figure Ground task where GPT-4o fails to identify all unique shapes overlaid on a background image.

Results and Performance Gap

The performance disparity between humans and AI is stark. While human participants achieved an average accuracy of 95.43%, the best-performing model reached only 55.22%. The divide between proprietary and open-source models is equally massive; top proprietary models hovered around 50-55%, while some open-source models struggled at a dismal 6.43%. In specific domains like color identification (colours_present), models were nearly blind, averaging only 2.33% accuracy.

Conclusion and Future Work

The Percept-V benchmark highlights a critical bottleneck in MLLM development. While these models are becoming better "thinkers," their foundational "vision" remains underdeveloped. Future research must determine whether these failures stem from the vision encoder (e.g., CLIP) losing information or the LLM failing to integrate that information. Addressing these fundamental perceptual blind spots is essential for creating truly robust multimodal systems.

Summary updated

BibTeX
@inproceedings{ghosh2026can,
  author = {Samrajnee Ghosh and Naman Agarwal and Hemanshu Garg and Chinmay Mittal and Mausam and Parag Singla},
  title = {{Can Multimodal LLMs Solve the Basic Perception Problems of Percept-V?}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=GDd3TTAbzR},
}
Figure