TL;DR
This paper introduces Jigsaw-Puzzles, a novel benchmark designed to rigorously evaluate the spatial reasoning capabilities of Vision-Language Models (VLMs) in realistic visual scenarios. The authors demonstrate that even state-of-the-art VLMs struggle significantly with these tasks, highlighting a critical gap in their ability to understand and reason about the physical world, which is essential for achieving Artificial General Intelligence (AGI).
What Problem Does This Paper Address?
Human cognition relies heavily on spatial reasoning – the ability to perceive, comprehend, and interact with the physical world by understanding spatial structures and relationships. This capability is fundamental for complex reasoning and decision-making. While current VLMs have made impressive strides in visual understanding and language processing, their ability to perform robust spatial reasoning remains largely untested and underdeveloped. Existing benchmarks often use synthetic images or simplified scenarios, failing to capture the complexity and ambiguity of real-world visual environments. This limits our understanding of how well VLMs can truly grasp spatial concepts, hindering progress towards more generalizable and human-like AI. The paper aims to bridge this gap by introducing a challenging benchmark that specifically targets spatial reasoning in realistic settings.
Methods
The core of this paper is the introduction of the Jigsaw-Puzzles benchmark. This benchmark is inspired by the cognitive process of solving jigsaw puzzles, which involves multiple stages: perception, structural understanding, and reasoning. The benchmark consists of five tasks designed to evaluate different facets of spatial reasoning:
- Missing Piece Selection: Tests basic spatial understanding by requiring VLMs to identify the correct piece to fill a missing region in an image.
- Piece Localization: Assesses spatial localization by asking VLMs to identify the original position of a masked patch.
- Connection Verification: Evaluates adjacency reasoning by determining the spatial relationship between two cropped image patches.
- Anomaly Detection: Targets local spatial transformation detection, requiring VLMs to identify and classify changes (rotation, mirroring) in a shuffled image.
- Order Restoration: A complex multi-step reasoning task where VLMs must reconstruct an original image from four shuffled patches.
The benchmark's tasks are generated automatically using specific templates, ensuring scalability and consistency. The difficulty of tasks is managed through variations in distractors, grid sizes, and the number of masked regions.
Datasets
The paper introduces Jigsaw-Puzzles, a novel benchmark consisting of 1,100 carefully curated real-world images with high spatial complexity. The dataset was built upon the CC3M dataset, with images filtered based on expert-defined rules to ensure structural diversity and avoid explicit content. The authors also created a smaller subset, Jigsaw-Puzzles-Lite, for human performance evaluation.
Key Insights and Ideas
- Spatial Reasoning is a Major Bottleneck: Even the most advanced VLMs, like Gemini-2.5-Pro, struggle significantly with spatial reasoning tasks, particularly multi-step ones like Order Restoration, falling far short of human performance.
- Foundational Understanding is Crucial: Performance on spatial reasoning tasks is strongly correlated with performance on basic spatial understanding tasks (like Missing Piece Selection), suggesting that a robust foundation in perceiving spatial arrangements is essential for higher-level reasoning.
- Model Size Matters: There's a clear positive correlation between VLM size (parameter count) and overall performance on Jigsaw-Puzzles, indicating that larger models generally exhibit stronger spatial reasoning capabilities.
- Reasoning-Enhanced Models Show Promise: VLMs specifically designed or fine-tuned for reasoning show a notable improvement over their base counterparts, especially in multi-step tasks, suggesting that architectural or training modifications can enhance spatial reasoning.
- Self-Correction is a Key Factor: Reasoning-enhanced models, particularly Gemini-2.5-Pro, demonstrate an ability to self-correct when faced with limited answer choices, which contributes to their superior performance on tasks like Order Restoration.
- Open-Ended Reasoning Remains Challenging: While VLMs can perform well with multiple-choice options, their ability to generate correct spatial sequences in an open-ended setting (Order Generation) is severely limited, highlighting a significant gap compared to humans.
Key Results
- The strongest model, Gemini-2.5-Pro, achieved an overall accuracy of 77.14% on Jigsaw-Puzzles, significantly lower than human performance (around 96.36%).
- Performance on the Order Restoration task was particularly poor, with Gemini-2.5-Pro achieving only 30.00% accuracy in an open-ended setting (Order Generation), compared to human performance of 94.09%.
- Proprietary VLMs generally outperformed open-source models across the board.
- Reasoning-enhanced models like Gemini-2.5-Flash-Thinking and Gemini-2.5-Pro showed substantial improvements over their base versions, with Gemini-2.5-Pro reaching 77.14% overall accuracy.
Limitations and Future Work
The authors acknowledge that Jigsaw-Puzzles is currently focused on 2D spatial reasoning in static images. They suggest future work should extend the benchmark to include:
- 3D perception: Evaluating understanding of three-dimensional space.
- Temporal sequences: Assessing reasoning about spatial changes over time.
- Embodied contexts: Incorporating scenarios that involve interaction with the physical environment.
They encourage future research to build upon this benchmark to advance spatial reasoning capabilities in VLMs, ultimately contributing to the development of more generalizable AI.
Summary updated