TL;DR
This paper introduces MV-RoboBench, the first benchmark specifically designed to evaluate the multi-view spatial reasoning capabilities of Vision-Language Models (VLMs) in robotic manipulation. By testing models across 1.7k human-curated questions, the authors reveal a massive performance gap between state-of-the-art models and humans, proving that scaling up standard text/image perception is insufficient for robust embodied AI.
What Problem Does This Paper Address?
Embodied AI relies heavily on VLMs to perceive environments and plan actions. While multi-camera setups (e.g., head-mounted and gripper-mounted cameras) are standard in modern robotics to overcome occlusions and depth ambiguity, most VLM evaluations remain strictly single-view.
Whether current VLMs can effectively integrate information across multiple synchronized viewpoints to build a coherent 3D understanding of a workspace remains an open question. This paper addresses this gap, demonstrating that a model's ability to reason within a single image does not reliably translate to the multi-view spatial understanding required for real-world robotic execution.
Methods
The authors developed a structured evaluation pipeline to test VLMs on multi-view inputs. They formulated 1,708 five-choice multiple-choice questions divided into two primary categories across eight subtasks:
- Spatial Understanding: Cross-View Matching, Distance Judgement, Viewpoint Identification, and 3D Spatial Consistency.
- Robotic Execution: Action Planning, Step Execution, Trajectory Selection, and Affordance Recognition.
Beyond standard zero-shot testing, the authors explored three Chain-of-Thought (CoT)-inspired enhancements to assist the models:
- Textual CoT: Adding holistic scene descriptions generated by GPT-4.1.
- Visual CoT: Generating intermediate synthetic views using a Visual Geometry Grounded Transformer (VGGT).
- Structural CoT: Providing predicted depth maps via MoGe-2 to introduce explicit geometric constraints.
To test the robustness of these models, they also introduced two stress tests: a vertical image inversion test (flipping views upside down) and an option rejection test (replacing the correct answer with "None of the above").
Datasets
The benchmark is constructed using real-world robotic demonstration data from two prominent datasets: AgiWorld and BridgeV2. It spans both single-arm and dual-arm robotic manipulation settings across 980 unique episodes. Human experts manually filtered the image pairs to ensure visual clarity and stable multi-view correspondences, avoiding the automated pattern-completion shortcuts common in synthetic datasets.
Key Insights and Ideas
- The Spatial-Robotic Correlation: The benchmark uncovers a clear positive correlation between spatial intelligence and robotic task execution. Models that fail to grasp the 3D spatial layout consistently fail at action planning and trajectory selection.
- Single-View Benchmarks Mislead: Strong performance on general-purpose, single-view spatial benchmarks (like OmniSpatial) does not guarantee success in multi-view robotic settings. Multi-view reasoning introduces entirely new challenges, such as viewpoint fusion and frame-of-reference transformations.
- Severe Over-Compliance: In the option rejection stress test, model performance collapsed universally. Current VLMs exhibit a strong bias toward selecting a "visually approximate" option rather than recognizing that no correct answer is present—a significant safety hazard for physical deployment.
- Orientation Bias: Advanced models suffer a dramatic performance drop when images are flipped upside down. This indicates that their spatial reasoning relies heavily on the canonical "upright" orientations learned during pre-training, rather than an intrinsic understanding of 3D geometry.
Key Results
- The Human Baseline: Human participants nearly solved the benchmark, achieving an average accuracy of 91.04%.
- The VLM Performance Gap: The highest-performing model, GPT-5, led all models but only achieved an average accuracy of 56.41%, leaving a substantial gap to human proficiency.
- Reasoning Models vs. Standard VLMs: Proprietary reasoning-optimized models (such as o4-mini at 46.47% and Gemini-2.5-pro at 49.52%) significantly outperformed standard open-source models. For instance, Qwen2.5-vl-72b scored 24.29%, hovering just above the 20% random choice baseline.
- The Hardest Task: 3D Spatial Consistency proved exceptionally difficult. Most non-reasoning models performed near or below random guessing (~19%), highlighting a fundamental inability to map objects into a coherent 3D coordinate grid from multiple views.
Limitations and Future Work
The qualitative error analysis reveals that current VLMs lack robust internal 3D scene representations, frequently substituting simple 2D screen-space heuristics (like apparent size or 2D pixel alignment) for true 3D reasoning. Furthermore, shallow prompt-level CoT enhancements often yielded mixed or negative results, showing that prompt engineering cannot fix deep geometric deficiencies.
The authors conclude that future progress in Embodied AI will depend on:
- Model architectures that explicitly encode geometric priors and enforce cross-view consistency.
- Training pipelines that tightly couple multi-view perception with action grounding.
- Larger-scale, real-world multi-camera datasets to move beyond pattern matching toward genuine spatial reasoning.
Summary updated