Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes

Zhiyuan Feng, Zhaolu Kang, Qijie Wang, Zhiying Du, Jiongrui Yan, Shubin Shi, Chengbo Yuan, Huizhi Liang, Yu Deng, Qixiu Li, Rushuai Yang, Arctanx An, Leqi Zheng, Weijie Wang, Shawn Chen, Sicheng Xu, Yaobo Liang, Jiaolong Yang, Baining Guo

TL;DR

This paper introduces MV-RoboBench, the first benchmark specifically designed to evaluate the multi-view spatial reasoning capabilities of Vision-Language Models (VLMs) in robotic manipulation. By testing models across 1.7k human-curated questions, the authors reveal a massive performance gap between state-of-the-art models and humans, proving that scaling up standard text/image perception is insufficient for robust embodied AI.

What Problem Does This Paper Address?

Embodied AI relies heavily on VLMs to perceive environments and plan actions. While multi-camera setups (e.g., head-mounted and gripper-mounted cameras) are standard in modern robotics to overcome occlusions and depth ambiguity, most VLM evaluations remain strictly single-view.

Whether current VLMs can effectively integrate information across multiple synchronized viewpoints to build a coherent 3D understanding of a workspace remains an open question. This paper addresses this gap, demonstrating that a model's ability to reason within a single image does not reliably translate to the multi-view spatial understanding required for real-world robotic execution.

Methods

The authors developed a structured evaluation pipeline to test VLMs on multi-view inputs. They formulated 1,708 five-choice multiple-choice questions divided into two primary categories across eight subtasks:

  1. Spatial Understanding: Cross-View Matching, Distance Judgement, Viewpoint Identification, and 3D Spatial Consistency.
  2. Robotic Execution: Action Planning, Step Execution, Trajectory Selection, and Affordance Recognition.
Representative multi-view QA instances from the eight tasks in MV-RoboBench.

Beyond standard zero-shot testing, the authors explored three Chain-of-Thought (CoT)-inspired enhancements to assist the models:

  • Textual CoT: Adding holistic scene descriptions generated by GPT-4.1.
  • Visual CoT: Generating intermediate synthetic views using a Visual Geometry Grounded Transformer (VGGT).
  • Structural CoT: Providing predicted depth maps via MoGe-2 to introduce explicit geometric constraints.

To test the robustness of these models, they also introduced two stress tests: a vertical image inversion test (flipping views upside down) and an option rejection test (replacing the correct answer with "None of the above").

Datasets

The benchmark is constructed using real-world robotic demonstration data from two prominent datasets: AgiWorld and BridgeV2. It spans both single-arm and dual-arm robotic manipulation settings across 980 unique episodes. Human experts manually filtered the image pairs to ensure visual clarity and stable multi-view correspondences, avoiding the automated pattern-completion shortcuts common in synthetic datasets.

Key Insights and Ideas

  • The Spatial-Robotic Correlation: The benchmark uncovers a clear positive correlation between spatial intelligence and robotic task execution. Models that fail to grasp the 3D spatial layout consistently fail at action planning and trajectory selection.
  • Single-View Benchmarks Mislead: Strong performance on general-purpose, single-view spatial benchmarks (like OmniSpatial) does not guarantee success in multi-view robotic settings. Multi-view reasoning introduces entirely new challenges, such as viewpoint fusion and frame-of-reference transformations.
  • Severe Over-Compliance: In the option rejection stress test, model performance collapsed universally. Current VLMs exhibit a strong bias toward selecting a "visually approximate" option rather than recognizing that no correct answer is present—a significant safety hazard for physical deployment.
  • Orientation Bias: Advanced models suffer a dramatic performance drop when images are flipped upside down. This indicates that their spatial reasoning relies heavily on the canonical "upright" orientations learned during pre-training, rather than an intrinsic understanding of 3D geometry.

Key Results

  • The Human Baseline: Human participants nearly solved the benchmark, achieving an average accuracy of 91.04%.
  • The VLM Performance Gap: The highest-performing model, GPT-5, led all models but only achieved an average accuracy of 56.41%, leaving a substantial gap to human proficiency.
  • Reasoning Models vs. Standard VLMs: Proprietary reasoning-optimized models (such as o4-mini at 46.47% and Gemini-2.5-pro at 49.52%) significantly outperformed standard open-source models. For instance, Qwen2.5-vl-72b scored 24.29%, hovering just above the 20% random choice baseline.
  • The Hardest Task: 3D Spatial Consistency proved exceptionally difficult. Most non-reasoning models performed near or below random guessing (~19%), highlighting a fundamental inability to map objects into a coherent 3D coordinate grid from multiple views.
Best-per-group model performance across MV-RoboBench subtasks compared to humans.

Limitations and Future Work

The qualitative error analysis reveals that current VLMs lack robust internal 3D scene representations, frequently substituting simple 2D screen-space heuristics (like apparent size or 2D pixel alignment) for true 3D reasoning. Furthermore, shallow prompt-level CoT enhancements often yielded mixed or negative results, showing that prompt engineering cannot fix deep geometric deficiencies.

The authors conclude that future progress in Embodied AI will depend on:

  1. Model architectures that explicitly encode geometric priors and enforce cross-view consistency.
  2. Training pipelines that tightly couple multi-view perception with action grounding.
  3. Larger-scale, real-world multi-camera datasets to move beyond pattern matching toward genuine spatial reasoning.

Summary updated

Abstract

Vision-language models (VLMs) are essential to Embodied AI, enabling robots to perceive, reason, and act in complex environments. They also serve as the foundation for the recent Vision-Language-Action (VLA) models. Yet most evaluations of VLMs focus on single-view settings, leaving their ability to integrate multi-view information underexplored. At the same time, multi-camera setups are increasingly standard in robotic platforms, as they provide complementary perspectives to mitigate occlusion and depth ambiguity. Whether VLMs can effectively leverage such multi-view inputs for robotic reasoning therefore remains an open question. To bridge this gap, we introduce MV-RoboBench, a benchmark specifically designed to evaluate the multi-view spatial reasoning capabilities of VLMs in robotic manipulation. MV-RoboBench consists of 1.7k manually curated QA items across eight subtasks, divided into two primary categories: spatial understanding and robotic execution. We evaluate a diverse set of existing VLMs, including both open-source and closed-source models, along with enhanced versions incorporating CoT-inspired techniques. The results show that state-of-the-art models remain far below human performance, underscoring the substantial challenges VLMs face in multi-view robotic perception. Additionally, our analysis uncovers two key findings: (i) spatial intelligence and robotic task execution are positively correlated in multi-view robotic scenarios; and (ii) strong performance on existing general-purpose single-view spatial understanding benchmarks does not reliably translate to success in the robotic spatial tasks assessed by our benchmark. We release MV-RoboBench as an open resource to foster progress in spatially grounded VLMs and VLAs, providing not only data but also a standardized evaluation protocol for multi-view embodied reasoning.
BibTeX
@inproceedings{feng2026seeing,
  author = {Zhiyuan Feng and Zhaolu Kang and Qijie Wang and Zhiying Du and Jiongrui Yan and Shubin Shi and Chengbo Yuan and Huizhi Liang and Yu Deng and Qixiu Li and Rushuai Yang and Arctanx An and Leqi Zheng and Weijie Wang and Shawn Chen and Sicheng Xu and Yaobo Liang and Jiaolong Yang and Baining Guo},
  title = {{Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=jXDZJAfRZB},
}
Figure