SpatialScout-V2: Can VLM Agents Actively Explore Real-World 3D Scenes and Localize Objects?

Zhaoxi Zhang, Hao Liang, Linxuan Wu, Yiming Xu, Yifu Yuan, Kun Liang, Chenming Tang, Meiyi Qiang, Wentao Zhang, Weikang Li, Yunfang WuarXiv 2026 · arXiv

TL;DR

SpatialScout-V2 is a new benchmark that challenges Vision-Language Model (VLM) agents to actively explore real-world residential 3D scenes and localize objects. Despite the impressive reasoning capabilities of frontier models like Claude 3.5 and GPT-4 (referred to here as GPT-6 Astra), the results are a reality check: even the best-performing agent achieves an average 3D localization accuracy of only 22.5%, highlighting a massive gap in current "spatial intelligence."

What Problem Does This Paper Address?

While VLMs have become adept at describing images and answering questions, "spatial intelligence"—the ability to understand and navigate the 3D world—remains a frontier. Most existing evaluations use static images or simulated environments. Simulation is a great starting point, but it often lacks the messy complexity of the real world: realistic lighting, complex occlusions, and the sheer variety of household clutter.

The authors argue that true spatial intelligence requires an agent to:

  1. Actively explore: Decide where to look next to resolve ambiguity.
  2. Integrate evidence: Combine partial 2D views into a coherent 3D representation.
  3. Reason geometrically: Understand how a point in a 2D image translates to a coordinate in 3D space.

SpatialScout-V2 moves the goalposts from simulation to reality, asking if VLM agents can build an accurate "object-center map" of a real home using only a camera and a basic triangulation tool.

Methods

The benchmark places a VLM agent in a "recorded" environment—a set of high-resolution panoramic captures from real homes. The agent is given a "target inventory" (e.g., "find the sofa and the microwave") and must navigate between fixed camera positions to find them.

The interaction loop is entirely autonomous:

  • Observation: The agent selects a viewpoint and viewing direction to receive a perspective image.
  • Reasoning: The agent processes the visual data and decides whether to move, look elsewhere, or attempt a measurement.
  • Triangulation: Since VLMs struggle with raw depth estimation, the benchmark provides a triangulate_observations tool. The agent marks the same object in two different views, and the tool calculates the 3D coordinate.
  • Mapping: The agent iteratively submits and revises the 3D centers of the target objects within a 120-turn budget.
Figure 1. Overview of SpatialScout-V2, a benchmark built from real-world scenes for 3D object localization. A VLM agent controls the camera, conducts automated reasoning, and submits the world coordinates of all spotted objects.

Datasets

SpatialScout-V2 is built upon Realsee3D, a massive dataset of real-world residential captures.

  • Scale: 100 diverse homes containing 3,171 target objects.
  • Diversity: 29 object categories, ranging from large furniture like sofas to smaller items like nightstands and kitchen sinks.
  • Quality: Unlike the original SpatialScout (v1), which used simulated assets, v2 uses real photographs. The ground truth 3D boxes were generated via an automated pipeline and then manually verified by humans to ensure accuracy.
The diversity of objects in the benchmark

Key Insights and Ideas

  • The "Fine Localization" Wall: Models are surprisingly good at finding the general area of an object (66% of targets had some overlap), but they fail miserably at precise placement. Only 0.19% of the best model's predictions achieved an IoU (Intersection over Union) greater than 0.9.
  • Coverage $\neq$ Accuracy: An agent might "find" every object in the house (high coverage), but if its 3D estimates are off by a meter, the map is useless. The paper shows that broad submission coverage often masks very poor localization quality.
  • Geometric Reasoning is the Bottleneck: Higher-performing models (Claude Opus, GPT-6 Astra) spent significantly more of their "turn budget" on geometric reasoning and triangulation compared to weaker models, which tended to just "point and click" without sufficient multi-view evidence.
  • The Cost of Intelligence: There is a clear Pareto frontier between cost and accuracy. The most accurate models are also the most expensive to run, yet even they are far from human-level performance.
Cost vs. Accuracy for various VLM agents

Key Results

The researchers evaluated seven frontier VLMs, and the results were humbling:

  • Claude Opus 5.5 took the top spot with a 22.52% mean IoU.
  • GPT-6 Astra followed with 15.50% (all-target). Interestingly, Astra had decent accuracy on the objects it did submit, but it suffered from many "omissions"—simply failing to find or map all the targets in the inventory.
  • The Long Tail of Failure: Six out of the seven models tested scored below 20% mean IoU.
  • Qualitative Failures: As seen in Figure 5, models often struggle with depth. A model might correctly identify a refrigerator but place its 3D center several feet inside the wall or floating in the middle of the room.
Qualitative examples of 3D localization successes and failures

Limitations and Future Work

The authors acknowledge several constraints that define the current state of the art:

  • Discrete Movement: Agents move between fixed "scan positions" rather than having continuous 3D movement.
  • Tool Dependency: The agents rely on a provided triangulation tool. Future work could explore if agents can learn to perform "end-to-end" geometric reasoning without such crutches.
  • Static Scenes: The environments are static captures. Real-world spatial intelligence eventually needs to handle dynamic environments where objects move or change.

SpatialScout-V2 serves as a challenging new North Star for the community, proving that while our models can talk the talk, they still have a long way to go before they can truly "see" and map the 3D world we live in.

Summary updated

Abstract

Spatial intelligence requires agents to acquire visual evidence and integrate partial observations into coherent three-dimensional representations. We introduce SpatialScout-V2, which extends SpatialScout's active mapping task from simulation to real residential scenes through three advances: real visual observations, preservation of original scene content, and interactive exploration of recorded views. Built from 100 homes in Realsee3D, the benchmark covers 3,171 target objects across 29 categories. Given a fixed target inventory, agents autonomously select recorded viewpoints and viewing directions, integrate calibrated RGB observations, and iteratively estimate and revise object centers within a fixed interaction budget. We evaluate the localization accuracy and completeness of object-center maps using mean IoU, IoU-threshold hit rates, and target coverage. Evaluating seven frontier vision-language models (VLMs), we find that six achieve an all-target mean IoU below 20\%. Even the best-performing model, Claude Opus 5.5, reaches only 22.52\%, with just 18.36\% of targets exceeding an IoU of 0.5. Further analysis shows that broad submission coverage can coexist with limited localization accuracy, while target omissions further reduce overall scores. These results highlight substantial challenges in constructing accurate and complete object-center maps through active exploration of real scenes.
BibTeX
@misc{zhang2026spatialscout,
  author = {Zhaoxi Zhang and Hao Liang and Linxuan Wu and Yiming Xu and Yifu Yuan and Kun Liang and Chenming Tang and Meiyi Qiang and Wentao Zhang and Weikang Li and Yunfang Wu},
  title = {{SpatialScout-V2: Can VLM Agents Actively Explore Real-World 3D Scenes and Localize Objects?}},
  year = {2026},
  eprint = {2611.26177},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26177},
}
Figure