Representing the Unseen: Rethinking Visual Imagination for Spatial Reasoning

Zengqi Zhao, Zun Wang, Shoubin Yu, Mohit Bansal, Yue ZhangarXiv 2026 · arXiv

TL;DR

This paper challenges the assumption that RGB images are the best way for AI to "imagine" unseen spaces. By introducing a new benchmark, IMAGIREP, the authors demonstrate that alternative representations like top-down maps and depth views often lead to much better spatial reasoning than standard photos, and they propose GEARS, a strategy that adaptively selects the most useful representation for a given question.

What Problem Does This Paper Address?

When we ask an AI to reason about something it can't see—like "What's around the corner?"—modern systems typically use a "visual world model" to generate a standard RGB image of that unseen area. However, the authors argue that RGB is a default choice because it's easy to generate, not because it's the best for reasoning.

For example, if you need to know which of two objects is farther away, a standard photo might be ambiguous due to lighting or perspective, whereas a depth map or a top-down floor plan provides explicit geometric data. The core challenge is identifying which representations actually help models reason and how to choose the right one at the right time.

Figure 1: Spatial imagination is more than generating future RGB views. The same imagined state may support different spatial questions (e.g., top-down geometry helps global distance comparisons; panoramic views facilitate reasoning about open space), yet existing approaches typically expose it to the reasoner only through RGB. IMAGIREP provides controlled access to the same underlying state through multiple representations, allowing us to study how the choice of representation affects downstream spatial reasoning.

Methods

The researchers took a two-pronged approach:

  1. Systematic Analysis: They used the IMAGIREP benchmark to compare how "oracle" (perfect, ground-truth) versions of seven different representations—RGB, Depth, Surface Normals, Panorama, Top-down Geometry, Spatial Text, and Video—affected the performance of state-of-the-art models like GPT-6.1-sol and Gemini 3.7 Flash.
  2. GEARS (Gated Evidence and Adaptive Representation Selection): They developed a lightweight test-time strategy to improve reasoning without needing perfect ground truth. GEARS works in two stages:
  3. Gating: It compares the model's answer based only on what it can see versus what it imagines in RGB. If the answers disagree, it signals that the imagination might be necessary but potentially misleading.
  4. Routing: If the gate is triggered, the system adaptively selects the best alternative representation (e.g., a panorama for "open space" questions) to provide the final answer.
The GEARS framework uses a gating and routing mechanism to select the best evidence for reasoning.

Datasets

  • Matterport3D: The foundational dataset of 3D indoor environments used to build the benchmark.
  • IMAGIREP (New): A newly introduced simulator-grounded benchmark containing 922 human-annotated questions. It covers two main scenarios:
  • Viewpoint Imagination: Reasoning about a scene after a specific movement (e.g., "Walk past the sofa and turn left").
  • Cross-view Integration: Combining information from multiple partial views to understand a central, unobserved point.
Figure 2: Overview of IMAGIREP. The top left illustrates benchmark construction, including a 3D scene, movement instructions, and observed view. The top right shows oracle RGB and six alternative representations of the same imagined state. The lower panels present example questions and ground-truth answers for the Viewpoint Imagination and Cross-view Integration question types.

Key Insights and Ideas

  • The "RGB Trap": Surprisingly, providing a perfect RGB image of an unseen state can actually hurt performance. For "Farthest Object" questions, GPT-6.1-sol's accuracy dropped from 68.4% to 58.2% when given the ground-truth RGB view.
  • Representation Matters: Different spatial tasks have "favorite" representations. Top-down geometry is most effective for distance comparisons, while panoramic views are superior for reasoning about open space and visibility.
  • Complementary Benefits: No single representation is the best across the board. The "Oracle Selection" (picking the best representation for every single question) reaches ~87-91% accuracy, far higher than any single fixed representation.
  • Robustness to Errors: Even when the world model makes mistakes (e.g., a slightly distorted "estimated" depth map), these alternative views still provide more useful cues than a similarly flawed RGB image.

Key Results

  • GEARS Performance: The proposed GEARS strategy achieved 61.4% accuracy on Viewpoint Imagination and 77.7% on Cross-view Integration, outperforming existing "imagination" methods like MindJourney and AVIC.
  • Recovery Scores: On Viewpoint Imagination, using a panoramic view instead of standard RGB recovered enough errors to jump from 66.1% to 79.3% accuracy (using oracle data).
  • Estimated vs. Oracle: While there is still a significant gap between estimated representations and "oracle" ground truth, the study proves that moving beyond RGB is a viable path to narrowing that gap.

Limitations and Future Work

  • Static Environments: The current benchmark is limited to static indoor scenes. Future work is needed to address dynamic environments with moving objects or people.
  • World Model Quality: The system's performance is still tethered to the quality of the underlying world models (like HY-World or SEVA). Errors in 3D reconstruction can still lead to reasoning failures.
  • Scope of Tasks: The authors suggest extending this framework to more complex interactions and a wider variety of outdoor or industrial environments.

Summary updated

Abstract

Visual world models increasingly support spatial reasoning by generating RGB views of unobserved states. However, RGB has become the default interface largely because it is well suited for visual generation, rather than necessarily being the most effective representation for downstream reasoning. Different spatial questions may benefit from different representations of the same imagined state: top-down geometry can facilitate global distance comparisons, while panoramic views provide broader context for reasoning about open space. To isolate the role of representation in spatial imagination, we introduce ImagiRep, a simulator-grounded benchmark of 922 questions spanning seven question types across two settings, Viewpoint Imagination and Cross-view Integration, providing oracle RGB and six alternative representations (e.g., depth, panoramic views, and top-down geometry) of the same unobserved state. This controlled comparison reveals that even accurate imagined RGB does not guarantee accurate spatial reasoning, and different questions benefit from different representations. Similar trends hold for representations derived from world-model-estimated states. Notably, estimated RGB alone can mislead spatial reasoning, while alternative representations retain complementary benefits. Building on these findings, we introduce GEARS, a lightweight test-time strategy for adaptive representation selection, outperforming existing spatial imagination methods and showing that reasoning about unseen states requires more than generating RGB views.
BibTeX
@misc{zhao2026representing,
  author = {Zengqi Zhao and Zun Wang and Shoubin Yu and Mohit Bansal and Yue Zhang},
  title = {{Representing the Unseen: Rethinking Visual Imagination for Spatial Reasoning}},
  year = {2026},
  eprint = {2610.37437},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37437},
}
Figure