TL;DR
This paper challenges the assumption that RGB images are the best way for AI to "imagine" unseen spaces. By introducing a new benchmark, IMAGIREP, the authors demonstrate that alternative representations like top-down maps and depth views often lead to much better spatial reasoning than standard photos, and they propose GEARS, a strategy that adaptively selects the most useful representation for a given question.
What Problem Does This Paper Address?
When we ask an AI to reason about something it can't see—like "What's around the corner?"—modern systems typically use a "visual world model" to generate a standard RGB image of that unseen area. However, the authors argue that RGB is a default choice because it's easy to generate, not because it's the best for reasoning.
For example, if you need to know which of two objects is farther away, a standard photo might be ambiguous due to lighting or perspective, whereas a depth map or a top-down floor plan provides explicit geometric data. The core challenge is identifying which representations actually help models reason and how to choose the right one at the right time.
Methods
The researchers took a two-pronged approach:
- Systematic Analysis: They used the IMAGIREP benchmark to compare how "oracle" (perfect, ground-truth) versions of seven different representations—RGB, Depth, Surface Normals, Panorama, Top-down Geometry, Spatial Text, and Video—affected the performance of state-of-the-art models like GPT-6.1-sol and Gemini 3.7 Flash.
- GEARS (Gated Evidence and Adaptive Representation Selection): They developed a lightweight test-time strategy to improve reasoning without needing perfect ground truth. GEARS works in two stages:
- Gating: It compares the model's answer based only on what it can see versus what it imagines in RGB. If the answers disagree, it signals that the imagination might be necessary but potentially misleading.
- Routing: If the gate is triggered, the system adaptively selects the best alternative representation (e.g., a panorama for "open space" questions) to provide the final answer.
Datasets
- Matterport3D: The foundational dataset of 3D indoor environments used to build the benchmark.
- IMAGIREP (New): A newly introduced simulator-grounded benchmark containing 922 human-annotated questions. It covers two main scenarios:
- Viewpoint Imagination: Reasoning about a scene after a specific movement (e.g., "Walk past the sofa and turn left").
- Cross-view Integration: Combining information from multiple partial views to understand a central, unobserved point.
Key Insights and Ideas
- The "RGB Trap": Surprisingly, providing a perfect RGB image of an unseen state can actually hurt performance. For "Farthest Object" questions, GPT-6.1-sol's accuracy dropped from 68.4% to 58.2% when given the ground-truth RGB view.
- Representation Matters: Different spatial tasks have "favorite" representations. Top-down geometry is most effective for distance comparisons, while panoramic views are superior for reasoning about open space and visibility.
- Complementary Benefits: No single representation is the best across the board. The "Oracle Selection" (picking the best representation for every single question) reaches ~87-91% accuracy, far higher than any single fixed representation.
- Robustness to Errors: Even when the world model makes mistakes (e.g., a slightly distorted "estimated" depth map), these alternative views still provide more useful cues than a similarly flawed RGB image.
Key Results
- GEARS Performance: The proposed GEARS strategy achieved 61.4% accuracy on Viewpoint Imagination and 77.7% on Cross-view Integration, outperforming existing "imagination" methods like MindJourney and AVIC.
- Recovery Scores: On Viewpoint Imagination, using a panoramic view instead of standard RGB recovered enough errors to jump from 66.1% to 79.3% accuracy (using oracle data).
- Estimated vs. Oracle: While there is still a significant gap between estimated representations and "oracle" ground truth, the study proves that moving beyond RGB is a viable path to narrowing that gap.
Limitations and Future Work
- Static Environments: The current benchmark is limited to static indoor scenes. Future work is needed to address dynamic environments with moving objects or people.
- World Model Quality: The system's performance is still tethered to the quality of the underlying world models (like HY-World or SEVA). Errors in 3D reconstruction can still lead to reasoning failures.
- Scope of Tasks: The authors suggest extending this framework to more complex interactions and a wider variety of outdoor or industrial environments.
Summary updated