TL;DR
ESI-BENCH introduces a comprehensive benchmark that shifts the evaluation of spatial intelligence from passive image analysis to active, embodied exploration. By testing state-of-the-art Multimodal Large Language Models (MLLMs) in a simulated environment, the paper reveals that while active exploration dramatically improves performance, current models suffer from "action blindness" and a severe lack of human-like epistemic caution.
What Problem Does This Paper Address?
Most current benchmarks evaluate spatial intelligence passively: models look at a single image or a fixed video and answer questions. However, true spatial intelligence is inherently active. In the real world, critical spatial information is often hidden behind occlusions, tucked inside containers, or distorted by perspective.
To resolve these ambiguities, an intelligent agent cannot just sit back and watch—it must act. It needs to move around an object, lift a cloth, or pour liquid to understand volume. This paper addresses the gap between passive perception and active, embodied spatial competence by forcing models to operate within a tight perception-action loop.
Methods
The authors recast the spatial observer as an active actor. They equip MLLMs with a high-level discrete action space that includes:
- Locomotion: Moving forward, backward, laterally, or vertically.
- Perception: Rotating the camera viewpoint horizontally or vertically.
- Manipulation: Interacting with objects via commands like
pick_up,put_inside,fill_with_water, andpour.
To understand what holds these models back, the authors evaluate them across four distinct paradigms:
- Passive Single-View: A baseline using only the initial fixed viewpoint.
- Passive Random Multi-View: Providing 30 randomly sampled viewpoints to simulate exhaustive, unguided coverage.
- Active Exploration: Letting the model actively choose its next action step-by-step up to a 30-step budget.
- Ground-Truth Passive (Oracle): Feeding the model the exact sequence of views generated by an optimal, human-mapped trajectory.
Additionally, they explore the impact of 3D representations by comparing standard 2D MLLMs against models augmented with reconstructed 3D scene graphs (VGGT) and perfect ground-truth 3D point clouds.
Datasets
The authors introduce ESI-BENCH, a benchmark consisting of 3,081 task instances across 10 task categories and 29 subcategories.
Built on top of the BEHAVIOR-1K dataset within the OmniGibson simulator, ESI-BENCH leverages advanced rigid-body contact physics, particle-based fluids, and realistic transparency/reflection rendering. The tasks are theoretically grounded in cognitive scientist Elizabeth Spelke’s four core knowledge systems: object representation, layout and geometry, number representation, and agents/goal-directed actions.
Key Insights and Ideas
- Emergent Strategies: Without explicit instructions, active MLLMs spontaneously develop clever, multi-step strategies—such as moving behind an object to check for hidden items or changing to a top-down view to compare sizes.
- Action Blindness > Perceptual Blindness: For most tasks, the bottleneck isn't that the model can't see or process the images; it's that it doesn't know where to look. When given the oracle viewpoint, performance skyrockets. Suboptimal action choices lead to uninformative views, causing a cascading chain of errors.
- More Data Can Be Noise: Interestingly, providing passive random multi-views often degrades model performance compared to a single view. Without selective action, unstructured data accumulation simply introduces distraction and cognitive overload.
- The Metacognitive Gap: Models completely lack "epistemic calibration." While humans actively seek out viewpoints that might falsify their current hypothesis and express lower confidence under ambiguity, MLLMs routinely commit prematurely with uniformly high confidence, ignoring contradictory evidence.
Key Results
- Active Exploration Boosts Performance: Active exploration yields massive gains over passive baselines. For instance, on the View Hallucination task, Gemini 3.1's accuracy jumped from 39.9% (single-view) to 68.1% (active).
- The Oracle Gap: On Rigid Containment, GPT-5 scored 42.5% under active exploration but achieved 95.0% when provided with the ground-truth passive trajectory, underscoring the severe cost of poor action selection.
- Imperfect 3D Hurts: While perfect 3D data (Ground-Truth 3D + Gemini) drastically improved geometric reasoning, noisy 3D reconstructions (VGGT + Gemini) actually performed worse than standard 2D baselines because geometric artifacts distorted fine-grained spatial relations.
- Humans Still Dominate: Humans dramatically outperform active MLLMs (e.g., 88.3% vs 64.2% for GPT-5 on Physical Contact) because humans naturally gather more diverse viewpoints and systematically update their beliefs when faced with counter-evidence.
Limitations and Future Work
The authors note two primary limitations of ESI-BENCH:
- Simulation Realism: While OmniGibson features advanced physics and rendering, its visual textures, lighting, and lack of motion blur still differ noticeably from real-world camera data.
- High-Level Abstraction: The benchmark intentionally factors out low-level motor control challenges (like grasp stability, locomotion drift, and collision recovery) to isolate high-level spatial reasoning.
Future work will need to address how to bridge this simulation-to-real world gap and explore how to build models capable of true human-like belief revision and uncertainty awareness.
Summary updated