ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop

Yining Hong, Jiageng Liu, Han Yin, Manling Li, Leonidas Guibas, Li Fei-Fei, Jiajun Wu, Yejin ChoiarXiv 2026 · arXiv

TL;DR

ESI-BENCH introduces a comprehensive benchmark that shifts the evaluation of spatial intelligence from passive image analysis to active, embodied exploration. By testing state-of-the-art Multimodal Large Language Models (MLLMs) in a simulated environment, the paper reveals that while active exploration dramatically improves performance, current models suffer from "action blindness" and a severe lack of human-like epistemic caution.

What Problem Does This Paper Address?

Most current benchmarks evaluate spatial intelligence passively: models look at a single image or a fixed video and answer questions. However, true spatial intelligence is inherently active. In the real world, critical spatial information is often hidden behind occlusions, tucked inside containers, or distorted by perspective.

To resolve these ambiguities, an intelligent agent cannot just sit back and watch—it must act. It needs to move around an object, lift a cloth, or pour liquid to understand volume. This paper addresses the gap between passive perception and active, embodied spatial competence by forcing models to operate within a tight perception-action loop.

ESI-BENCH is a comprehensive benchmark spanning 10 task categories and 29 subcategories organized around Spelke's core knowledge systems.

Methods

The authors recast the spatial observer as an active actor. They equip MLLMs with a high-level discrete action space that includes:

  • Locomotion: Moving forward, backward, laterally, or vertically.
  • Perception: Rotating the camera viewpoint horizontally or vertically.
  • Manipulation: Interacting with objects via commands like pick_up, put_inside, fill_with_water, and pour.

To understand what holds these models back, the authors evaluate them across four distinct paradigms:

  1. Passive Single-View: A baseline using only the initial fixed viewpoint.
  2. Passive Random Multi-View: Providing 30 randomly sampled viewpoints to simulate exhaustive, unguided coverage.
  3. Active Exploration: Letting the model actively choose its next action step-by-step up to a 30-step budget.
  4. Ground-Truth Passive (Oracle): Feeding the model the exact sequence of views generated by an optimal, human-mapped trajectory.

Additionally, they explore the impact of 3D representations by comparing standard 2D MLLMs against models augmented with reconstructed 3D scene graphs (VGGT) and perfect ground-truth 3D point clouds.

Datasets

The authors introduce ESI-BENCH, a benchmark consisting of 3,081 task instances across 10 task categories and 29 subcategories.

Built on top of the BEHAVIOR-1K dataset within the OmniGibson simulator, ESI-BENCH leverages advanced rigid-body contact physics, particle-based fluids, and realistic transparency/reflection rendering. The tasks are theoretically grounded in cognitive scientist Elizabeth Spelke’s four core knowledge systems: object representation, layout and geometry, number representation, and agents/goal-directed actions.

Key Insights and Ideas

  • Emergent Strategies: Without explicit instructions, active MLLMs spontaneously develop clever, multi-step strategies—such as moving behind an object to check for hidden items or changing to a top-down view to compare sizes.
  • Action Blindness > Perceptual Blindness: For most tasks, the bottleneck isn't that the model can't see or process the images; it's that it doesn't know where to look. When given the oracle viewpoint, performance skyrockets. Suboptimal action choices lead to uninformative views, causing a cascading chain of errors.
  • More Data Can Be Noise: Interestingly, providing passive random multi-views often degrades model performance compared to a single view. Without selective action, unstructured data accumulation simply introduces distraction and cognitive overload.
  • The Metacognitive Gap: Models completely lack "epistemic calibration." While humans actively seek out viewpoints that might falsify their current hypothesis and express lower confidence under ambiguity, MLLMs routinely commit prematurely with uniformly high confidence, ignoring contradictory evidence.
Qualitative examples highlighting success and failure modes, including cascading errors and the lack of epistemic calibration.

Key Results

  • Active Exploration Boosts Performance: Active exploration yields massive gains over passive baselines. For instance, on the View Hallucination task, Gemini 3.1's accuracy jumped from 39.9% (single-view) to 68.1% (active).
  • The Oracle Gap: On Rigid Containment, GPT-5 scored 42.5% under active exploration but achieved 95.0% when provided with the ground-truth passive trajectory, underscoring the severe cost of poor action selection.
  • Imperfect 3D Hurts: While perfect 3D data (Ground-Truth 3D + Gemini) drastically improved geometric reasoning, noisy 3D reconstructions (VGGT + Gemini) actually performed worse than standard 2D baselines because geometric artifacts distorted fine-grained spatial relations.
  • Humans Still Dominate: Humans dramatically outperform active MLLMs (e.g., 88.3% vs 64.2% for GPT-5 on Physical Contact) because humans naturally gather more diverse viewpoints and systematically update their beliefs when faced with counter-evidence.

Limitations and Future Work

The authors note two primary limitations of ESI-BENCH:

  1. Simulation Realism: While OmniGibson features advanced physics and rendering, its visual textures, lighting, and lack of motion blur still differ noticeably from real-world camera data.
  2. High-Level Abstraction: The benchmark intentionally factors out low-level motor control challenges (like grasp stability, locomotion drift, and collision recovery) to isolate high-level spatial reasoning.

Future work will need to address how to bridge this simulation-to-real world gap and explore how to build models capable of true human-like belief revision and uncertainty awareness.

Summary updated

Abstract

Spatial intelligence unfolds through a perception-action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke's core knowledge systems. Agents must decide what abilities to deploy - perception, locomotion, and manipulation - and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close.
BibTeX
@misc{hong2026esi,
  author = {Yining Hong and Jiageng Liu and Han Yin and Manling Li and Leonidas Guibas and Li Fei-Fei and Jiajun Wu and Yejin Choi},
  title = {{ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop}},
  year = {2026},
  eprint = {2605.18746},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.18746},
}

Loading related content...

Figure