AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Models

Shouwei Ruan, Bin Wang, Zhenyu Wu, Qihui Zhu, Yuxiang Zhang, Jingzhi Li, Yubin Wang, Xingxing WeiarXiv 2026 · arXiv

TL;DR

Current Multimodal Foundation Models (MFMs) struggle with 3D spatial reasoning because they rely on local, transient, egocentric views. AlloSpatial addresses this bottleneck by introducing an agentic framework that transforms egocentric observations into persistent, global "allocentric" cognitive maps. By coupling these structured representations with a verifiable reasoning harness and reinforcement learning, AlloSpatial allows models to perform robust spatial mental simulations, significantly outperforming much larger general-purpose models.

What Problem Does This Paper Address?

When humans navigate a room, we don't just look at what's directly in front of us; we build a persistent mental map of our environment. In contrast, modern MFMs largely operate on egocentric observations—processing video frames or images in isolation without a unified sense of 3D space.

Existing attempts to fix this usually fall into three categories:

  1. Vision-centric post-training: Teaching models to guess depth or relations directly from 2D pixels, which easily breaks when the room layout changes.
  2. Geometry-centric injection: Feeding raw 3D point clouds or depth maps directly into the model, which requires expensive paired data and complex cross-modal alignment.
  3. Tool-augmented reasoning: Letting models call 3D reconstruction tools, which usually return low-level geometric data that the model struggles to assemble over long, error-prone reasoning chains.

AlloSpatial bridges this gap by providing a framework that actively builds, queries, and cross-validates structured, object-centric allocentric maps.

Methods

The AlloSpatial framework consists of two primary components operating in an agentic loop:

Figure 1: AlloSpatial inference and training pipeline. A, At inference time, AlloSpatial takes egocentric videos or multi-view observations and follows a three-stage Spatial Reasoning Harness to invoke World2Mind, acquire allocentric spatial knowledge, and arbitrate evidence before answering. B, To internalize this harness, we distill and filter high-quality harness-following trajectories from proprietary models for supervised cold start, and further optimize the policy with live World2Mind interaction and a Harness-Gated Trajectory Reward.

1. World2Mind Sandbox

This plug-and-play cognitive mapping sandbox converts egocentric video frames or multi-view images into queryable allocentric spatial priors. It uses monocular depth estimation and open-vocabulary segmentation (SAM 3) to construct a sparse semantic point cloud. This point cloud is distilled into two maps:

  • Landmark Cognitive Map: Structured as an Allocentric-Spatial Tree (AST). This directed acyclic graph represents stable environmental landmarks as nodes, encoding their 3D centroids, footprints, orientations, and hierarchical containment relations in a clean YAML format.
  • Route Cognitive Map: A top-down grid map tracking traversability, passability, and camera trajectories.

2. Spatial Reasoning Harness

To prevent the model from blindly trusting noisy reconstructions or getting stuck in "modality lock-in," the framework enforces a three-stage cyclic protocol:

  • Stage I: Tool Invocation Judgment: The model decides if the spatial question requires metric or topological mapping.
  • Stage II: Modality-Decoupled Cue Collection: The model gathers visual clues, AST text data, and top-down rendered maps independently to prevent premature fusion.
  • Stage III: Geometry-Semantics Interwoven Arbitration: The model actively cross-validates the modalities, identifying conflicts (e.g., reconstruction drift or false detections) before outputting a final answer.

Internalization via Reinforcement Learning

To embed this behavior into open-weight models (like Qwen3-VL), the authors use a two-stage training process. First, they bootstrap the model using Supervised Cold Start on high-quality trajectories distilled from proprietary models. Second, they optimize the policy using Group Sequence Policy Optimization (GSPO) combined with a novel Harness-Gated Trajectory Reward (HGTR). HGTR ensures that the model only receives accuracy rewards if it strictly follows the required reasoning structure and uses tools efficiently.

Datasets

The framework was trained and evaluated using two primary spatial benchmarks:

  • VSI-Bench (and VSI-590K): An egocentric video-based spatial intelligence benchmark testing object counting, absolute distance, room size, and route planning.
  • MindCube (and MindCube-10K): A sparse multi-view benchmark focusing on cognitive mapping, spatial relations ("around", "among"), and viewpoint-dependent rotations.

Key Insights and Ideas

  • Mental Simulation Without Vision: One of the most striking findings is that the text-serialized AST representation is so rich that models can perform accurate 3D spatial reasoning in a completely "blind" setting (where visual inputs are removed entirely).
  • The Power of Active Arbitration: Rather than treating tool outputs as ground truth, treating them as falsifiable evidence that must be cross-validated against visual cues drastically reduces errors caused by noisy 3D reconstructions.
  • Structure-Gated Rewards Prevent Hacking: Standard reinforcement learning on long tool-use trajectories often leads to reward hacking (e.g., making redundant tool calls or guessing correctly via malformed steps). Gating rewards based on harness compliance forces the model to be deliberate and efficient.

Key Results

  • Boosting Proprietary Models: As a training-free plug-in, AlloSpatial improved frontier commercial models (GPT-5.2, Claude-4.6-Opus, Gemini-3-Pro) by 5% to 18% across VSI-Bench and MindCube.
  • Outperforming Larger Models: After RL training, the open-weight AlloSpatial-4B and 8B agents outperformed much larger general-purpose models like Qwen3-VL-32B and specialized spatial baselines (such as Spatial-MLLM and Cambrian-S).
  • Robustness to Sparse Inputs: AlloSpatial shows its strongest advantages when visual observations are highly limited (e.g., only 3 or 7 frames), proving that allocentric memory effectively compensates for sparse visual coverage.
Figure 2: VSI-Bench performance un- der different input-frame number.

Limitations and Future Work

While AlloSpatial excels at relational and topological tasks (like route planning and relative directions), its performance on fine-grained numerical estimation (like exact absolute distances) remains limited. This is primarily due to downstream errors in the underlying monocular depth estimation and reconstruction drift within the World2Mind sandbox.

The authors note that future work will focus on integrating stronger metric calibration techniques and uncertainty-aware reconstructions directly into the cognitive mapping backend.

Summary updated

Abstract

Multimodal Foundation Models (MFMs) have made substantial progress, yet remain fragile in spatial reasoning over the physical world. A key bottleneck lies in their inability to transform local egocentric observations into a global allocentric spatial representation. To address this, we propose AlloSpatial, an agentic framework for allocentric spatial cognition in foundation models. AlloSpatial introduces World2Mind, a plug-and-play cognitive mapping sandbox that converts egocentric observations into structured allocentric priors, including Allocentric-Spatial Trees and route maps that support querying object topology, geometric relations, passability, and trajectories. To utilize these priors reliably under noisy reconstruction and ambiguous visual evidence, AlloSpatial introduces a Spatial Reasoning Harness for tool-use judgment, modality-decoupled cue collection, and geometry-semantic arbitration. We further internalize this process in Qwen3-VL through cold-start reinforcement learning with a harness-gated trajectory-level reward. Experiments on VSI-Bench and MindCube show that AlloSpatial improves proprietary models by 5%-18% in a training-free setting, while ASTs alone support strong spatial reasoning even when visual inputs are removed. The trained AlloSpatial agents further outperform larger general-purpose models and competitive spatial baselines, suggesting that structured allocentric representations, active tool use, and verifiable reasoning offer a promising route toward spatially capable foundation models.
BibTeX
@misc{ruan2026allospatial,
  author = {Shouwei Ruan and Bin Wang and Zhenyu Wu and Qihui Zhu and Yuxiang Zhang and Jingzhi Li and Yubin Wang and Xingxing Wei},
  title = {{AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Models}},
  year = {2026},
  eprint = {2606.08952},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.08952},
}

Loading related content...

Figure