SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models

Wufei Ma, Luoxin Ye, Celso M. de Melo, Alan L. Yuille, Jieneng Chen

Introduction

Current Large Multimodal Models (LMMs), such as GPT-4o and LLaVA, demonstrate impressive capabilities in semantic scene description but often lack a fundamental "sense of depth." These models frequently struggle with basic 3D spatial reasoning tasks, such as determining relative distances between objects or identifying their orientations. This "3D blindness" stems from two main factors: a scarcity of training data containing explicit 3D information (depth, 6D pose, etc.) and a 2D bias in model architectures optimized primarily for recognition.

The SpatialLLM Framework

To bridge this gap, SpatialLLM introduces a Compound 3D-Informed Design that integrates spatial awareness across architecture, data, and training strategies.

Overview of the SpatialLLM framework, illustrating 3D spatial reasoning examples and the 3D-informed design components.

1. Mixed Visual Encoders

SpatialLLM moves beyond standard CLIP encoders by adopting a mixed approach. It combines CLIP's semantic strengths with DINOv2, a self-supervised model that excels at capturing fine-grained structural and geometric details necessary for spatial reasoning.

2. 3D-Informed Data Engine

The researchers curated a massive dataset to ground the model in 3D space:

  • 3D Probing Data (3DI-Pb): Focuses on fundamental geometric properties like 3D location, depth, and orientation.
  • 3D Conversation Data (3DI-Ft): Comprises 1 million multi-turn dialogues regarding complex spatial relationships, synthesized from human-annotated 3D datasets like ImageNet3D and Omni3D.

3. Multi-Stage Training Roadmap

Rather than treating 3D awareness as a post-processing step, SpatialLLM injects it throughout the training pipeline:

  • Stage 0: Vision encoder tuning with probing data.
  • Stage 1: Multimodal alignment to link visual features with 3D properties.
  • Stage 2: Final instruction tuning using complex 3D-informed conversations.

Benchmarking Spatial Intelligence

The model was evaluated on SpatialVQA, a new benchmark of 1,323 questions designed to test distance, orientation, and complex interactions in real-world scenes. SpatialLLM achieved an overall accuracy of 62.7%, significantly outperforming GPT-4o (54.0%) and previous leaders like SpatialVLM (52.2%). Notably, it reached 86.3% accuracy in distance reasoning and established a new state-of-the-art for orientation tasks on real images.

Key Insights

  • Orientation is Critical: Understanding which way an object faces is essential for reasoning about interactions and potential collisions, a dimension often ignored by previous depth-only models.
  • Data-Centric Gains: While architectural changes provided a 1.3% boost, the systematic injection of 3D-informed data yielded a substantial 13.7% improvement.
  • Early Alignment Matters: Aligning visual features with 3D properties early in the training process (Stage 1) is more effective than relying solely on final instruction tuning.

Limitations and Future Directions

Despite its success, SpatialLLM relies partly on pseudo-labels from depth estimators, which can introduce noise. Future work may involve incorporating explicit 3D inputs like point clouds or regional depth maps directly into the inference process to further enhance real-world performance in fields like robotics and autonomous driving.

Summary updated

Abstract

Humans naturally understand 3D spatial relationships, enabling complex reasoning like predicting collisions of vehicles from different directions. Current large multimodal models (LMMs), however, lack of this capability of 3D spatial reasoning. This limitation stems from the scarcity of 3D training data and the bias in current model designs toward 2D data. In this paper, we systematically study the impact of 3D-informed data, architecture, and training setups, introducing SpatialLLM, a large multi-modal model with advanced 3D spatial reasoning abilities. To address data limitations, we develop two types of 3D-informed training datasets: (1) 3D-informed probing data focused on object's 3D location and orientation, and (2) 3D-informed conversation data for complex spatial relationships. Notably, we are the first to curate VQA data that incorporate 3D orientation relationships on real images. Furthermore, we systematically integrate these two types of training data with the architectural and training designs of LMMs, providing a roadmap for optimal design aimed at achieving superior 3D reasoning capabilities. Our SpatialLLM advances machines toward highly capable 3D-informed reasoning, surpassing GPT-4o performance by 8.7%. Our systematic empirical design and the resulting findings offer valuable insights for future research in this direction.
BibTeX
@inproceedings{ma2025spatialllm,
  author = {Wufei Ma and Luoxin Ye and Celso M. de Melo and Alan L. Yuille and Jieneng Chen},
  title = {{SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=XAYKPOmBcf},
}
Figure