SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models

Wufei Ma, Luoxin Ye, Nessa McWeeney, Celso M. de Melo, Jieneng Chen, Alan L. Yuille

Introduction

Current Large Multimodal Models (LMMs), such as GPT-4o and LLaVA, demonstrate impressive capabilities in semantic scene description but often lack a fundamental "sense of depth." These models frequently struggle with basic 3D spatial reasoning tasks, such as determining relative distances between objects or identifying their orientations. This "3D blindness" stems from two main factors: a scarcity of training data containing explicit 3D information (depth, 6D pose, etc.) and a 2D bias in model architectures optimized primarily for recognition.

The SpatialLLM Framework

To bridge this gap, SpatialLLM introduces a Compound 3D-Informed Design that integrates spatial awareness across architecture, data, and training strategies.

Overview of the SpatialLLM framework, illustrating 3D spatial reasoning examples and the 3D-informed design components.

1. Mixed Visual Encoders

SpatialLLM moves beyond standard CLIP encoders by adopting a mixed approach. It combines CLIP's semantic strengths with DINOv2, a self-supervised model that excels at capturing fine-grained structural and geometric details necessary for spatial reasoning.

2. 3D-Informed Data Engine

The researchers curated a massive dataset to ground the model in 3D space:

  • 3D Probing Data (3DI-Pb): Focuses on fundamental geometric properties like 3D location, depth, and orientation.
  • 3D Conversation Data (3DI-Ft): Comprises 1 million multi-turn dialogues regarding complex spatial relationships, synthesized from human-annotated 3D datasets like ImageNet3D and Omni3D.

3. Multi-Stage Training Roadmap

Rather than treating 3D awareness as a post-processing step, SpatialLLM injects it throughout the training pipeline:

  • Stage 0: Vision encoder tuning with probing data.
  • Stage 1: Multimodal alignment to link visual features with 3D properties.
  • Stage 2: Final instruction tuning using complex 3D-informed conversations.

Benchmarking Spatial Intelligence

The model was evaluated on SpatialVQA, a new benchmark of 1,323 questions designed to test distance, orientation, and complex interactions in real-world scenes. SpatialLLM achieved an overall accuracy of 62.7%, significantly outperforming GPT-4o (54.0%) and previous leaders like SpatialVLM (52.2%). Notably, it reached 86.3% accuracy in distance reasoning and established a new state-of-the-art for orientation tasks on real images.

Key Insights

  • Orientation is Critical: Understanding which way an object faces is essential for reasoning about interactions and potential collisions, a dimension often ignored by previous depth-only models.
  • Data-Centric Gains: While architectural changes provided a 1.3% boost, the systematic injection of 3D-informed data yielded a substantial 13.7% improvement.
  • Early Alignment Matters: Aligning visual features with 3D properties early in the training process (Stage 1) is more effective than relying solely on final instruction tuning.

Limitations and Future Directions

Despite its success, SpatialLLM relies partly on pseudo-labels from depth estimators, which can introduce noise. Future work may involve incorporating explicit 3D inputs like point clouds or regional depth maps directly into the inference process to further enhance real-world performance in fields like robotics and autonomous driving.

Summary updated

BibTeX
@inproceedings{ma2025spatialllm,
  author = {Wufei Ma and Luoxin Ye and Nessa McWeeney and Celso M. de Melo and Jieneng Chen and Alan L. Yuille},
  title = {{SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=XAYKPOmBcf},
}
Figure