SpatialGeo:Boosting Spatial Reasoning in Multimodal LLMs via Geometry-Semantics Fusion

Jiajie Guo, Qingpeng Zhu, Jin Zeng, Xiaolong Wu, Changyong He, Weida Wang2025 · arXiv

TL;DR

This paper introduces SpatialGeo, a novel multimodal large language model (MLLM) that significantly improves spatial reasoning capabilities by fusing geometric and semantic visual features. By addressing the spatial ambiguity inherent in current vision encoders like CLIP, SpatialGeo achieves state-of-the-art performance on spatial VQA tasks with improved efficiency.

What Problem Does This Paper Address?

Large language models (LLMs) have shown remarkable progress in language-based tasks, but extending them to real-world applications like robotics requires them to understand and reason about the physical world. This necessitates multimodal large language models (MLLMs) that can ground language understanding in visual input, particularly by comprehending spatial relationships between objects. Existing MLLMs, even advanced ones like LLaVA, struggle with spatial reasoning. The paper identifies a key issue: the vision encoders used (e.g., CLIP) encode images taken at different focal lengths of the same scene into highly similar features, leading to a lossy spatial information encoding. This "spatial ambiguity" limits the MLLM's ability to accurately infer object sizes, relative positions, and other spatial attributes, hindering their application in critical areas like robotic navigation and visual planning.

Methods

SpatialGeo tackles the spatial ambiguity problem by introducing a novel vision encoder that fuses geometry and semantics. The core idea is to complement the semantic features from a standard vision encoder (like CLIP) with geometry-aware features.

  • Geometry-Aware Vision Encoder (MoGe): The paper leverages MoGe, a vision-only self-supervised model designed for precise 3D geometry reconstruction from monocular images. MoGe is shown to capture distinct spatial information, unlike CLIP.
  • Hierarchical Adapter for Feature Fusion: To effectively integrate MoGe's geometry features with CLIP's semantic features, a hierarchical adapter is designed. This adapter progressively fuses features from multiple layers of the MoGe encoder. The rationale is that shallower MoGe layers preserve fine-grained spatial details, while deeper layers capture more abstract semantics. This hierarchical fusion creates a richer, spatially aware visual embedding.
  • Interleaved Feature Fusion: The fused geometry-semantic features are then interleaved with CLIP features, maintaining their spatial order, before being fed into the LLM.
  • Efficient Training Strategy: To promote training and data efficiency, SpatialGeo is initialized with a pre-trained LLaVA model. The training involves a two-stage process:
  • Pre-training: The hierarchical adapter is trained while CLIP and its adapter are frozen, focusing on aligning MoGe features.
  • Instruction Fine-tuning: Adapters for CLIP and MoGe, along with the LLM, are fine-tuned using instruction data and spatial VQA datasets. Crucially, a random feature dropping strategy is employed, where CLIP embeddings are masked with a certain probability. This forces the LLM to rely more on the geometry-rich MoGe features, preventing trivial solutions that depend solely on CLIP.

Datasets

  • Training:
    • LLaVA-Pretrain dataset (for the first stage of training).
  • COCO and part of VisualGenome (for the second stage of training).
  • Open Spatial Dataset (OSD) [14]: A subset of 160k images from OSD was used for training efficiency. Bounding box annotations were adjusted for relative positions.
  • Evaluation:
    • SpatialRGPT-Bench [14]: Used as the primary testing dataset for spatial VQA.
  • SpatialScore [20]: Used to evaluate generalization to unseen spatial reasoning benchmarks.
  • General VQA Benchmarks: POPE, MM-Vet, MME, MMVP, and BLINK (Relative Depth) were used to assess performance on non-spatial tasks and ensure no sacrifice in general VQA capabilities.

Key Insights and Ideas

  • Spatial Ambiguity in CLIP is a Root Cause: The paper provides compelling evidence that the lossy spatial encoding in CLIP's embedding space is a primary reason for MLLMs' poor spatial reasoning. This is demonstrated by comparing CLIP and MoGe embeddings for images with varying focal lengths.
  • Geometry is Crucial for Spatial Grounding: Integrating geometry-aware features (from MoGe) significantly enhances spatial grounding, allowing models to infer precise spatial attributes that semantic features alone miss.
  • Hierarchical Fusion Captures Nuance: The hierarchical adapter effectively fuses geometry and semantic features by leveraging different levels of abstraction from the MoGe encoder, leading to a more comprehensive visual representation.
  • Random Feature Dropping is Key for Training: This technique is vital for ensuring that the LLM learns to effectively utilize the newly integrated geometry features, rather than defaulting to the familiar semantic features from CLIP.

Key Results

  • State-of-the-Art Spatial Reasoning: SpatialGeo achieves superior performance on spatial VQA tasks, outperforming SOTA models like LLaVA-v1.5, GPT-4.1, and SpatialRGPT.
  • Significant Accuracy Improvement: On SpatialRGPT-Bench, SpatialGeo enhances accuracy by at least 8.0% compared to the second-best model, despite using only a fraction of the training data compared to SpatialRGPT.
  • Improved Efficiency: SpatialGeo achieves approximately 50% less memory cost during inference compared to SpatialRGPT, highlighting its computational efficiency.
  • Strong Generalization: The model demonstrates strong generalization capabilities on the SpatialScore benchmark, showing significant improvements over LLaVA-1.5, particularly on QSpatial-PluS (51% improvement).
  • No Sacrifice in General VQA: Evaluation on general VQA benchmarks shows that SpatialGeo performs comparably to the LLaVA-1.5 baseline, indicating that its spatial reasoning improvements do not come at the cost of general visual-language understanding.

Limitations and Future Work

  • Limitations: The paper acknowledges that while SpatialGeo significantly improves spatial reasoning, it still faces challenges in absolute spatial metrics, even for larger models. The ablation study also suggests that while a single adapter can leverage geometry features, the hierarchical approach is superior on average.
  • Future Work: The authors suggest that their work provides a strong foundation for future research in enhancing spatial reasoning in MLLMs. The proposed architecture and training strategy can be extended to other MLLM frameworks and applied to more complex 3D understanding tasks. The paper also hints at the potential for further improvements by exploring different fusion strategies or more advanced geometry-aware encoders.

Summary updated

Abstract

Multimodal large language models (MLLMs) have achieved significant progress in image and language tasks due to the strong reasoning capability of large language models (LLMs). Nevertheless, most MLLMs suffer from limited spatial reasoning ability to interpret and infer spatial arrangements in three-dimensional space. In this work, we propose a novel vision encoder based on hierarchical fusion of geometry and semantics features, generating spatial-aware visual embedding and boosting the spatial grounding capability of MLLMs. Specifically, we first unveil that the spatial ambiguity shortcoming stems from the lossy embedding of the vision encoder utilized in most existing MLLMs (e.g., CLIP), restricted to instance-level semantic features. This motivates us to complement CLIP with the geometry features from vision-only self-supervised learning via a hierarchical adapter, enhancing the spatial awareness in the proposed SpatialGeo. The network is efficiently trained using pretrained LLaVA model and optimized with random feature dropping to avoid trivial solutions relying solely on the CLIP encoder. Experimental results show that SpatialGeo improves the accuracy in spatial reasoning tasks, enhancing state-of-the-art models by at least 8.0% in SpatialRGPT-Bench with ∼50% less memory cost during inference. The source code is available via https://ricky-plus.github.io/SpatialGeoPages/.
BibTeX
@misc{guo2025spatialgeo,
  author = {Jiajie Guo and Qingpeng Zhu and Jin Zeng and Xiaolong Wu and Changyong He and Weida Wang},
  title = {{SpatialGeo:Boosting Spatial Reasoning in Multimodal LLMs via Geometry-Semantics Fusion}},
  year = {2025},
  eprint = {2511.17308},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.17308},
}
Figure