LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations

Mingjie Xu, Mengyang Wu, Yuzhi Zhao, Jason Chun Lok Li, Weifeng Ou

Abstract

Scene Graph Generation (SGG) converts visual scenes into structured graph representations, providing deeper scene understanding for complex vision tasks. However, existing SGG models often overlook essential spatial relationships and struggle with generalization in open-vocabulary contexts. To address these limitations, we propose LLaVA-SpaceSGG, a multimodal large language model (MLLM) designed for open-vocabulary SGG with enhanced spatial relation modeling. To train it, we collect the SGG instruction-tuning dataset, named SpaceSGG. This dataset is constructed by combining publicly available datasets and synthesizing data using open-source models within our data construction pipeline. It combines object locations, object relations, and depth information, resulting in three data formats: spatial SGG description, question-answering, and conversation. To enhance the transfer of MLLMs' inherent capabilities to the SGG task, we introduce a two-stage training paradigm. Experiments show that LLaVA-SpaceSGG outperforms other open-vocabulary SGG methods, boosting recall by 8.6% and mean recall by 28.4% compared to the baseline.

Introduction

Scene Graph Generation (SGG) is a fundamental task in computer vision that transforms visual scenes into structured graphs of (subject, predicate, object) triplets. While critical for high-level reasoning, current SGG models face two primary challenges: open-vocabulary generalization and 3D spatial neglect. Most models are constrained by fixed training vocabularies and struggle with depth-based relations like "in front of" or "behind" because they rely on 2D image annotations.

Comparison of Object Detection, standard SGG, and SGG with enhanced spatial relations.

LLaVA-SpaceSGG addresses these by integrating depth information and 3D scene reconstruction into the training process, allowing the model to move beyond simple 2D proximity to a more robust 3D understanding of the environment.

Methodology

The LLaVA-SpaceSGG framework extends the LLaVA-1.5 architecture through a novel data construction pipeline and a specialized two-stage training paradigm.

The SpaceSGG Pipeline

To incorporate 3D spatial awareness, the pipeline utilizes:

  1. Depth Estimation: Depth-Anything generates depth maps from 2D input images.
  2. 3D Reconstruction: Point clouds are reconstructed and processed via a "layering algorithm" to cluster objects into spatial layers (near, middle, and far).
  3. LLM Synthesis: Llama 3 70B translates this geometric data into natural language, producing spatial descriptions, QA pairs, and multi-turn conversations.
The two-stage pipeline for LLaVA-SpaceSGG, showing 2D spatial representation discovery and LLM-based data generation.

Two-Stage Training

  • Stage 1 (Alignment): Standard alignment of the CLIP image encoder with the language model using large-scale image-text pairs.
  • Stage 2 (Refinement): Task-specific training on a mixture of general SGG data and the new SpaceSGG dataset, focusing on region-level spatial grounding and depth reasoning.

Evaluation and Results

The model was evaluated on the Panoptic Scene Graph (PSG) benchmark and a newly proposed Spatial Relation Validation Set.

  • Open-Vocabulary Performance: LLaVA-SpaceSGG achieved a significant performance boost on the PSG benchmark, increasing Recall by 8.6% and Mean Recall by 28.4% over the ASMv2 baseline.
  • Spatial Reasoning: On the spatial validation set (271 manually annotated QA pairs), the model reached 52.48% accuracy, outperforming LLaVA-1.5 (45.13%) and ASMv2 (50.52%).
  • Qualitative Analysis: The model demonstrates superior ability in distinguishing nuanced spatial terms like "beside" vs. "in front of," avoiding the "flat" 2D descriptions common in previous models.

Conclusion

LLaVA-SpaceSGG demonstrates that explicitly modeling 3D spatial layers and leveraging LLMs for geometric translation significantly enhances SGG. While absolute recall remains a challenge for the field, the relative gains in open-vocabulary and spatial reasoning mark a substantial step forward. Future work will explore more complex 3D dynamics and video-based spatial reasoning.

Code and models are available at: https://github.com/Endlinc/LLaVA-SpaceSGG

Summary updated

Abstract

Scene Graph Generation (SGG) converts visual scenes into structured graph representations, providing deeper scene understanding for complex vision tasks. However, existing SGG models often overlook essential spatial relationships and struggle with generalization in open-vocabulary contexts. To address these limitations, we propose LLaVA-SpaceSGG, a multimodal large language model (MLLM) designed for open-vocabulary SGG with enhanced spatial relation modeling. To train it, we collect the SGG instruction-tuning dataset, named SpaceSGG. This dataset is constructed by combining publicly available datasets and synthesizing data using open-source models within our data construction pipeline. It combines object locations, object relations, and depth information, resulting in three data formats: spatial SGG description, question-answering, and conversation. To enhance the transfer of MLLMs' inherent capabilities to the SGG task, we introduce a two-stage training paradigm. Experiments show that LLaVA-SpaceSGG outperforms other open-vocabulary SGG methods, boosting recall by 8.6% and mean recall by 28.4% compared to the baseline. Our codebase, dataset, and trained models are publicly accessible on GitHub at the following URL: https://github.com/Endlinc/LLaVA-SpaceSGG.
BibTeX
@inproceedings{xu2025llava,
  author = {Mingjie Xu and Mengyang Wu and Yuzhi Zhao and Jason Chun Lok Li and Weifeng Ou},
  title = {{LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00620},
}
Figure