Fast Contextual Scene Graph Generation with Unbiased Context Augmentation

Tianlei Jin, Fangtai Guo, Qiwei Meng, Shiqiang Zhu, Xiangming Xi, Wen Wang, Zonghao Mu, Wei SongCVPR 2023 · DOI

Abstract

Scene Graph Generation (SGG) is a critical task for visual understanding, yet it is often hindered by long-tail distribution bias and high computational costs. This paper presents a novel framework that leverages object context—specifically labels and spatial positions—to perform real-time relationship detection. By introducing "Context Augmentation" to generate unbiased training data and a lightweight "Contextual SGG" (C-SGG) model, the proposed approach achieves state-of-the-art performance on major benchmarks while maintaining high inference speeds.

Introduction

Scene Graph Generation (SGG) involves identifying triplets of <subject, predicate, object> within an image. Despite its importance for downstream tasks like robotics and navigation, SGG faces two primary challenges. First, real-world datasets exhibit a significant long-tail bias, where common predicates (e.g., "on", "near") dominate, causing models to ignore rarer but more descriptive relationships. Second, the computational complexity of analyzing all possible object pairs ($O(N^2)$) typically prevents real-time application.

Observations suggest that humans often infer relationships based on spatial context rather than fine visual details. For instance, knowing a "cup" is above a "table" strongly suggests the relationship "on" or "near" without needing to process high-resolution pixels. This paper formalizes this intuition into a context-driven SGG framework.

Methodology

The proposed framework consists of two primary components designed to balance speed and accuracy:

Contextual SGG (C-SGG)

The C-SGG model operates entirely on object categories and bounding box coordinates, ignoring image pixels. It utilizes a Context Knowledge Network (CKN), a simple three-layer fully connected network, to predict relationship probabilities based on semantic vectors (GloVe) and spatial offsets. This allows for extremely fast inference.

Context Guided Visual SGG (CV-SGG)

For higher precision, the C-SGG output acts as a "mask" to guide the Vision Differentiation Network (VDN). By filtering out impossible relationships (e.g., a person "wearing" a car), the VDN focuses its visual processing only on plausible candidates, significantly reducing redundant computation.

The proposed framework for Context Guided Visual Scene Graph Generation, highlighting the CKN and VDN components.

Unbiased Context Augmentation

To address long-tail bias, the authors introduce Context Augmentation. This technique "hallucinates" new training data by applying random perturbations to bounding box coordinates. Since spatial relationships like "on" are invariant to minor shifts, this method generates a balanced variety of context descriptions for rare predicates without the need for expensive new image annotations.

Conceptual overview of human speculation, context augmentation (Fake Image Production), and the resulting contextual vs. visual SGG predictions.

Key Results

The framework was evaluated on the Visual Genome (VG) and Panoptic Scene Graph (PSG) datasets. Key findings include:

  • Real-Time Performance: Paired with a YOLOv5 detector, the pipeline achieves 33.5 FPS, a significant improvement over traditional models like Motifs or VCTree (~4 FPS).
  • Balanced Accuracy: The method achieves state-of-the-art results on Mean Recall (mR@K) and F@K metrics, demonstrating its ability to handle tail predicates effectively.
  • Visual Genome Success: On the Predicate Classification (PredCls) task, CV-SGG achieved a Mean@100 score of 49.3, outperforming vision-heavy models while maintaining better predicate balance.

Conclusion

By prioritizing spatial and semantic context over raw visual features, this work demonstrates that SGG can be both fast and unbiased. The use of context augmentation provides a low-cost solution to the long-tail problem, while the two-stage C-SGG/CV-SGG architecture enables real-time visual understanding suitable for practical applications.

Summary updated

BibTeX
@inproceedings{jin2023fast,
  author = {Tianlei Jin and Fangtai Guo and Qiwei Meng and Shiqiang Zhu and Xiangming Xi and Wen Wang and Zonghao Mu and Wei Song},
  title = {{Fast Contextual Scene Graph Generation with Unbiased Context Augmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00610},
}
Figure