Abstract
Scene Graph Generation (SGG) is a critical task for visual understanding, yet it is often hindered by long-tail distribution bias and high computational costs. This paper presents a novel framework that leverages object context—specifically labels and spatial positions—to perform real-time relationship detection. By introducing "Context Augmentation" to generate unbiased training data and a lightweight "Contextual SGG" (C-SGG) model, the proposed approach achieves state-of-the-art performance on major benchmarks while maintaining high inference speeds.
Introduction
Scene Graph Generation (SGG) involves identifying triplets of <subject, predicate, object> within an image. Despite its importance for downstream tasks like robotics and navigation, SGG faces two primary challenges. First, real-world datasets exhibit a significant long-tail bias, where common predicates (e.g., "on", "near") dominate, causing models to ignore rarer but more descriptive relationships. Second, the computational complexity of analyzing all possible object pairs ($O(N^2)$) typically prevents real-time application.
Observations suggest that humans often infer relationships based on spatial context rather than fine visual details. For instance, knowing a "cup" is above a "table" strongly suggests the relationship "on" or "near" without needing to process high-resolution pixels. This paper formalizes this intuition into a context-driven SGG framework.
Methodology
The proposed framework consists of two primary components designed to balance speed and accuracy:
Contextual SGG (C-SGG)
The C-SGG model operates entirely on object categories and bounding box coordinates, ignoring image pixels. It utilizes a Context Knowledge Network (CKN), a simple three-layer fully connected network, to predict relationship probabilities based on semantic vectors (GloVe) and spatial offsets. This allows for extremely fast inference.
Context Guided Visual SGG (CV-SGG)
For higher precision, the C-SGG output acts as a "mask" to guide the Vision Differentiation Network (VDN). By filtering out impossible relationships (e.g., a person "wearing" a car), the VDN focuses its visual processing only on plausible candidates, significantly reducing redundant computation.
Unbiased Context Augmentation
To address long-tail bias, the authors introduce Context Augmentation. This technique "hallucinates" new training data by applying random perturbations to bounding box coordinates. Since spatial relationships like "on" are invariant to minor shifts, this method generates a balanced variety of context descriptions for rare predicates without the need for expensive new image annotations.
Key Results
The framework was evaluated on the Visual Genome (VG) and Panoptic Scene Graph (PSG) datasets. Key findings include:
- Real-Time Performance: Paired with a YOLOv5 detector, the pipeline achieves 33.5 FPS, a significant improvement over traditional models like Motifs or VCTree (~4 FPS).
- Balanced Accuracy: The method achieves state-of-the-art results on Mean Recall (mR@K) and F@K metrics, demonstrating its ability to handle tail predicates effectively.
- Visual Genome Success: On the Predicate Classification (PredCls) task, CV-SGG achieved a Mean@100 score of 49.3, outperforming vision-heavy models while maintaining better predicate balance.
Conclusion
By prioritizing spatial and semantic context over raw visual features, this work demonstrates that SGG can be both fast and unbiased. The use of context augmentation provides a low-cost solution to the long-tail problem, while the two-stage C-SGG/CV-SGG architecture enables real-time visual understanding suitable for practical applications.
Summary updated