Abstract
Scene Graph Generation (SGG) is traditionally constrained by the high cost of manual annotation and a closed-vocabulary limitation. This paper presents $VS^3$, a framework that utilizes pre-trained Visual-Semantic Spaces (VSS) to enable SGG training via image captions alone. By leveraging foundation models like GLIP, $VS^3$ achieves state-of-the-art performance in fully supervised settings and demonstrates remarkable generalization to novel, unseen object categories.
Introduction
Scene Graph Generation (SGG) transforms an image into a structured graph where nodes represent objects and edges represent their relationships (e.g., [man] -holding-> [baseball bat]). While essential for high-level visual reasoning, SGG faces two major hurdles:
- The Annotation Bottleneck: Creating ground-truth scene graphs is tedious, requiring manual bounding boxes and relationship labels for every interaction.
- The Closed-Vocabulary Limitation: Most SGG models are trained on a fixed set of categories. Encountering a "player" when only trained on "man" typically leads to failure.
$VS^3$ addresses these barriers by using "cheap" language supervision (captions) and enabling "open-vocabulary" recognition through a pre-trained Visual-Semantic Space.
Methodology
The core idea of $VS^3$ is to exploit a powerful pre-trained Visual-Semantic Space (VSS)—specifically building upon GLIP (Grounded Language-Image Pre-training). GLIP excels at "phrase grounding," matching words in a sentence to specific regions in an image.
The $VS^3$ Framework
- Weak Supervision from Captions: Instead of manual labels, raw image captions are passed through a semantic parser to create "unlocalized" scene graphs. The pre-trained VSS then automatically grounds these noun phrases to image regions, creating a training signal.
- Open-Vocabulary Generalization: To detect novel objects at test time, the model uses prompt-based inference. By adding new object names to the text prompt, the VSS aligns these words to regions even if they were not present during the SGG fine-tuning phase.
Architecture Details
The $VS^3$ architecture inherits image and text encoders from GLIP and introduces a specialized Relation Embedding Module.
For every pair of detected objects, the model extracts:
- Visual Features: The difference and sum of the region embeddings from the VSS. Ablation studies show these high-dimensional embeddings are more critical for recognizing relations than simple spatial coordinates.
- Spatial Features: Geometric data including box coordinates, distance, and the angle between objects.
- Relation Prediction: A lightweight head predicts "relateness" and classifies the predicate (e.g., "standing on").
Key Results and Insights
The authors evaluated $VS^3$ primarily on the Visual Genome (VG150) benchmark, introducing a filtered split to prevent data leakage from the pre-trained GLIP model.
- State-of-the-Art Performance: In fully supervised settings, $VS^3$ (Swin-L backbone) achieved a Recall@100 of 41.5% on the Scene Graph Detection (SGDET) task, outperforming specialized models like HL-Net (38.1%).
- Weak Supervision Success: When trained only on unlocalized graphs from captions, $VS^3$ achieved 34.96% R@100, outperforming many older fully-supervised methods.
- Open-Vocabulary Capability: On a split with 30% novel objects, $VS^3$ achieved 27.62% R@100, whereas traditional closed-vocabulary models score 0%.
Limitations and Future Work
Despite its success, $VS^3$ faces challenges with predicate bias, where language-supervised models tend to predict simpler, more common predicates like "on" or "of" due to the nature of natural language captions. Additionally, domain shifts between training captions (e.g., COCO) and testing benchmarks (Visual Genome) can impact performance. Future directions include more advanced prompt engineering for relations and scaling to larger foundation models to capture nuanced, long-tail relationships.
Summary updated