Introduction
Visual Relationship Detection (VRD) is a fundamental computer vision task that involves identifying objects and the semantic interactions between them, typically represented as <subject-predicate-object> triplets (e.g., "person riding bicycle"). Traditional VRD systems often rely on multi-stage pipelines where an object detector is followed by a separate relationship decoder or interaction module. These architectures tend to be computationally heavy, slow, and restricted to a "closed-vocabulary" setting, limiting their utility in real-world, dynamic environments.
Scene-Graph ViT introduces a streamlined, encoder-only Transformer architecture that treats relationships as primary entities within the image encoder. By leveraging the inherent self-attention mechanisms of Vision Transformers (ViT), the model achieves state-of-the-art performance in open-vocabulary scenarios while maintaining real-time inference speeds.
Methodology: An Encoder-Only Approach
The core innovation of Scene-Graph ViT lies in its simplicity. Unlike previous models that utilize complex Transformer decoders, this architecture extracts relationship information directly from the output tokens of a standard ViT encoder. The insight is that the encoder's self-attention mechanism is already modeling all-to-all interactions between image regions.
Relationship Attention Layer
To identify interactions, the model projects image tokens into two distinct embedding spaces: "subject" and "object." This is achieved using two lightweight Multi-Layer Perceptrons (MLPs). The authors emphasize that this "symmetry breaking" is essential; using a single projection for both roles would prevent the model from distinguishing directional interactions, such as the difference between "person riding horse" and "horse riding person."
Efficient Pair Selection
A significant challenge in VRD is the quadratic complexity ($N^2$) of checking every possible pair of detected objects. Scene-Graph ViT addresses this through a "hard attention" selection mechanism. This self-supervised component identifies and selects only the top $k$ most likely interacting pairs for further processing. This approach reduces the computational workload by over 99%, enabling the model to scale effectively without needing explicit gradients to flow through the selection process.
Open-Vocabulary Generalization
By integrating a pre-trained Vision-Language Model (VLM) backbone, such as CLIP, Scene-Graph ViT can perform open-vocabulary classification. The model compares generated relationship embeddings against text embeddings of various predicates (e.g., "riding," "near," "inside"). Because the architecture disentangles object identities from their actions, it can generalize to novel combinations of subjects and predicates not seen during training.
Experimental Results
The model was evaluated across several rigorous benchmarks, including Visual Genome (VG150), GQA, HICO-DET, and Objects365.
- State-of-the-Art Performance: On the VG150 benchmark, Scene-Graph ViT achieved a mean Recall (mR@100) of 26.1%, surpassing the previous best-in-class model (DT2-ACBS) at 24.4%.
- Large-Vocabulary Scalability: The model showed marked improvements on the GQA dataset, demonstrating its ability to handle a diverse and expansive set of labels.
- Inference Speed: Efficiency is a standout feature. The B/32 variant of the model operates at 52.8 FPS on an NVIDIA V100 GPU, reaching approximately 90% of the speed of a standalone object detector. This makes it viable for real-time applications like robotics and live video analysis.
Discussion and Limitations
While Scene-Graph ViT represents a significant step forward, certain challenges remain. There is still a performance gap in zero-shot scenarios compared to supervised classes. Additionally, while the model is competitive on the HICO-DET dataset for Human-Object Interaction (HOI), it does not exhibit the same transformative leap in performance as seen in general scene graph tasks, suggesting that HOI may benefit from more specialized human-centric data.
Finally, the model occasionally struggles with linguistic nuances, such as pluralization. For instance, it may group multiple "bananas" into a single bounding box when a singular query is provided. This limitation is likely inherited from the underlying VLM's embedding space and remains an area for future refinement.
Conclusion
Scene-Graph ViT demonstrates that a simplified, encoder-only architecture can outperform complex, multi-stage systems in visual relationship detection. By integrating relationship modeling directly into the visual representation process, the model achieves a rare balance of high accuracy, open-vocabulary flexibility, and real-time efficiency.
Summary updated