Composite Relationship Fields with Transformers for Scene Graph Generation

George Adaimi, David Mizrahi, Alexandre AlahiWACV 2023 · DOI

Introduction

Scene Graph Generation (SGG) is a critical task in computer vision that involves transforming an image into a structured graph where nodes represent objects and edges represent their semantic relationships (e.g., <person, riding, horse>). Traditionally, SGG has been approached through top-down methods: detecting all objects first and then evaluating every possible pair for a relationship. While accurate, this approach is computationally expensive, as the number of pairs grows quadratically with the number of objects, hindering real-time performance.

Bottom-up approaches attempt to detect objects and relationships simultaneously in a single pass. However, they often suffer from discretization errors—where coarse grids fail to distinguish between closely situated objects—and limited receptive fields, making it difficult to link objects across large spatial distances. This paper introduces a high-performance, real-time SGG method that treats relationship detection as a dense regression task, significantly improving accuracy and speed.

Methodology

The proposed architecture addresses the limitations of previous bottom-up methods through two primary innovations: a Transformer-based refinement head and Composite Relationship Fields (CoRF).

The proposed architecture featuring a backbone, dual Transformer-based detection heads, and a scene graph decoder.

Transformer Refinement Head

To overcome the "local vision" constraint of standard Convolutional Neural Networks (CNNs), the authors integrate a Transformer encoder before the detection heads. While the backbone (e.g., ResNet or Swin Transformer) extracts local features, the Transformer uses self-attention mechanisms to allow global communication across the entire image. This global context is essential for identifying relationships between objects that are spatially distant.

Composite Relationship Fields (CoRF)

The core of the relationship detection head is the Composite Relationship Field (CoRF). Unlike traditional affinity fields that use discrete bins, CoRF predicts continuous, real-valued offsets. For every cell in the output feature map, the model predicts:

  • Confidence: The likelihood of a relationship center.
  • Subject/Object Vectors: Precise pointers to the centers of the involved objects.
  • Scales: The dimensions of the subject and object.
Visualization of relationship fields where dense pointers (lines) connect the relationship center to the subject and object.

By using continuous vectors, the model can accurately differentiate between overlapping or very small features that would typically be lost in a discretized grid.

Evaluation and Key Results

The model was evaluated on the Visual Genome (VG-150) dataset, focusing on Scene Graph Detection (SGDet) and Zero-Shot Recall.

  • Accuracy and Speed: The method outperformed the previous state-of-the-art bottom-up model (FCSGG) by 26% on Visual Genome. Crucially, it maintained real-time performance, achieving 15 to 30 frames per second (FPS) depending on the backbone.
  • Zero-Shot Generalization: The model demonstrated a remarkable 50% improvement in zero-shot recall. This suggests that the dense field representation helps the model learn the underlying spatial and semantic logic of relationships rather than simply memorizing common triplets.
  • Closing the Gap: While top-down methods still hold a slight edge in some accuracy metrics, this approach significantly narrows the gap while offering the speed necessary for robotics and video processing.

Conclusion and Future Work

By combining the precision of real-valued pointers (CoRF) with the global reasoning of Transformers, this work establishes a new benchmark for efficient Scene Graph Generation. The primary remaining challenge is the "long-tail" distribution of relationships, where rare predicates remain difficult to predict. Future research may extend this dense field approach to temporal domains like action recognition in video or 3D scene understanding.

Summary updated

BibTeX
@inproceedings{adaimi2023composite,
  author = {George Adaimi and David Mizrahi and Alexandre Alahi},
  title = {{Composite Relationship Fields with Transformers for Scene Graph Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00014},
}
Figure