Iterative Scene Graph Generation

Siddhesh Khandelwal, Leonid Sigal

Introduction

In computer vision, a Scene Graph is a structured representation of an image where nodes represent objects (e.g., "person," "bike") and edges represent their relationships (e.g., "riding," "near"). Generating these graphs accurately is crucial for deep visual understanding, but the combinatorially large solution space makes it a challenging task.

Most existing models follow a "one-shot" pipeline: first, detect all objects, then predict the relationships between them. This fixed factorization (Objects $\rightarrow$ Predicates) creates a "chicken-and-egg" problem. For example, it is difficult to identify a tiny, blurry object as a "ring" unless you already know the person is "wearing" it. Conversely, you cannot confirm the "wearing" relationship without identifying the ring. Because errors in object detection propagate downstream and cannot be fixed, these models also struggle with the "long-tail" problem, where common relationships like "on" or "has" drown out more descriptive but rare ones like "parked on" or "eating."

To address this, researchers from the University of British Columbia have developed a new Transformer-based framework that generates scene graphs through iterative refinement. By treating scene graph generation as a multi-step process rather than a single-pass prediction, the model allows objects and their relationships to "talk" to each other, significantly improving accuracy on complex interactions and rare "tail" classes.

Methodology: The IterativeSG Framework

The authors propose IterativeSG, a framework that treats scene graph generation as an iterative refinement process, allowing joint reasoning over entities and relations.

IterativeSG Architecture
  1. Unrolled Markov Random Field (MRF): The dependencies between subjects, objects, and predicates are modeled as an MRF. By "unrolling" this message-passing process, the framework creates a sequence of refinement steps where each step improves upon the last.
  2. Triple-Decoder Transformer: Unlike standard object detectors, this architecture uses three separate but synchronized decoders for Subjects, Objects, and Predicates.
  3. Dynamic Conditioning: At each refinement step $t$, the decoders are conditioned on the output of step $t-1$. Crucially, they use cross-decoder attention, allowing the predicate decoder to "look" at what the subject/object decoders are predicting, and vice versa.
  4. Joint Bipartite Matching Loss: Instead of calculating loss for each component separately, the model uses a global matching strategy to ensure the entire $<subject, predicate, object>$ triplet is optimized as a single unit.
  5. Loss Re-weighting: To handle the long-tail distribution of relationships, they implement a simple but effective frequency-based loss weighting scheme, avoiding the need for complex data oversampling.

Key Results and Insights

The model was evaluated on two primary benchmarks: Visual Genome (VG) (the gold standard for scene graphs) and Action Genome (AG) (a video-based dataset focused on human-object interactions). The authors report significant gains across standard metrics like Mean Recall (mR@K) and a new metric they emphasize, Harmonic Recall (hR@K), which balances performance on both common and rare classes.

Visual Genome Results Comparison
  • Mastering the Long Tail: On the most challenging "tail" classes in the Visual Genome dataset, IterativeSG outperformed the previous state-of-the-art (SGTR) by a wide margin (e.g., 15.8 vs 7.1 on tail class mR@100).
  • Action Genome Improvements: The model achieved a 3.1% absolute improvement in Recall@20 and a massive 12.6% jump in Mean Recall@20 compared to the SimpleBase baseline.
  • Joint Reasoning is Essential: The paper demonstrates that subjects, objects, and predicates are deeply intertwined. Allowing the model to update its "belief" about an object based on a predicted relationship leads to much more coherent graphs.
  • Model Agnostic: The iterative formulation is not exclusive to Transformers. The authors applied it to MOTIF (a popular older LSTM-based model) and saw a ~6% performance boost, proving the concept's generality.
  • Qualitative Self-Correction: Visualizations show the model successfully correcting its own mistakes—for instance, changing a generic "at" relationship to a more specific "walking on" as the refinement steps progress.

Limitations and Future Work

While IterativeSG sets a new standard, it inherits some limitations from its base architecture. Because the model builds on the DETR (Detection Transformer) architecture, it struggles with very small objects compared to traditional anchor-based detectors like Faster R-CNN. Additionally, iterative refinement naturally requires more computation than a single-pass model, presenting a trade-off between the number of refinement steps and inference speed.

Finally, the model uses a shared image encoder for all decoders. The authors suggest that exploring more diverse, specialized feature representations for different decoder layers could be a promising future direction to push scene graph generation even further.

Summary updated

Abstract

The task of scene graph generation entails identifying object entities and their corresponding interaction predicates in a given image (or video). Due to the combinatorially large solution space, existing approaches to scene graph generation assume certain factorization of the joint distribution to make the estimation feasible (e.g., assuming that objects are conditionally independent of predicate predictions). However, this fixed factorization is not ideal under all scenarios (e.g., for images where an object entailed in interaction is small and not discernible on its own). In this work, we propose a novel framework for scene graph generation that addresses this limitation, as well as introduces dynamic conditioning on the image, using message passing in a Markov Random Field. This is implemented as an iterative refinement procedure wherein each modification is conditioned on the graph generated in the previous iteration. This conditioning across refinement steps allows joint reasoning over entities and relations. This framework is realized via a novel and end-to-end trainable transformer-based architecture. In addition, the proposed framework can improve existing approach performance. Through extensive experiments on Visual Genome and Action Genome benchmark datasets we show improved performance on the scene graph generation.
BibTeX
@inproceedings{khandelwal2022iterative,
  author = {Siddhesh Khandelwal and Leonid Sigal},
  title = {{Iterative Scene Graph Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=FBwGpnrATC},
}
Figure