Differentiable Scene Graphs

Moshiko Raboh, Roei Herzig, Jonathan Berant, Gal Chechik, Amir Globerson

Introduction

In the field of computer vision, the ability to not only detect objects but also understand the complex relationships between them is a cornerstone of visual reasoning. Scene Graphs (SGs) have emerged as a powerful abstraction for this purpose, representing images as structured sets of entities (nodes) and their interactions (edges). However, traditional scene graphs are typically discrete and categorical, which introduces significant challenges for modern deep learning pipelines.

Standard systems often rely on a disjointed two-step process: a detector first predicts a fixed graph with discrete labels (e.g., "man" "riding" "horse"), and a subsequent reasoning module operates on this static structure. This approach suffers from two primary limitations. First, the discrete nature of the graph prevents the backpropagation of gradients, making it impossible to fine-tune the initial detector based on the success or failure of the final reasoning task. Second, these models are limited by a fixed vocabulary; if a specific concept is missing from the pre-trained graph generator, the reasoning module cannot access that information, even if the visual evidence is present in the image.

To address these issues, this research introduces Differentiable Scene Graphs (DSGs). By replacing discrete labels with dense, differentiable descriptors, DSGs allow for end-to-end training of the entire pipeline—from raw pixels to complex relational reasoning.

The Differentiable Scene Graph Architecture

The core innovation of this work is the DSG layer, which maintains the structural benefits of a scene graph while remaining "soft" and continuous. Instead of committing to a single classification label for a node or edge, the model represents each element as a high-dimensional feature vector (a descriptor).

The architecture consists of four primary stages:

  1. Object Detection: A Region Proposal Network (RPN) identifies candidate bounding boxes for potential entities within the image.
  2. Feature Extraction: Visual features are extracted for each individual box (representing nodes) and for the "union boxes" that encompass pairs of objects (representing potential edges).
  3. DSG Generation via Graph Neural Networks (GNN): These features are processed through a GNN. This step is critical as it allows the model to incorporate global context. The representation of an object is refined based on its neighbors; for instance, the descriptor for a "man" is adjusted differently if he is "holding" a "surfboard" versus "riding" a "horse."
  4. Downstream Task Modules: The resulting differentiable descriptors are passed to task-specific components, such as a Referring Relationships Classifier (to match boxes to a query) and a Box Refiner (to optimize bounding box coordinates).

By utilizing a multi-task loss function, the entire system can be optimized via gradient descent, ensuring that the learned descriptors are specifically tuned for the reasoning task at hand without requiring exhaustive manual scene graph annotations.

Experimental Evaluation

The efficacy of the DSG framework was evaluated on the "Referring Relationships" task across three diverse benchmarks:

  • Visual Genome: A large-scale dataset featuring real-world images with complex, dense relationship annotations.
  • VRD (Visual Relationship Detection): A dataset focused on a broad variety of relationship types and object categories.
  • CLEVR: A synthetic dataset designed to test multi-step spatial reasoning and logic.

Performance was primarily measured using Mean Intersection-Over-Union (IOU), which evaluates the accuracy of the localized subjects and objects relative to the ground truth.

Key Results and Insights

The DSG model established new state-of-the-art (SOTA) performance across all tested datasets. On the synthetic CLEVR dataset, the model achieved a remarkable 96.3% IOU, effectively halving the error rate of previous SOTA models. In the more challenging Visual Genome environment, the model significantly outperformed the previous leading method (SASS), increasing subject localization accuracy from 0.421 to 0.489.

Ablation studies further confirmed the importance of the DSG component. When the relational structure was removed (the "No-DSG" baseline), performance dropped sharply, demonstrating that the structured, context-aware descriptors are essential for accurate reasoning. Furthermore, the researchers demonstrated that while the DSG is continuous, it still captures interpretable structure; a traditional discrete scene graph can be "decoded" from the DSG with 76% accuracy for object labels, proving the model learns the underlying scene structure.

Conclusion and Future Outlook

Differentiable Scene Graphs represent a significant step forward in bridging the gap between low-level perception and high-level reasoning. By making the scene representation differentiable, the model avoids the bottlenecks of fixed vocabularies and non-differentiable pipelines.

While the model is highly effective, it remains dependent on the quality of the initial Object Detector. If the RPN fails to propose a box for a relevant entity (noted in roughly 30% of error cases), the DSG cannot reason about it. Future research will explore extending DSGs to tasks like Visual Question Answering (VQA) and Image Captioning, as well as incorporating unlabeled data to further enhance the robustness of the graph generation process.

Summary updated

Abstract

Reasoning about complex visual scenes involves perception of entities and their relations. Scene Graphs (SGs) provide a natural representation for reasoning tasks, by assigning labels to both entities (nodes) and relations (edges). Reasoning systems based on SGs are typically trained in a two-step procedure: first, a model is trained to predict SGs from images, and next a separate model is trained to reason based on the predicted SGs. However, it would seem preferable to train such systems in an end-to-end manner. The challenge, which we address here is that scene-graph representations are non-differentiable and therefore it isn't clear how to use them as intermediate components. Here we propose Differentiable Scene Graphs (DSGs), an image representation that is amenable to differentiable end-to-end optimization, and requires supervision only from the downstream tasks. DSGs provide a dense representation for all regions and pairs of regions, and do not spend modelling capacity on regions of the image that do not contain objects or relations of interest. We evaluate our model on the challenging task of identifying referring relationships (RR) in three benchmark datasets: Visual Genome, VRD and CLEVR. Using DSGs as an intermediate representation leads to new state-of-the-art performance. The full code is available at https://github.com/shikorab/DSG.
BibTeX
@inproceedings{raboh2020differentiable,
  author = {Moshiko Raboh and Roei Herzig and Jonathan Berant and Gal Chechik and Amir Globerson},
  title = {{Differentiable Scene Graphs}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093297},
}
Figure