Learning Latent Scene-Graph Representations for Referring Relationships

Moshiko Raboh, Roei Herzig, Gal Chechik, Jonathan Berant, Amir Globerson

Abstract

Reasoning about complex visual scenes involves perception of entities and their relations. Scene graphs provide a natural representation for reasoning tasks, by assigning labels to both entities (nodes) and relations (edges). Unfortunately, reasoning systems based on SGs are typically trained in a two-step procedure: First, training a model to predict SGs from images; Then, a separate model is created to reason based on predicted SGs. In many domains, it is preferable to train systems jointly in an end-to-end manner, but SGs are not commonly used as intermediate components in visual reasoning systems because being discrete and sparse, scene-graph representations are non-differentiable and difficult to optimize. Here we propose Differentiable Scene Graphs (DSGs), an image representation that is amenable to differentiable end-to-end optimization, and requires supervision only from the downstream tasks. DSGs provide a dense representation for all regions and pairs of regions, and do not spend modelling capacity on areas of the images that do not contain objects or relations of interest. We evaluate our model on the challenging task of identifying referring relationships (RR) in three benchmark datasets, Visual Genome, VRD and CLEVR. We describe a multi-task objective, and train in an end-to-end manner supervised by the downstream RR task. Using DSGs as an intermediate representation leads to new state-of-the-art performance.

Introduction

Scene Graphs (SGs) are a powerful way to represent visual data, mapping objects (nodes) and their relationships (edges) into a structured format. However, using them for complex reasoning tasks like "find the man catching the frisbee" has historically been a two-step, "broken" process.

Overview of the visual reasoning task using an intermediate graph representation.

First, a model is trained to predict a discrete scene graph (e.g., "man," "throwing," "frisbee"). Second, a separate model reasons over that graph. This approach suffers from several flaws:

  1. Non-Differentiability: Because standard SGs are discrete (using hard labels), you can't easily propagate gradients back from the reasoning task to the image perception layer.
  2. Annotation Bottleneck: Training a good SG predictor requires massive amounts of manually labeled data.
  3. Low Coverage: Pre-trained SG predictors are limited by their fixed vocabulary. If a downstream task asks about an "officer" but the SG only knows "man," the system fails.

The authors propose DSGs to solve these issues by keeping the representation "soft" and differentiable throughout the entire pipeline.

Methodology

The core innovation is the DSG Layer, which replaces discrete labels with dense, distributed representation vectors (node and edge descriptors).

The Differentiable Scene Graph architecture, showing the flow from object detection to the DSG generator and downstream reasoning modules.

The architecture follows a four-stage pipeline:

  1. Object Detection: A Region Proposal Network (RPN) identifies candidate bounding boxes.
  2. Feature Extraction: ROI-Align extracts visual features for each box and "union boxes" (the area covering a pair of objects).
  3. DSG Generator: This uses a Graph Neural Network (GNN)—specifically a Graph Permutation Invariant (GPI) architecture—to transform local features into context-aware descriptors. Each node and edge descriptor now contains information about the entire scene, not just its local patch.
  4. Downstream Reasoning: These differentiable descriptors are fed into two modules:
  5. Box Refiner: Fine-tunes the coordinates of the bounding boxes to fit the specific entities in the query.
  6. Referring Relationships Classifier: Predicts whether a box is the "Subject," "Object," "Other," or "Background" relative to the input query.

The entire system is trained with a multi-task loss, allowing the model to learn what a "good" scene graph looks like based purely on whether it helps solve the final reasoning task.

Experiments and Results

The model was evaluated on three major benchmarks for the Referring Relationships (RR) task: Visual Genome (VG), VRD (Visual Relationship Detection), and CLEVR.

The DSG approach significantly outperformed previous state-of-the-art (SOTA) methods across all datasets:

  • Visual Genome: Achieved a mean Intersection-over-Union (IOU) of 0.489 for subjects and 0.539 for objects, a substantial jump over the previous SASS baseline (0.421 / 0.482).
  • CLEVR: The model reached near-perfect performance with 96.3% IOU. Notably, even a "No-DSG" version of their end-to-end pipeline performed well, but the DSG layer provided the final boost to SOTA.
  • Efficiency: Despite the added complexity of the GNN, the DSG component only adds about 10% more parameters and 15% more training time compared to a standard backbone like ResNet101.

Discussion and Conclusion

By using a GNN to generate descriptors, the model can disambiguate similar-looking objects. For example, if there are two "men" in an image, the DSG can distinguish them based on their relationships to other objects. Interestingly, even though the model isn't strictly forced to learn human-readable labels, the authors found they could "decode" a traditional scene graph from the DSG with high accuracy (76% for objects).

While the model is powerful, it remains dependent on the initial object detector; about 30% of errors stem from the RPN failing to propose the correct bounding box. Future work may extend DSGs to tasks like Visual Question Answering (VQA) or Image Captioning, where structured but differentiable representations are highly desirable.

Summary updated

Abstract

Reasoning about complex visual scenes involves perception of entities and their relations. Scene graphs provide a natural representation for reasoning tasks, by assigning labels to both entities (nodes) and relations (edges). Unfortunately, reasoning systems based on SGs are typically trained in a two-step procedure: First, training a model to predict SGs from images; Then, a separate model is created to reason based on predicted SGs. In many domains, it is preferable to train systems jointly in an end-to-end manner, but SGs are not commonly used as intermediate components in visual reasoning systems because being discrete and sparse, scene-graph representations are non-differentiable and difficult to optimize. Here we propose Differentiable Scene Graphs (DSGs), an image representation that is amenable to differentiable end-to-end optimization, and requires supervision only from the downstream tasks. DSGs provide a dense representation for all regions and pairs of regions, and do not spend modelling capacity on areas of the images that do not contain objects or relations of interest. We evaluate our model on the challenging task of identifying referring relationships (RR) in three benchmark datasets, Visual Genome, VRD and CLEVR. We describe a multi-task objective, and train in an end-to-end manner supervised by the downstream RR task. Using DSGs as an intermediate representation leads to new state-of-the-art performance.
BibTeX
@inproceedings{raboh2020learning,
  author = {Moshiko Raboh and Roei Herzig and Gal Chechik and Jonathan Berant and Amir Globerson},
  title = {{Learning Latent Scene-Graph Representations for Referring Relationships}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093297},
}
Figure