Neural Motifs: Scene Graph Parsing with Global Context

Rowan Zellers, Mark Yatskar, Sam Thomson, Yejin Choi

Abstract

We investigate the problem of producing structured graph representations of visual scenes. Our work analyzes the role of motifs: regularly appearing substructures in scene graphs. We present new quantitative insights on such repeated structures in the Visual Genome dataset. Our analysis shows that object labels are highly predictive of relation labels but not vice-versa. We also find that there are recurring patterns even in larger subgraphs: more than 50% of graphs contain motifs involving at least two relations. Our analysis motivates a new baseline: given object detections, predict the most frequent relation between object pairs with the given labels, as seen in the training set. This baseline improves on the previous state-of-the-art by an average of 3.6% relative improvement across evaluation settings. We then introduce Stacked Motif Networks, a new architecture designed to capture higher order motifs in scene graphs that further improves over our strong baseline by an average 7.1% relative gain.

Introduction

Scene graph parsing aims to transform an image into a structured representation where nodes represent objects and edges represent their relationships. While object detection has matured, predicting relationships remains challenging due to the complexity of visual context and the vast space of possible interactions.

Example of a scene graph representation where objects (nodes) are linked by relationships (edges).

The authors argue that visual scenes are not random collections of objects but follow strong structural regularities, termed motifs. For instance, a "man" and a "shirt" almost always imply a "wearing" relationship. This paper quantifies these motifs and introduces a model that leverages global context to improve parsing accuracy.

Quantifying Scene Graph Motifs

A key insight of this work is the asymmetry of information between objects and relationships. Object labels are highly predictive of relationship labels, but the reverse is not true. Knowing the objects are "cow" and "grass" makes "eating" a highly probable relationship.

Analysis of the Visual Genome dataset reveals that over 50% of images contain recurring subgraphs (motifs) involving at least two relationships. These patterns serve as templates for our visual world. This observation led to a surprisingly strong Frequency Baseline: simply predicting the most frequent relationship for a given pair of object categories from the training set outperforms previous state-of-the-art models by 3.6% relatively.

Stacked Motif Networks (MOTIFNET)

To capture higher-order motifs beyond simple pairwise frequencies, the authors propose Stacked Motif Networks (MOTIFNET). The architecture uses LSTMs to encode global context across the entire scene.

The MOTIFNET architecture uses stacked biLSTMs to capture object and edge context from bounding box proposals.

The model operates in stages:

  1. Bounding Box Proposals: A Faster R-CNN identifies potential object regions.
  2. Object Context: A bidirectional LSTM (biLSTM) processes the sequence of bounding boxes (ordered left-to-right) to encode global scene context, which is then used to predict object labels.
  3. Edge Context: A second biLSTM takes the predicted object labels and box features to create a context specifically for relationships.
  4. Relationship Prediction: Edge labels are predicted by combining contextualized representations of the "head" and "tail" objects with the visual features of their union bounding box.

Experimental Results

The model was evaluated on the Visual Genome dataset (VG150 split). MOTIFNET achieved significant improvements:

  • Relative Gains: MOTIFNET provided a 7.1% relative gain over the already strong frequency baseline.
  • Scene Graph Detection: In the most challenging setting—where the model must detect boxes, labels, and relations simultaneously—MOTIFNET achieved a Recall@100 of 30.3%, a massive leap from the 4.2% achieved by previous message-passing approaches.

Conclusion

Neural Motifs demonstrates that capturing global context and structural regularities is essential for scene understanding. By explicitly modeling motifs through stacked LSTMs, MOTIFNET sets a new standard for scene graph parsing, proving that "common sense" statistical priors are as important as local visual features.

Summary updated

Abstract

We investigate the problem of producing structured graph representations of visual scenes. Our work analyzes the role of motifs: regularly appearing substructures in scene graphs. We present new quantitative insights on such repeated structures in the Visual Genome dataset. Our analysis shows that object labels are highly predictive of relation labels but not vice-versa. We also find that there are recurring patterns even in larger subgraphs: more than 50% of graphs contain motifs involving at least two relations. Our analysis motivates a new baseline: given object detections, predict the most frequent relation between object pairs with the given labels, as seen in the training set. This baseline improves on the previous state-of-the-art by an average of 3.6% relative improvement across evaluation settings. We then introduce Stacked Motif Networks, a new architecture designed to capture higher order motifs in scene graphs that further improves over our strong baseline by an average 7.1% relative gain. Our code is available at github.com/rowanz/neural-motifs.
BibTeX
@inproceedings{zellers2018neural,
  author = {Rowan Zellers and Mark Yatskar and Sam Thomson and Yejin Choi},
  title = {{Neural Motifs: Scene Graph Parsing with Global Context}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2018},
  url = {https://openreview.net/forum?id=H1bWNxMubS},
}
Figure