Image-Level Attentional Context Modeling Using Nested-Graph Neural Networks

Guillaume Jaume, Behzad Bozorgtabar, Hazim Kemal Ekenel, Jean-Philippe Thiran, Maria GabraniarXiv 2018 · arXiv

Abstract

Scene graph generation requires understanding both individual objects and their complex inter-relationships. Traditional methods rely on local message-passing, which is computationally expensive and often fails to capture global scene context. This paper introduces Image-Level Attentional Context (ILAC), a streamlined approach that utilizes a global "context agent" to efficiently share information across an entire image. ILAC achieves state-of-the-art performance with significantly fewer parameters than existing models.

Introduction

Identifying objects like "man" or "horse" is a fundamental computer vision task, but understanding the relationship "man feeding horse" requires a deeper level of contextual reasoning. Previous scene graph generation methods typically employed object-centric context, where information propagates slowly through a graph via node-to-edge and edge-to-node messages. This local approach is inefficient at capturing global scene properties—for instance, knowing a scene is a "kitchen" provides a strong prior for the relationship "cutting." The ILAC model addresses this by introducing a global hub for information exchange, facilitating faster and more effective propagation of contextual cues.

Methodology

The core of the proposed system is the Image-Level Attentional Context (ILAC) model, which moves beyond local message passing to a global context-aware architecture.

Initialization and the Context Agent

The process begins with bounding boxes and visual features extracted from a standard object detector. A fully connected graph is constructed where every object serves as a node and every potential relationship as an edge. The "context agent" is a global attribute that aggregates information from all nodes and edges using an attentional mechanism. This mechanism recognizes that certain objects, such as a traffic light in a street scene, are more informative for global context than others.

Iterative Refinement (Nested GNN)

ILAC employs a nested Graph Neural Network (GNN) for iterative refinement. The model uses a "graph-to-graph" approach to update representations across multiple iterations:

  1. Aggregation: The context agent collects new information from nodes and edges.
  2. Edge Refinement: The updated global context is used to refine relationship (predicate) predictions.
  3. Node Refinement: Both the updated context and refined edges are used to improve object classification.
The ILAC pipeline overview showing the iterative refinement process from initial features to the final scene graph.

By embedding visual features into a lower-dimensional space and utilizing this centralized hub, the model maintains high efficiency, requiring roughly three times fewer parameters than leading models like Neural Motifs.

Key Insights

The Homogeneity Principle

An entropy study conducted by the authors demonstrated that objects in a scene are highly dependent on their neighbors. Knowing just one other random object in an image significantly reduces the mathematical uncertainty (entropy) regarding other objects or relationships.

Global Hubs vs. Local Message Passing

The global context agent allows information to propagate across the graph instantly, avoiding the bottlenecks inherent in traditional message passing where information must traverse multiple edges to reach distant nodes.

Relational Bias in Detection

Interestingly, ILAC improves the underlying object detector. By understanding the broader context, the model can correct misclassifications made by detectors that analyze objects in isolation. For example, understanding the "train" context helps refine the detection of internal components like "windshields" or external tracks.

Results and Discussion

ILAC demonstrates competitive performance against state-of-the-art models. On the Scene Graph Classification (SGCls) task using the Visual Genome dataset, it achieved 43.7% R@50.

  • Efficiency: While the Neural Motifs (NM) model uses 40 million parameters, ILAC achieves similar accuracy with only 14.8 million parameters.
  • Ablation Success: Removing the image-level context module caused a 3% drop in performance, proving that the global context agent is doing the heavy lifting.
  • Detection Boost: Incorporating ILAC improved baseline object detection accuracy by 1.5%.
Qualitative example of object detection and potential scene graph nodes within a complex environment.

Conclusion

ILAC provides a highly efficient and effective framework for scene graph generation by leveraging global attentional context. While the benefits of iterative refinement plateau after two iterations, the model's lightweight nature and ability to improve base detection tasks make it a significant contribution. Future work may explore integrating external linguistic priors or making the architecture more detector-agnostic.

Summary updated

Abstract

We introduce a new scene graph generation method called image-level attentional context modeling (ILAC). Our model includes an attentional graph network that effectively propagates contextual information across the graph using image-level features. Whereas previous works use an object-centric context, we build an image-level context agent to encode the scene properties. The proposed method comprises a single-stream network that iteratively refines the scene graph with a nested graph neural network. We demonstrate that our approach achieves competitive performance with the state-of-the-art for scene graph generation on the Visual Genome dataset, while requiring fewer parameters than other methods. We also show that ILAC can improve regular object detectors by incorporating relational image-level information.
BibTeX
@misc{jaume2018image,
  author = {Guillaume Jaume and Behzad Bozorgtabar and Hazim Kemal Ekenel and Jean-Philippe Thiran and Maria Gabrani},
  title = {{Image-Level Attentional Context Modeling Using Nested-Graph Neural Networks}},
  year = {2018},
  eprint = {1811.03830},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.03830},
}
Figure