Abstract
Scene graph generation requires understanding both individual objects and their complex inter-relationships. Traditional methods rely on local message-passing, which is computationally expensive and often fails to capture global scene context. This paper introduces Image-Level Attentional Context (ILAC), a streamlined approach that utilizes a global "context agent" to efficiently share information across an entire image. ILAC achieves state-of-the-art performance with significantly fewer parameters than existing models.
Introduction
Identifying objects like "man" or "horse" is a fundamental computer vision task, but understanding the relationship "man feeding horse" requires a deeper level of contextual reasoning. Previous scene graph generation methods typically employed object-centric context, where information propagates slowly through a graph via node-to-edge and edge-to-node messages. This local approach is inefficient at capturing global scene properties—for instance, knowing a scene is a "kitchen" provides a strong prior for the relationship "cutting." The ILAC model addresses this by introducing a global hub for information exchange, facilitating faster and more effective propagation of contextual cues.
Methodology
The core of the proposed system is the Image-Level Attentional Context (ILAC) model, which moves beyond local message passing to a global context-aware architecture.
Initialization and the Context Agent
The process begins with bounding boxes and visual features extracted from a standard object detector. A fully connected graph is constructed where every object serves as a node and every potential relationship as an edge. The "context agent" is a global attribute that aggregates information from all nodes and edges using an attentional mechanism. This mechanism recognizes that certain objects, such as a traffic light in a street scene, are more informative for global context than others.
Iterative Refinement (Nested GNN)
ILAC employs a nested Graph Neural Network (GNN) for iterative refinement. The model uses a "graph-to-graph" approach to update representations across multiple iterations:
- Aggregation: The context agent collects new information from nodes and edges.
- Edge Refinement: The updated global context is used to refine relationship (predicate) predictions.
- Node Refinement: Both the updated context and refined edges are used to improve object classification.
By embedding visual features into a lower-dimensional space and utilizing this centralized hub, the model maintains high efficiency, requiring roughly three times fewer parameters than leading models like Neural Motifs.
Key Insights
The Homogeneity Principle
An entropy study conducted by the authors demonstrated that objects in a scene are highly dependent on their neighbors. Knowing just one other random object in an image significantly reduces the mathematical uncertainty (entropy) regarding other objects or relationships.
Global Hubs vs. Local Message Passing
The global context agent allows information to propagate across the graph instantly, avoiding the bottlenecks inherent in traditional message passing where information must traverse multiple edges to reach distant nodes.
Relational Bias in Detection
Interestingly, ILAC improves the underlying object detector. By understanding the broader context, the model can correct misclassifications made by detectors that analyze objects in isolation. For example, understanding the "train" context helps refine the detection of internal components like "windshields" or external tracks.
Results and Discussion
ILAC demonstrates competitive performance against state-of-the-art models. On the Scene Graph Classification (SGCls) task using the Visual Genome dataset, it achieved 43.7% R@50.
- Efficiency: While the Neural Motifs (NM) model uses 40 million parameters, ILAC achieves similar accuracy with only 14.8 million parameters.
- Ablation Success: Removing the image-level context module caused a 3% drop in performance, proving that the global context agent is doing the heavy lifting.
- Detection Boost: Incorporating ILAC improved baseline object detection accuracy by 1.5%.
Conclusion
ILAC provides a highly efficient and effective framework for scene graph generation by leveraging global attentional context. While the benefits of iterative refinement plateau after two iterations, the model's lightweight nature and ability to improve base detection tasks make it a significant contribution. Future work may explore integrating external linguistic priors or making the architecture more detector-agnostic.
Summary updated