Deep Generative Probabilistic Graph Neural Networks for Scene Graph Generation

Mahmoud Khademi, Oliver Schulte

Abstract

Deep Generative Probabilistic Graph Neural Networks (DG-PGNN) represent a significant advancement in scene graph generation. By treating graph construction as a sequential reinforcement learning problem and explicitly modeling uncertainty through a Probabilistic Graph Network (PGN), the framework achieves state-of-the-art results. The integration of language priors from image captions further enhances its ability to resolve visual ambiguities, leading to improved performance in complex visual question-answering (VQA) tasks.

Introduction and Problem Statement

In computer vision, understanding a scene goes beyond simple object detection; it requires identifying the complex web of relationships between objects (e.g., "man holding camera," "trees on mountain"). These relationships are structured as scene graphs.

Overview of scene graph generation and its application in Visual Question Answering (VQA).

Existing methods for generating these graphs often struggle with two primary challenges:

  1. Ordering: They usually process objects in a fixed, arbitrary order (like left-to-right), which doesn't reflect how humans perceive scenes or how contextual information flows.
  2. Uncertainty: Most models make "hard" assignments for object and relationship types early on. They lack a robust way to represent and update their "belief" about a relationship as more of the scene is revealed.

The authors introduce DG-PGNN to build graphs dynamically, learning the best sequence to add nodes while leveraging both visual context and textual "priors" to resolve ambiguity.

Methodology

The core of the framework is the sequential construction of a Probabilistic Graph Network (PGN).

The architecture of the Probabilistic Graph Network (PGN) and the Q-Network used for sequential node selection.

Key components include:

  • Probabilistic Graph Network (PGN): Unlike standard graphs, a PGN represents every node and edge with a Probability Mass Function (PMF). This allows the model to maintain a "degree of belief" about categories that can be refined over time.
  • Deep Q-Learning for Node Selection: The model uses a Reinforcement Learning (RL) framework where a Q-network decides which bounding box to add to the graph next. The "state" is the current partial graph, and the "reward" is based on ground-truth matching.
  • Contextual Message Passing: Every time a new node is added, the model performs message passing using Gated Recurrent Units (GRUs), allowing information to flow between nodes and update their feature vectors.
  • Caption-Guided Updates: The model uses "region-grounded captions" as language priors. If a caption mentions a "yellow umbrella," the model uses that text to fine-tune the probabilities of the corresponding node, acting as a semantic anchor against noisy visual data.

Datasets and Evaluation

The researchers evaluated their model on three major benchmarks:

  • Visual Genome: The gold standard for scene graph generation with dense relationship annotations.
  • Visual7W: Used to test the model's utility in Visual Question Answering (VQA) on real-world images.
  • CLEVR: A synthetic dataset designed to test complex spatial reasoning.

Key Insights

  • The Order Matters: By using RL, the model learns that some objects provide more "contextual value" than others. Adding a "table" first can make it much easier to identify "forks" and "plates" later.
  • Uncertainty as a Feature: By modeling types as probabilities, the model can "change its mind." As more nodes are added, the uncertainty about specific relationships (like "holding" vs. "carrying") decreases.
  • Text as a Bridge: The integration of captions acts as a powerful regularizer, providing high-level semantic guidance when visual features are noisy or occluded.

Results and Performance

  • State-of-the-Art Performance: On the Visual Genome "Scene Graph Generation" (SG-GEN) task, DG-PGNN achieved a Recall@50 of 32.1%, significantly outperforming previous top models like VCTREE (27.9%) and MOTIF (27.2%).
  • VQA Improvements: When integrated into a VQA pipeline, the DG-PGNN-based approach outperformed specialized models like MCB and MAN on the Visual7W dataset.
  • Qualitative Success: The model demonstrated a superior ability to recognize rare or specific interactions, such as correctly identifying "zebra in snow."

Limitations and Conclusion

While powerful, the model's quality is dependent on the initial object detector (Faster R-CNN). Furthermore, sequential generation via RL is more computationally expensive than "one-shot" methods. Future work involves creating a fully end-to-end model where VQA tasks and graph construction are trained simultaneously.

Summary updated

Abstract

We propose a new algorithm, called Deep Generative Probabilistic Graph Neural Networks (DG-PGNN), to generate a scene graph for an image. The input to DG-PGNN is an image, together with a set of region-grounded captions and object bounding-box proposals for the image. To generate the scene graph, DG-PGNN constructs and updates a new model, called a Probabilistic Graph Network (PGN). A PGN can be thought of as a scene graph with uncertainty: it represents each node and each edge by a CNN feature vector and defines a probability mass function (PMF) for node-type (object category) of each node and edge-type (predicate class) of each edge. The DG-PGNN sequentially adds a new node to the current PGN by learning the optimal ordering in a Deep Q-learning framework, where states are partial PGNs, actions choose a new node, and rewards are defined based on the ground-truth. After adding a node, DG-PGNN uses message passing to update the feature vectors of the current PGN by leveraging contextual relationship information, object co-occurrences, and language priors from captions. The updated features are then used to fine-tune the PMFs. Our experiments show that the proposed algorithm significantly outperforms the state-of-the-art results on the Visual Genome dataset for scene graph generation. We also show that the scene graphs constructed by DG-PGNN improve performance on the visual question answering task, for questions that need reasoning about objects and their interactions in the scene context.
BibTeX
@inproceedings{khademi2020deep,
  author = {Mahmoud Khademi and Oliver Schulte},
  title = {{Deep Generative Probabilistic Graph Neural Networks for Scene Graph Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6783},
}
Figure