Joint learning of object graph and relation graph for visual question answering

Hao Li, Xu Li, Belhal Karimi, Jie Chen, Mingming Sun

Abstract

Visual Question Answering (VQA) requires a sophisticated understanding of both visual entities and their semantic interactions. While Scene Graphs (SGs) provide a structured framework for this task, traditional models often exhibit a bias toward object features, treating relations and attributes as secondary information. This paper introduces the Dual Message-passing enhanced Graph Neural Network (DM-GNN), a framework that treats objects and relations as equal entities. By employing a dual-graph structure—one focusing on objects and the other on relations—and utilizing a bidirectional message-passing mechanism, the model achieves significant improvements in complex reasoning, particularly for "why" and "where" questions.

Introduction

To answer natural language questions about an image, a model must move beyond simple object detection to understand the underlying scene logic. Scene Graphs (SGs) have emerged as a powerful tool for this, representing images as nodes (objects) and edges (relations). However, existing SG-based approaches frequently suffer from an "object-centric bias." These models often fail to capture the nuances of attributes (e.g., distinguishing between a monitor that is "black" versus one that is "off") or the spatial logic of relations (e.g., identifying that a man is "behind" a horse).

Researchers from Peking University and Baidu propose that robust visual reasoning requires balancing three distinct types of information: objects, relations, and attributes. The DM-GNN framework is designed to address this by elevating relations to first-class citizens within the graph architecture.

The DM-GNN Framework

The core of the DM-GNN is its dual-modality representation of the scene graph, which ensures that relational data is processed with the same complexity as object data.

1. Dual Graph Encoders

Instead of relying on a single graph where relations are merely edge labels, DM-GNN constructs two simultaneous representations:

  • Object-Significant Graph: A standard representation where nodes are objects and edges represent the relations between them.
  • Relation-Significant Graph: A "flipped" or dual view where the relations themselves are the nodes, and the edges represent the objects that connect those relations.

This dual structure forces the model to learn high-quality, high-dimensional representations for relations, preventing them from being overshadowed by object features.

2. Enhanced Message-Passing (MP) Module

To facilitate information flow across these graphs, the authors replaced standard fully-connected layers with a bidirectional Gated Recurrent Unit (GRU). This module allows for a more nuanced exchange of information between a node and its neighbors. In this setup, an object’s representation is dynamically informed by the specific relations and attributes surrounding it, and vice versa.

3. Attribute-Aware Fusion

Recognizing that fine-grained details are often lost in graph convolutions, the DM-GNN explicitly injects attribute features into the final feature map. This ensures that specific states—such as whether a light is "on" or "off"—are preserved and utilized during the final reasoning stage.

Experimental Results

The DM-GNN was evaluated on three major benchmarks: GQA, Visual Genome (VG), and motif-VG. The results demonstrate the effectiveness of the dual-graph approach in handling complex reasoning tasks.

Performance Gains

  • State-of-the-Art Results: DM-GNN achieved new SOTA performance on both the Visual Genome and GQA datasets.
  • Reasoning Accuracy: The model showed a 6.5% accuracy improvement in "why" type questions on the Visual Genome dataset compared to previous scene-graph models.
  • Open-Domain Questions: On the GQA dataset, DM-GNN outperformed baselines by 9.63% on open-domain questions, which typically require more complex compositional reasoning than binary questions.

Ablation Insights

Ablation studies highlighted the necessity of the dual structure. When the model used only an object-focused or only a relation-focused graph, accuracy hovered around 35%. However, combining them into the dual structure resulted in a jump to approximately 68%, proving that reasoning is inherently a multi-modal graph problem.

Discussion and Limitations

The success of DM-GNN suggests that symmetry between objects and relations is key to visual logic. By treating relations as nodes, the model can apply powerful GNN machinery to spatial logic in the same way it does to object identification. Furthermore, the integration of attributes as essential components rather than metadata allows the model to excel at identifying complex states, such as a car being "parked" rather than just "present."

Despite these strengths, the authors noted a limitation in handling binary (Yes/No) questions. In the GQA dataset, the model's binary scores were lower than some competitors. This is likely because binary answers are often not explicitly represented as nodes in a scene graph and require a different form of symbolic logic or global inference that the current graph-matching approach has not yet mastered.

Conclusion

DM-GNN represents a significant step forward in structured visual reasoning. By balancing the representation of objects and relations through a dual-graph architecture and enhanced message passing, the model provides a more holistic understanding of visual scenes. Future work will likely focus on better integrating symbolic logic to improve performance on binary questions and further refining the fusion of attributes within the graph structure.

Summary updated

Abstract

Modeling visual question answering(VQA) through scene graphs can significantly improve the reasoning accuracy and interpretability. However, existing models answer poorly for complex reasoning questions with attributes or relations, which causes false attribute selection or missing relation in Figure 1(a). It is because these models cannot balance all kinds of information in scene graphs, neglecting relation and attribute information. In this paper, we introduce a novel Dual Message-passing enhanced Graph Neural Network (DM-GNN), which can obtain a balanced representation by properly encoding multi-scale scene graph information. Specifically, we (i)transform the scene graph into two graphs with diversified focuses on objects and relations; Then we design a dual structure to encode them, which increases the weights from relations (ii)fuse the encoder output with attribute features, which increases the weights from attributes; (iii)propose a message-passing mechanism to enhance the information transfer between objects, relations and attributes. We conduct extensive experiments on datasets including GQA, VG, motif-VG and achieve new state of the art.
BibTeX
@inproceedings{li2022joint,
  author = {Hao Li and Xu Li and Belhal Karimi and Jie Chen and Mingming Sun},
  title = {{Joint learning of object graph and relation graph for visual question answering}},
  booktitle = {ICME},
  year = {2022},
  doi = {10.1109/icme52920.2022.9859766},
}
Figure