From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering

Zihao Zhu

TL;DR

This paper introduces the Hierarchical Graph Neural Module Network (HGNMN), a novel approach to Visual Question Answering (VQA) that tackles "deep" questions requiring multi-step reasoning. By representing images as multi-layer graphs (visual, semantic, commonsense) and employing a suite of specialized neural modules, HGNMN achieves state-of-the-art performance while offering explicit and interpretable reasoning processes.

What Problem Does This Paper Address?

Traditional VQA systems often struggle with questions that go beyond simple object recognition. These "deep" questions demand a more sophisticated understanding, requiring the model to not only perceive visual content but also to integrate semantic information and external commonsense knowledge. Furthermore, understanding how a model arrives at an answer is crucial for trust and debugging, a challenge that many existing VQA models, which often perform reasoning implicitly, fail to address. The authors aim to bridge this gap by developing a VQA system capable of explicit, compositional reasoning over multiple knowledge sources.

Methods

The core of the HGNMN is its multi-layer graph representation of an image. Instead of a single graph, the system constructs three distinct graphs:

  • Visual Graph: Captures objects and their spatial relationships, derived from object detection.
  • Semantic Graph: Represents high-level object abstractions and relationships, generated from image captions.
  • Commonsense Graph: Incorporates external knowledge about objects and their typical uses, extracted from ConceptNet.

These graphs are then processed by a set of specialized neural modules designed for specific reasoning tasks:

  • Find: Attends to relevant objects based on a query.
  • And: Combines attention maps from previous steps.
  • Filter: Narrows down object relevance based on new information.
  • Relate: Transfers attention along graph edges to capture object interactions.
  • CrossGraph: Enables reasoning between different graphs, crucial for integrating diverse knowledge.
  • Describe: Summarizes graph features into an embedding for answer prediction.
  • NoOp: A placeholder for padding reasoning steps.

A Module Controller dynamically guides the execution of these modules at each reasoning step. It predicts soft module weights and generates input queries, allowing the entire network to be trained end-to-end using gradient descent without requiring explicit "expert layouts" of reasoning steps. This controller ensures that the model can adapt its reasoning strategy based on the question.

Datasets

The paper evaluates its model on the CRIC dataset. This dataset is highlighted as being particularly suitable for testing deeper VQA questions, containing over 1.3 million such queries. It also provides expert layout annotations, which the authors note their model does not require for training, differentiating it from some prior work.

Key Insights and Ideas

  • Multi-layer graphs are essential for deep VQA: The paper strongly suggests that relying on a single modality (visual, semantic, or commonsense) is insufficient for complex reasoning. Integrating these perspectives into distinct but interconnected graphs allows for a more comprehensive understanding.
  • Explicit reasoning is achievable and valuable: By designing specialized modules and a controller that dictates their use, the HGNMN can trace its reasoning steps. This interpretability is a significant advantage over implicit attention mechanisms.
  • Cross-graph reasoning is a critical component: The CrossGraph module is highlighted as particularly important, enabling the model to bridge information across the visual, semantic, and commonsense graphs, which is vital for answering questions that require combining different types of knowledge.
  • End-to-end training without expert layouts: The module controller's ability to learn soft module selection and query generation allows for a more flexible and less labor-intensive training process compared to methods relying on predefined reasoning chains.

Key Results

The HGNMN achieves a new state-of-the-art (SOTA) result on the CRIC dataset, reaching an accuracy of 60.32%. This outperforms previous methods, including RVC (58.38%), by a notable margin. Ablation studies confirm the importance of each graph layer, with the commonsense graph proving most critical. The Relate and CrossGraph modules are also shown to be vital for multi-step reasoning. Qualitative evaluations demonstrate the model's ability to explicitly trace its reasoning, identifying specific objects, relationships, and knowledge used at each step.

Limitations and Future Work

The paper acknowledges that while their model achieves SOTA, there is always room for improvement in VQA accuracy. The authors suggest that future work could explore more sophisticated graph construction techniques or further refine the neural modules for even more nuanced reasoning. The interpretability of the model is a key strength, and future research could focus on further enhancing this aspect or applying similar graph-based reasoning frameworks to other complex AI tasks.

Summary updated

Abstract

In order to achieve a general visual question answering (VQA) system, it is essential to learn to answer deeper questions that require compositional reasoning on the image and external knowledge. Meanwhile, the reasoning process should be explicit and explainable to understand the working mechanism of the model. It is effortless for human but challenging for machines. In this paper, we propose a Hierarchical Graph Neural Module Network (HGNMN) that reasons over multi-layer graphs with neural modules to address the above issues. Specifically, we first encode the image by multi-layer graphs from the visual, semantic and commonsense views since the clues that support the answer may exist in different modalities. Our model consists of several well-dasigned neural modules that perform specific functions over graphs, which can be used to conduct multi-step reasoning within and between different graphs. Compared to existing modular networks, we extend visual reasoning from one graph to more graphs. We can explicitly trace the reasoning process according to module weights and graph attentions. Experiments show that our model not only achieves state-of-the-art performance on the CRIC dataset but also obtains explicit and explainable reasoning procedures.
BibTeX
@inproceedings{zhu2022from,
  author = {Zihao Zhu},
  title = {{From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=9k1LUGgbrH},
}
Figure