Bilinear Graph Networks for Visual Question Answering

Dalu Guo, Chang Xu, Dacheng TaoIEEE TNNLS 2023 · OpenReview

TL;DR

This paper introduces Bilinear Graph Networks (BGNs) to improve visual question answering (VQA) by explicitly modeling relationships between image objects and question words using graph structures. Their novel approach, combining image- and question-focused graphs, achieves state-of-the-art performance on the VQA v2.0 dataset, demonstrating enhanced reasoning capabilities for complex questions.

What Problem Does This Paper Address?

Visual Question Answering (VQA) is a challenging task that requires machines to understand both an image and a natural language question to provide an accurate answer. While existing methods like Bilinear Attention Networks (BAN) attempt to model interactions between image objects and question words, they often lack a comprehensive exploitation of the contextual relationships within these modalities, particularly for complex, multi-step reasoning questions. The paper argues that previous approaches, while building attention maps, fail to fully explore the intricate dependencies between words and objects, leading to limitations in handling nuanced questions.

Methods

The core contribution of this paper is the development of Bilinear Graph Networks (BGNs). BGNs are built upon the idea of representing the VQA problem as a graph, where nodes are either image objects or question words, and edges represent their relationships. The proposed architecture consists of two types of graphs that work cooperatively:

  • Image-Graph: This graph focuses on transferring features from detected image objects to related query words in the question. It aims to create joint embeddings that capture both semantic and factual information. This is achieved through a multi-glimpse attention mechanism, similar to BAN, but with an emphasis on building a richer representation.
  • Question-Graph: This graph takes the joint embeddings generated by the image-graph and exchanges information between them. It amplifies the implicit relationships between objects and words, enabling more complex reasoning.

These two graphs are stacked in multiple layers, allowing for iterative refinement of the understanding. The authors also highlight their reformulation of BAN as a bilinear graph, identifying its limitations in capturing inter-node relationships and motivating their proposed graph-based approach.

Datasets

The paper uses the VQA v2.0 dataset for training and evaluation. This dataset is known for its emphasis on visual understanding and reduction of text bias, making it a challenging benchmark for VQA models. They also mention using data from Visual Genome for additional training to improve model performance.

Key Insights and Ideas

  • Graph-based reasoning is crucial for complex VQA: The paper strongly advocates for explicit graph modeling to capture the intricate relationships between visual and textual information, which is essential for multi-step reasoning.
  • Separating image-object and word-word relationships is beneficial: The introduction of distinct image-graphs and question-graphs allows for specialized modeling of different types of dependencies, leading to improved performance.
  • Iterative refinement through stacked layers: Stacking multiple layers of image- and question-graphs allows the model to progressively refine its understanding and tackle more complex questions.
  • BERT compatibility: The proposed BGNs are shown to be compatible with pre-trained language models like BERT, further boosting performance and demonstrating the model's flexibility.

Key Results

  • State-of-the-art performance: The proposed BGNs achieve state-of-the-art accuracy on the VQA v2.0 test-std dataset, reaching 72.41%.
  • Significant improvements over BAN: On the VQA v2.0 validation set, their one-layer graph networks boost accuracy by 0.62% compared to BAN. Stacking multiple layers shows further advantages, with a total improvement of 1.4% for multi-step reasoning.
  • BERT integration yields further gains: With the help of pre-trained BERT, their models achieve an additional 1.5% increase in accuracy.
  • Improved performance on long questions: Ablation studies show that their multi-layer models significantly outperform single-layer BAN, especially on longer questions, indicating better handling of complex reasoning.

Limitations and Future Work

  • Performance drop at four layers: The authors note an interesting observation where performance drops at the four-layer model, suggesting that different question complexities might benefit from different graph depths. This inspires future work in classifying questions to match them with appropriate graph depths.
  • Future work on question classification: The observation about performance varying with the number of layers suggests a potential direction for future research in designing networks that can dynamically adapt their depth based on question complexity.

Summary updated

BibTeX
@article{guo2023bilinear,
  author = {Dalu Guo and Chang Xu and Dacheng Tao},
  title = {{Bilinear Graph Networks for Visual Question Answering}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=LOOBEVEVMS},
}
Figure