Guiding Visual Question Answering with Attention Priors

Thao Minh Le, Vuong Le, Sunil Gupta, Svetha Venkatesh, Truyen Tran

Abstract

The current success of modern visual reasoning systems is arguably attributed to cross-modality attention mechanisms. However, in deliberative reasoning such as in VQA, attention is unconstrained at each step, and thus may serve as a statistical pooling mechanism rather than a semantic operation intended to select information relevant to inference. This is because at training time, attention is only guided by a very sparse signal (i.e. the answer label) at the end of the inference chain. This causes the cross-modality attention weights to deviate from the desired visual-language bindings. To rectify this deviation, we propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. Here we learn the grounding from the pairing of questions and images alone, without the need for answer annotation or external grounding supervision. This grounding guides the attention mechanism inside VQA models through a duality of mechanisms: pre-training attention weight calculation and directly guiding the weights at inference time on a case-by-case basis. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process. This scalable enhancement improves the performance of VQA models, fortifies their robustness to limited access to supervised data, and increases interpretability.

Introduction

Modern Visual Question Answering (VQA) systems rely heavily on attention mechanisms to weigh specific parts of an image based on the query. However, these attention weights are often unconstrained, supervised only by the final answer label. This sparse signal can lead the model to act as a statistical pooling tool rather than performing true semantic selection. For instance, a model might correctly answer a question about a red ball by looking at unrelated green grass, succeeding only by chance. Such "noisy" attention makes models brittle and data-hungry.

To address this, the Grounding-based Attention Prior (GAP) framework is introduced. GAP teaches models "where to look" using unsupervised linguistic cues, aligning structured linguistic concepts in the query to their visual referents without requiring expensive human-annotated attention maps.

Methodology: The GAP Framework

The GAP framework operates in two primary stages: unsupervised grounding and dual-pathway guidance.

Unsupervised Grounding

Instead of aligning individual words, GAP utilizes a constituency parse tree to decompose questions into "Referring Expressions" (REs)—complex noun phrases like "the wooden cabinets." It then employs contrastive learning (specifically the InfoNCE loss) to align these phrases with image regions. By observing frequent co-occurrences across a dataset, the model learns to associate specific phrases with their corresponding visual objects. This phrase-level grounding is more effective than word-level grounding, as phrases provide the necessary context to disambiguate visual referents.

Dual-Pathway Guidance

Once these grounding priors are established, they guide the VQA model through two mechanisms:

  1. Pre-training: The VQA model’s attention layers are trained to mimic the unsupervised grounding maps before being exposed to answer labels.
  2. Inference Gating: During runtime, a learnable gating mechanism blends the model's calculated attention with the grounding prior. This acts as a "sanity check," ensuring the model maintains focus on relevant objects even if its internal logic becomes confused.
Overview of the GAP framework architecture, showing the integration of pre-computed grounding and the gating mechanism.

Experimental Evaluation

The researchers evaluated GAP on two major benchmarks: VQA v2, the standard large-scale open-ended dataset, and GQA, which focuses on compositional visual reasoning. Additionally, Flickr30K Entities was used to validate the quality of the unsupervised grounding against ground-truth object boxes.

Key Results and Insights

GAP consistently improved the performance of various baseline architectures, including UpDn, MACNet, and BAN. On VQA v2, the UpDn model's accuracy increased from 63.3% to 64.3%. The gains were even more significant on the GQA dataset, which demands more complex reasoning.

Experimental results on VQA v2 and GQA datasets, demonstrating consistent accuracy improvements across different baseline models when enhanced with GAP.

A standout finding is GAP's extreme sample efficiency. A model utilizing GAP with only 10% of the training data outperformed a standard baseline trained on 100% of the data. This suggests that providing a semantic "prior" for attention significantly accelerates the learning process. Furthermore, GAP's unsupervised grounding proved more effective for VQA than state-of-the-art supervised grounding models, likely because it captures domain-specific nuances of the target VQA dataset.

Conclusion and Future Work

The GAP framework demonstrates that unsupervised linguistic-visual grounding can effectively regularize attention in VQA models, leading to better performance, robustness, and interpretability. While the current study focused on RNN and CNN-based architectures, the principles of GAP are theoretically applicable to Transformers. Future research could explore integrating these priors into the cross-attention layers of large-scale Vision-Language Transformers to enhance convergence and out-of-distribution generalization.

Summary updated

Abstract

The current success of modern visual reasoning systems is arguably attributed to cross-modality attention mechanisms. However, in deliberative reasoning such as in VQA, attention is unconstrained at each step, and thus may serve as a statistical pooling mechanism rather than a semantic operation intended to select information relevant to inference. This is because at training time, attention is only guided by a very sparse signal (i.e. the answer label) at the end of the inference chain. This causes the cross-modality attention weights to deviate from the desired visual-language bindings. To rectify this deviation, we propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. Here we learn the grounding from the pairing of questions and images alone, without the need for answer annotation or external grounding supervision. This grounding guides the attention mechanism inside VQA models through a duality of mechanisms: pre-training attention weight calculation and directly guiding the weights at inference time on a case- by-case basis. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process. This scalable enhancement improves the performance of VQA models, fortifies their robustness to limited access to supervised data, and increases interpretability.
BibTeX
@inproceedings{le2023guiding,
  author = {Thao Minh Le and Vuong Le and Sunil Gupta and Svetha Venkatesh and Truyen Tran},
  title = {{Guiding Visual Question Answering with Attention Priors}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=8uJu3VE420V},
}
Figure