Abstract
The current success of modern visual reasoning systems is arguably attributed to cross-modality attention mechanisms. However, in deliberative reasoning such as in VQA, attention is unconstrained at each step, and thus may serve as a statistical pooling mechanism rather than a semantic operation intended to select information relevant to inference. This is because at training time, attention is only guided by a very sparse signal (i.e. the answer label) at the end of the inference chain. This causes the cross-modality attention weights to deviate from the desired visual-language bindings. To rectify this deviation, we propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. Here we learn the grounding from the pairing of questions and images alone, without the need for answer annotation or external grounding supervision. This grounding guides the attention mechanism inside VQA models through a duality of mechanisms: pre-training attention weight calculation and directly guiding the weights at inference time on a case-by-case basis. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process. This scalable enhancement improves the performance of VQA models, fortifies their robustness to limited access to supervised data, and increases interpretability.
Introduction
Modern Visual Question Answering (VQA) systems rely heavily on attention mechanisms to weigh specific parts of an image based on the query. However, these attention weights are often unconstrained, supervised only by the final answer label. This sparse signal can lead the model to act as a statistical pooling tool rather than performing true semantic selection. For instance, a model might correctly answer a question about a red ball by looking at unrelated green grass, succeeding only by chance. Such "noisy" attention makes models brittle and data-hungry.
To address this, the Grounding-based Attention Prior (GAP) framework is introduced. GAP teaches models "where to look" using unsupervised linguistic cues, aligning structured linguistic concepts in the query to their visual referents without requiring expensive human-annotated attention maps.
Methodology: The GAP Framework
The GAP framework operates in two primary stages: unsupervised grounding and dual-pathway guidance.
Unsupervised Grounding
Instead of aligning individual words, GAP utilizes a constituency parse tree to decompose questions into "Referring Expressions" (REs)—complex noun phrases like "the wooden cabinets." It then employs contrastive learning (specifically the InfoNCE loss) to align these phrases with image regions. By observing frequent co-occurrences across a dataset, the model learns to associate specific phrases with their corresponding visual objects. This phrase-level grounding is more effective than word-level grounding, as phrases provide the necessary context to disambiguate visual referents.
Dual-Pathway Guidance
Once these grounding priors are established, they guide the VQA model through two mechanisms:
- Pre-training: The VQA model’s attention layers are trained to mimic the unsupervised grounding maps before being exposed to answer labels.
- Inference Gating: During runtime, a learnable gating mechanism blends the model's calculated attention with the grounding prior. This acts as a "sanity check," ensuring the model maintains focus on relevant objects even if its internal logic becomes confused.
Experimental Evaluation
The researchers evaluated GAP on two major benchmarks: VQA v2, the standard large-scale open-ended dataset, and GQA, which focuses on compositional visual reasoning. Additionally, Flickr30K Entities was used to validate the quality of the unsupervised grounding against ground-truth object boxes.
Key Results and Insights
GAP consistently improved the performance of various baseline architectures, including UpDn, MACNet, and BAN. On VQA v2, the UpDn model's accuracy increased from 63.3% to 64.3%. The gains were even more significant on the GQA dataset, which demands more complex reasoning.
A standout finding is GAP's extreme sample efficiency. A model utilizing GAP with only 10% of the training data outperformed a standard baseline trained on 100% of the data. This suggests that providing a semantic "prior" for attention significantly accelerates the learning process. Furthermore, GAP's unsupervised grounding proved more effective for VQA than state-of-the-art supervised grounding models, likely because it captures domain-specific nuances of the target VQA dataset.
Conclusion and Future Work
The GAP framework demonstrates that unsupervised linguistic-visual grounding can effectively regularize attention in VQA models, leading to better performance, robustness, and interpretability. While the current study focused on RNN and CNN-based architectures, the principles of GAP are theoretically applicable to Transformers. Future research could explore integrating these priors into the cross-attention layers of large-scale Vision-Language Transformers to enhance convergence and out-of-distribution generalization.
Summary updated