Abstract
The current success of modern visual reasoning systems is arguably attributed to cross-modality attention mechanisms. However, in deliberative reasoning such as in VQA, attention is unconstrained at each step, and thus may serve as a statistical pooling mechanism rather than a semantic operation intended to select information relevant to inference. This is because at training time, attention is only guided by a very sparse signal (i.e. the answer label) at the end of the inference chain. This causes the cross-modality attention weights to deviate from the desired visual-language bindings. To rectify this deviation, we propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. Here we learn the grounding from the pairing of questions and images alone, without the need for answer annotation or external grounding supervision. This grounding guides the attention mechanism inside VQA models through a duality of mechanisms: pre-training attention weight calculation and directly guiding the weights at inference time on a case-by-case basis. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process. This scalable enhancement improves the performance of VQA models, fortifies their robustness to limited access to supervised data, and increases interpretability.
Introduction
Visual Question Answering (VQA) is a hallmark task in AI that requires a model to understand both an image and a natural language question. Most modern VQA systems rely on attention mechanisms to "look" at relevant parts of the image. However, these models are typically trained using only the final answer as a signal. This "sparse signal" often leads to the "Clever Hans" effect, where models achieve correct answers through statistical coincidence or dataset biases rather than true visual-language understanding. This lack of constraint causes attention weights to deviate from desired semantic bindings.
Methodology: Grounding-based Attention Prior (GAP)
To address this, we introduce the Grounding-based Attention Prior (GAP) framework. GAP guides a model’s attention using a two-step unsupervised process that requires zero extra human labels for attention:
- Extracting the Prior: Instead of processing a question as a whole, GAP uses a constituency parse tree to break it into "Referring Expressions" (REs)—phrases like "the wooden cabinets." It then uses unsupervised contrastive learning to match these phrases to specific image regions, creating a spatial "prior" map.
- The Duality of Guidance: This prior is utilized in two ways. During pre-training, the model is trained to align its internal attention weights with these unsupervised grounding priors. At inference time, a learnable gating mechanism blends the model's predicted attention with the pre-computed prior, allowing for dynamic refinement.
Linguistic-Visual Grounding
A key insight of this work is that grounding structured "Referring Expressions" (phrases) is far more effective than grounding individual words. While a word like "right" is visually ambiguous, the phrase "the right side of the image" provides a concrete visual anchor. By leveraging linguistic structure, GAP identifies these concepts and aligns them with visual objects in an unsupervised manner.
Key Results
The researchers evaluated GAP across major benchmarks including VQA v2, GQA, and the bias-sensitive VQA-CP2.
- Performance Boost: Adding GAP to the standard "Bottom-Up Top-Down" (UpDn) model increased accuracy on VQA v2 from 63.3% to 64.3%, outperforming several complex regularization methods.
- Sample Efficiency: GAP significantly improves performance when data is scarce. When trained on only 10% of the data, a model with GAP achieved 54.2% accuracy on VQA v2, compared to 50.1% for the strongest baselines. This makes GAP a powerful tool for domains where labeled data is hard to come by.
- Interpretability: Qualitative results show that GAP-equipped models shift their focus from irrelevant background pixels to the actual objects mentioned in the question, leading to more trustworthy predictions.
Conclusion
GAP provides a scalable, unsupervised enhancement that improves the performance, robustness, and interpretability of VQA models. By teaching models to "look before they leap" based on linguistic structure, it mitigates the impact of dataset biases and improves sample efficiency, particularly in data-scarce scenarios.
Summary updated