Abstract
Modern Visual Question Answering (VQA) models have been shown to rely heavily on superficial correlations between question and answer words learned during training—such as overwhelmingly reporting the type of room as "kitchen" or the sport being played as "tennis," irrespective of the image. Most alarmingly, this shortcoming is often not well reflected during evaluation because the same strong priors exist in test distributions; however, a VQA system that fails to ground questions in image content would likely perform poorly in real-world settings. In this work, we present a novel regularization scheme for VQA that reduces this effect. We introduce a question-only model that takes as input the question encoding from the VQA model and must leverage language biases in order to succeed. We then pose training as an adversarial game between the VQA model and this question-only adversary—discouraging the VQA model from capturing language biases in its question encoding. Further, we leverage this question-only model to estimate the increase in model confidence after considering the image, which we maximize explicitly to encourage visual grounding. Our approach is a model-agnostic training procedure and simple to implement. We show empirically that it can improve performance significantly on a bias-sensitive split of the VQA dataset for multiple base models—achieving state-of-the-art on this task. Further, on standard VQA tasks, our approach shows significantly less drop in accuracy compared to existing bias-reducing VQA models.
Introduction
The core challenge in VQA is "grounding"—ensuring the model's answer is derived from the visual content provided. However, current datasets contain strong "language priors." For instance, if 90% of questions about banana color in a training set are answered with "yellow," a model can achieve high accuracy by simply ignoring the image and guessing "yellow" every time. Standard evaluation metrics often fail to catch this because test sets usually share the same biases. When these models encounter real-world scenarios—like a green banana—they fail. This research aims to break this over-reliance on language shortcuts without sacrificing the model's fundamental ability to understand questions.
Methodology
The proposed model-agnostic training procedure consists of two main components designed to enforce visual grounding:
- Question-Only Adversary (Q-Adv): A small adversary network is attached to the base VQA model. This adversary receives only the question encoding and attempts to predict the correct answer. Training is framed as an adversarial game: the adversary tries to succeed, while the base VQA model is penalized if the adversary is successful. This forces the question encoder to strip away "answer-discriminative" features that do not require an image to decode.
- Difference of Entropies (DoE) Regularizer: To prevent the model from losing critical question context under adversarial pressure, the authors introduce a second term. This regularizer maximizes the difference between the uncertainty (entropy) of the "blind" question-only model and the "sighted" VQA model. It rewards the model for becoming significantly more confident in its answer after processing the image.
Experimental Setup
The approach was evaluated on several key benchmarks:
- VQA-CP (Visual Question Answering under Changing Priors): A diagnostic split where answer distributions in the test set intentionally differ from the training set. This serves as the primary stress test for language bias.
- VQA v1 and v2: Standard industry benchmarks used to ensure the regularization does not degrade performance on balanced data.
Results and Discussion
The adversarial regularization scheme achieved state-of-the-art results on bias-sensitive benchmarks while maintaining competitive performance on standard sets.
Key Findings
- Performance Gains: On the VQA-CP v2 benchmark, the method improved the Stacked Attention Network (SAN) from 24.96% to 33.29% and the Bottom-Up Top-Down (UpDn) model from 39.74% to 41.17%.
- Superiority over Specialized Architectures: The approach outperformed GVQA, a complex model specifically hand-designed for bias resistance, by nearly 10% in some metrics, despite being much simpler to implement.
- Minimal "Bias Tax": Unlike previous bias-reduction methods that caused massive accuracy drops on standard datasets (VQA v2), this method showed a negligible drop (only ~0.1% for SAN), making it viable for general-purpose applications.
- Information Gain as a Signal: The results demonstrate that the Q-Adv and DoE terms work synergistically. Q-Adv reduces bias, while DoE ensures the model retains enough question context to provide a relevant, grounded answer.
Conclusion
By framing VQA training as an adversarial game and explicitly rewarding information gain from visual content, this work provides a robust framework for reducing language bias. While the method requires careful hyperparameter tuning for different base architectures, it represents a significant step toward VQA systems that truly "see" the world rather than just memorizing its statistics.
Summary updated