Overcoming Language Priors in Visual Question Answering with Adversarial Regularization

Sainandan Ramakrishnan, Aishwarya Agrawal, Stefan Lee

Abstract

Modern Visual Question Answering (VQA) models have been shown to rely heavily on superficial correlations between question and answer words learned during training—such as overwhelmingly reporting the type of room as "kitchen" or the sport being played as "tennis," irrespective of the image. Most alarmingly, this shortcoming is often not well reflected during evaluation because the same strong priors exist in test distributions; however, a VQA system that fails to ground questions in image content would likely perform poorly in real-world settings. In this work, we present a novel regularization scheme for VQA that reduces this effect. We introduce a question-only model that takes as input the question encoding from the VQA model and must leverage language biases in order to succeed. We then pose training as an adversarial game between the VQA model and this question-only adversary—discouraging the VQA model from capturing language biases in its question encoding. Further, we leverage this question-only model to estimate the increase in model confidence after considering the image, which we maximize explicitly to encourage visual grounding. Our approach is a model-agnostic training procedure and simple to implement. We show empirically that it can improve performance significantly on a bias-sensitive split of the VQA dataset for multiple base models—achieving state-of-the-art on this task. Further, on standard VQA tasks, our approach shows significantly less drop in accuracy compared to existing bias-reducing VQA models.

Introduction

The core challenge in VQA is "grounding"—ensuring the model's answer is derived from the visual content provided. However, current datasets contain strong "language priors." For instance, if 90% of questions about banana color in a training set are answered with "yellow," a model can achieve high accuracy by simply ignoring the image and guessing "yellow" every time. Standard evaluation metrics often fail to catch this because test sets usually share the same biases. When these models encounter real-world scenarios—like a green banana—they fail. This research aims to break this over-reliance on language shortcuts without sacrificing the model's fundamental ability to understand questions.

Methodology

The proposed model-agnostic training procedure consists of two main components designed to enforce visual grounding:

  1. Question-Only Adversary (Q-Adv): A small adversary network is attached to the base VQA model. This adversary receives only the question encoding and attempts to predict the correct answer. Training is framed as an adversarial game: the adversary tries to succeed, while the base VQA model is penalized if the adversary is successful. This forces the question encoder to strip away "answer-discriminative" features that do not require an image to decode.
  2. Difference of Entropies (DoE) Regularizer: To prevent the model from losing critical question context under adversarial pressure, the authors introduce a second term. This regularizer maximizes the difference between the uncertainty (entropy) of the "blind" question-only model and the "sighted" VQA model. It rewards the model for becoming significantly more confident in its answer after processing the image.
The proposed architecture featuring the Base VQA Model, Question-Only Adversary (B), and Difference of Entropies component (C).

Experimental Setup

The approach was evaluated on several key benchmarks:

  • VQA-CP (Visual Question Answering under Changing Priors): A diagnostic split where answer distributions in the test set intentionally differ from the training set. This serves as the primary stress test for language bias.
  • VQA v1 and v2: Standard industry benchmarks used to ensure the regularization does not degrade performance on balanced data.

Results and Discussion

The adversarial regularization scheme achieved state-of-the-art results on bias-sensitive benchmarks while maintaining competitive performance on standard sets.

Performance comparison on VQA-CP v2 and VQA v2 datasets across different base models and regularization components.

Key Findings

  • Performance Gains: On the VQA-CP v2 benchmark, the method improved the Stacked Attention Network (SAN) from 24.96% to 33.29% and the Bottom-Up Top-Down (UpDn) model from 39.74% to 41.17%.
  • Superiority over Specialized Architectures: The approach outperformed GVQA, a complex model specifically hand-designed for bias resistance, by nearly 10% in some metrics, despite being much simpler to implement.
  • Minimal "Bias Tax": Unlike previous bias-reduction methods that caused massive accuracy drops on standard datasets (VQA v2), this method showed a negligible drop (only ~0.1% for SAN), making it viable for general-purpose applications.
  • Information Gain as a Signal: The results demonstrate that the Q-Adv and DoE terms work synergistically. Q-Adv reduces bias, while DoE ensures the model retains enough question context to provide a relevant, grounded answer.

Conclusion

By framing VQA training as an adversarial game and explicitly rewarding information gain from visual content, this work provides a robust framework for reducing language bias. While the method requires careful hyperparameter tuning for different base architectures, it represents a significant step toward VQA systems that truly "see" the world rather than just memorizing its statistics.

Summary updated

Abstract

Modern Visual Question Answering (VQA) models have been shown to rely heavily on superficial correlations between question and answer words learned during training such as overwhelmingly reporting the type of room as kitchen or the sport being played as tennis, irrespective of the image. Most alarmingly, this shortcoming is often not well reflected during evaluation because the same strong priors exist in test distributions; however, a VQA system that fails to ground questions in image content would likely perform poorly in real-world settings. In this work, we present a novel regularization scheme for VQA that reduces this effect. We introduce a question-only model that takes as input the question encoding from the VQA model and must leverage language biases in order to succeed. We then pose training as an adversarial game between the VQA model and this question-only adversary -- discouraging the VQA model from capturing language biases in its question encoding. Further,we leverage this question-only model to estimate the increase in model confidence after considering the image, which we maximize explicitly to encourage visual grounding. Our approach is a model agnostic training procedure and simple to implement. We show empirically that it can improve performance significantly on a bias-sensitive split of the VQA dataset for multiple base models -- achieving state-of-the-art on this task. Further, on standard VQA tasks, our approach shows significantly less drop in accuracy compared to existing bias-reducing VQA models.
BibTeX
@inproceedings{ramakrishnan2018overcoming,
  author = {Sainandan Ramakrishnan and Aishwarya Agrawal and Stefan Lee},
  title = {{Overcoming Language Priors in Visual Question Answering with Adversarial Regularization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=H1bRzP-_-r},
}

Loading related content...

Figure