AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss

Yangyang Guo, Liqiang Nie, Zhiyong Cheng, Feng Ji, Ji Zhang, Alberto Del Bimbo

TL;DR

This paper tackles the "language prior problem" in Visual Question Answering (VQA), where models rely too heavily on learned language shortcuts instead of visual information. They propose a novel "adapted margin cosine loss" that specifically optimizes the feature space of answers, forcing the model to better distinguish between frequent and rare answers within specific question types. This approach significantly improves VQA model performance on challenging benchmarks designed to expose language biases.

What Problem Does This Paper Address?

Visual Question Answering (VQA) aims to answer natural language questions about images. A persistent issue in VQA is the "language prior problem," where models learn to associate certain question patterns (e.g., "how many") with specific answers (e.g., "2") from the training data, even if the image doesn't support that answer. This "shortcut learning" leads to poor generalization and a failure to truly understand the visual content. While previous work has tried to address this by balancing datasets or adding auxiliary prediction branches, this paper highlights a gap: no one has focused on how the learned feature representations of answers themselves contribute to this problem. The authors argue that by better structuring the answer feature space, they can directly combat language priors.

Methods

The core contribution of this paper is the AdaVQA (Adapted Margin Cosine Loss). The authors start by reformulating the standard softmax loss into a cosine loss by L2-normalizing both answer features and weight vectors. This converts the problem into one of maximizing the angle between the feature vector and its corresponding weight vector.

The key innovation is the introduction of an adapted margin ($m_i$) within this cosine loss. Unlike a fixed margin, this adapted margin is dynamically computed based on the frequency of an answer within a specific question type in the training set. The intuition is that frequent answers (which have more training data) should occupy a broader feature space (smaller margin), while rare answers should be confined to a tighter space (larger margin). This helps the model learn more discriminative representations for answers, especially those that are less common but visually important. The paper also introduces an "entropy threshold" to selectively apply regularization to question types that exhibit stronger language priors.

Datasets

The paper evaluates its proposed method on two benchmarks specifically designed to expose language priors:

  • VQA-CP v2
  • VQA-CP v1

These datasets are derived from the original VQA datasets but are re-split such that the answer distributions for specific question types differ significantly between the training and testing sets. This makes them ideal for evaluating a model's ability to overcome language shortcuts.

Key Insights and Ideas

  • Focusing on Answer Feature Space is Crucial: The paper's central insight is that optimizing the learned feature representations of answers is a direct and effective way to combat language priors, an angle previously unexplored.
  • Frequency Matters (Within Question Types): The adapted margin loss leverages the frequency of answers within specific question types. This is more nuanced than simply considering overall answer frequency and allows for better discrimination.
  • Visualizing Embeddings Reveals the Problem: The paper effectively uses t-SNE visualizations to show how answers are intertwined in the baseline model's feature space, highlighting the need for better separation. Their method's visualizations show clearer clusters.
  • Model-Agnostic Approach: The AdaVQA loss can be plugged into existing VQA models without requiring architectural changes, making it a versatile tool for improving performance.

Key Results

The proposed AdaVQA loss function, when applied to three baseline VQA models (Strong-BL, Counter, and UpDn), consistently and significantly improved performance on both VQA-CP v2 and VQA-CP v1 datasets.

  • The models equipped with AdaVQA achieved an absolute performance gain of 15% on average across the baselines.
  • For instance, on VQA-CP v2, the UpDn baseline achieved 39.74% accuracy, while UpDn+Ours reached 54.67%.
  • On VQA-CP v1, Counter+Ours achieved 55.92%, a substantial improvement over the Counter baseline.
  • Qualitative results (Figure 5) demonstrate AdaVQA's ability to correct incorrect predictions driven by language priors, showing more relevant attention maps and correct answers.

Limitations and Future Work

  • Not a SOTA Model: The authors explicitly state that their work is not aimed at achieving state-of-the-art results with a novel architecture, but rather at providing a new loss function to enhance existing models.
  • "Other" Category Performance: The paper notes that their method doesn't always outperform other specialized methods on the "Other" answer category, which they attribute to the inherent diversity of answers in such questions, making them less susceptible to language priors in the first place.
  • Future Work: The authors hope that their AdaVQA loss will enable future research to focus more on enhancing visual reasoning capabilities without being overly influenced by language priors. They also suggest further exploration into the optimal entropy threshold and scaling factor parameters.

Summary updated

Abstract

A number of studies point out that current Visual Question Answering (VQA) models are severely affected by the language prior problem, which refers to blindly making predictions based on the language shortcut. Some efforts have been devoted to overcoming this issue with delicate models. However, there is no research to address it from the view of the answer feature space learning, despite the fact that existing VQA methods all cast VQA as a classification task. Inspired by this, in this work, we attempt to tackle the language prior problem from the viewpoint of the feature space learning. An adapted margin cosine loss is designed to discriminate the frequent and the sparse answer feature space under each question type properly. In this way, the limited patterns within the language modality can be largely reduced to eliminate the language priors. We apply this loss function to several baseline models and evaluate its effectiveness on two VQA-CP benchmarks. Experimental results demonstrate that our proposed adapted margin cosine loss can enhance the baseline models with an absolute performance gain of 15\% on average, strongly verifying the potential of tackling the language prior problem in VQA from the angle of the answer feature space learning.
BibTeX
@inproceedings{guo2021adavqa,
  author = {Yangyang Guo and Liqiang Nie and Zhiyong Cheng and Feng Ji and Ji Zhang and Alberto Del Bimbo},
  title = {{AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/98},
}
Figure