TL;DR
This paper tackles the "language prior problem" in Visual Question Answering (VQA), where models rely too heavily on learned language shortcuts instead of visual information. They propose a novel "adapted margin cosine loss" that specifically optimizes the feature space of answers, forcing the model to better distinguish between frequent and rare answers within specific question types. This approach significantly improves VQA model performance on challenging benchmarks designed to expose language biases.
What Problem Does This Paper Address?
Visual Question Answering (VQA) aims to answer natural language questions about images. A persistent issue in VQA is the "language prior problem," where models learn to associate certain question patterns (e.g., "how many") with specific answers (e.g., "2") from the training data, even if the image doesn't support that answer. This "shortcut learning" leads to poor generalization and a failure to truly understand the visual content. While previous work has tried to address this by balancing datasets or adding auxiliary prediction branches, this paper highlights a gap: no one has focused on how the learned feature representations of answers themselves contribute to this problem. The authors argue that by better structuring the answer feature space, they can directly combat language priors.
Methods
The core contribution of this paper is the AdaVQA (Adapted Margin Cosine Loss). The authors start by reformulating the standard softmax loss into a cosine loss by L2-normalizing both answer features and weight vectors. This converts the problem into one of maximizing the angle between the feature vector and its corresponding weight vector.
The key innovation is the introduction of an adapted margin ($m_i$) within this cosine loss. Unlike a fixed margin, this adapted margin is dynamically computed based on the frequency of an answer within a specific question type in the training set. The intuition is that frequent answers (which have more training data) should occupy a broader feature space (smaller margin), while rare answers should be confined to a tighter space (larger margin). This helps the model learn more discriminative representations for answers, especially those that are less common but visually important. The paper also introduces an "entropy threshold" to selectively apply regularization to question types that exhibit stronger language priors.
Datasets
The paper evaluates its proposed method on two benchmarks specifically designed to expose language priors:
- VQA-CP v2
- VQA-CP v1
These datasets are derived from the original VQA datasets but are re-split such that the answer distributions for specific question types differ significantly between the training and testing sets. This makes them ideal for evaluating a model's ability to overcome language shortcuts.
Key Insights and Ideas
- Focusing on Answer Feature Space is Crucial: The paper's central insight is that optimizing the learned feature representations of answers is a direct and effective way to combat language priors, an angle previously unexplored.
- Frequency Matters (Within Question Types): The adapted margin loss leverages the frequency of answers within specific question types. This is more nuanced than simply considering overall answer frequency and allows for better discrimination.
- Visualizing Embeddings Reveals the Problem: The paper effectively uses t-SNE visualizations to show how answers are intertwined in the baseline model's feature space, highlighting the need for better separation. Their method's visualizations show clearer clusters.
- Model-Agnostic Approach: The AdaVQA loss can be plugged into existing VQA models without requiring architectural changes, making it a versatile tool for improving performance.
Key Results
The proposed AdaVQA loss function, when applied to three baseline VQA models (Strong-BL, Counter, and UpDn), consistently and significantly improved performance on both VQA-CP v2 and VQA-CP v1 datasets.
- The models equipped with AdaVQA achieved an absolute performance gain of 15% on average across the baselines.
- For instance, on VQA-CP v2, the UpDn baseline achieved 39.74% accuracy, while UpDn+Ours reached 54.67%.
- On VQA-CP v1, Counter+Ours achieved 55.92%, a substantial improvement over the Counter baseline.
- Qualitative results (Figure 5) demonstrate AdaVQA's ability to correct incorrect predictions driven by language priors, showing more relevant attention maps and correct answers.
Limitations and Future Work
- Not a SOTA Model: The authors explicitly state that their work is not aimed at achieving state-of-the-art results with a novel architecture, but rather at providing a new loss function to enhance existing models.
- "Other" Category Performance: The paper notes that their method doesn't always outperform other specialized methods on the "Other" answer category, which they attribute to the inherent diversity of answers in such questions, making them less susceptible to language priors in the first place.
- Future Work: The authors hope that their AdaVQA loss will enable future research to focus more on enhancing visual reasoning capabilities without being overly influenced by language priors. They also suggest further exploration into the optimal entropy threshold and scaling factor parameters.
Summary updated