Overcoming Language Priors for Visual Question Answering Based on Knowledge Distillation

Daowan Peng, Wei Wei

TL;DR

Visual Question Answering (VQA) models often "cheat" by memorizing linguistic shortcuts (e.g., assuming all bananas are yellow) rather than looking at the image. This paper introduces KDAR, a method that uses knowledge distillation and adaptive reweighting to penalize these language biases, achieving state-of-the-art performance on out-of-distribution benchmarks.

What Problem Does This Paper Address?

In the world of Visual Question Answering (VQA), models are notorious for being "lazy." Instead of performing complex multimodal reasoning, they often exploit language priors—statistical correlations found in the training data. For example, if a dataset contains thousands of images of "tennis" and very few of "frisbee," a model might learn to answer "tennis" whenever it sees the prompt "What sport...?" regardless of what is actually in the image.

This reliance on linguistic shortcuts leads to poor generalization. When these models encounter "out-of-distribution" (OOD) scenarios—like a red banana or a game of frisbee—they fail. The authors argue that this is essentially a fitting problem: models overfit the most frequent "head" samples and underfit the rare "tail" samples.

Methods

The authors propose KDAR (Knowledge Distillation for Addressing Prior-dependency), which consists of two primary strategies:

  1. Knowledge Distillation (KD) as Regularization: Standard training uses "one-hot" labels (where only the correct answer has a value of 1). KDAR instead uses "soft labels" from a well-trained teacher model. These soft labels provide a distribution of probabilities across many potential answers. This acts as a form of regularization, preventing the student model from becoming over-confident in the most frequent answers and providing semantic guidance (e.g., showing that "red" and "orange" are more related than "red" and "tennis").

  2. Adaptive Sample-wise Reweighting: The authors designed a dynamic loss function that adjusts the importance of each sample during training. If the student model's prediction for a sample is significantly "better" (higher probability) than the teacher's, the system suspects the student might be exploiting a shortcut. The model then reduces the weight of that sample, forcing the student to focus more on harder, rarer "tail" samples that require genuine visual understanding.

Datasets

The researchers evaluated their method on two major benchmarks:

  • VQA-CPv2 (Vision and Question Answering-Challenge Priors): The gold standard for testing OOD generalization. In this dataset, the answer distribution in the test set is intentionally different from the training set to expose language bias.
  • VQAv2: The standard "in-distribution" (IID) dataset used to ensure that fixing the bias doesn't break the model's performance on normal data.

Key Insights and Ideas

  • Bias as an Overfitting Issue: Rather than trying to "remove" language information (which can be useful), the authors treat bias as a failure to balance learning between common and rare samples.
  • Soft Labels are Semantic Maps: By using soft labels, the model learns that answers aren't just "right or wrong." It learns the relationships between categories, which helps narrow down the search space for the correct answer based on the question type.
  • Teacher-Student Synergy: The method is model-agnostic. You can take a standard VQA architecture (like UpDn or LXMERT) and apply the KDAR framework to significantly boost its robustness without changing the underlying architecture.

Key Results

The KDAR method showed impressive gains across the board:

  • New State-of-the-Art: When combined with the LXMERT backbone, KDAR achieved an overall accuracy of 71.33% on the VQA-CPv2 benchmark, a significant improvement over previous methods.
  • Balanced Improvement: Unlike many debiasing techniques that only improve "Yes/No" questions, KDAR showed strong performance on "Number" and "Other" (open-ended) questions, which are typically much harder to debias.
  • No "Debiasing Tax": Many models that perform well on OOD data perform poorly on standard IID data. KDAR maintained high performance on VQAv2 (65.54%), proving it generalizes well without sacrificing standard accuracy.

Limitations and Future Work

The authors acknowledge that their current implementation relies on a single debiased model acting as the teacher. They suggest that future research could explore using multiple teachers—perhaps one biased and one debiased—to provide a more nuanced supervisory signal. Additionally, while the method is effective, the choice of hyperparameters (like the temperature $\tau$ and the scaling factor $\beta$) requires careful tuning to achieve the best results.

Summary updated

Abstract

Previous studies have pointed out that visual question answering (VQA) models are prone to relying on language priors for answer predictions. In this context, predictions often depend on linguistic shortcuts rather than a comprehensive grasp of multimodal knowledge, which diminishes their generalization ability. In this paper, we propose a novel method, namely, KDAR, leveraging knowledge distillation to address the prior-dependency dilemmas within the VQA task. Specifically, the regularization effect facilitated by soft labels from a well-trained teacher is employed to penalize overfitting to the most common answers. The soft labels, which serve a regularization role, also provide semantic guidance that narrows the range of candidate answers. Additionally, we design an adaptive samplewise reweighting learning strategy to further mitigate bias by dynamically adjusting the importance of each sample. Experimental results demonstrate that our method enhances performance in both OOD and IID settings. Our method achieves state-of-the-art performance on the VQA-CPv2 out-of-distribution (OOD) benchmark, significantly outperforming previous state-of-the-art approaches.
BibTeX
@inproceedings{peng2024overcoming,
  author = {Daowan Peng and Wei Wei},
  title = {{Overcoming Language Priors for Visual Question Answering Based on Knowledge Distillation}},
  booktitle = {ICME},
  year = {2024},
  url = {https://openreview.net/forum?id=v4d45nUtLm},
}

Loading related content...

Figure