Overcoming Language Priors for Visual Question Answering Based on Knowledge Distillation

Daowan Peng, Wei Wei

Abstract

Previous studies have pointed out that visual question answering (VQA) models are prone to relying on language priors for answer predictions. In this context, predictions often depend on linguistic shortcuts rather than a comprehensive grasp of multimodal knowledge, which diminishes their generalization ability. In this paper, we propose a novel method, namely, KDAR, leveraging knowledge distillation to address the prior-dependency dilemmas within the VQA task. Specifically, the regularization effect facilitated by soft labels from a well-trained teacher is employed to penalize overfitting to the most common answers. The soft labels, which serve a regularization role, also provide semantic guidance that narrows the range of candidate answers. Additionally, we design an adaptive sample-wise reweighting learning strategy to further mitigate bias by dynamically adjusting the importance of each sample. Experimental results demonstrate that our method enhances performance in both OOD and IID settings. Our method achieves state-of-the-art performance on the VQA-CPv2 out-of-distribution (OOD) benchmark, significantly outperforming previous state-of-the-art approaches.

Introduction

Visual Question Answering (VQA) models often struggle with "language priors"—statistical correlations in training data that allow models to guess answers without truly analyzing the image. For instance, if a dataset frequently pairs "banana" with "yellow," a model might ignore visual evidence of a red banana. This reliance on linguistic shortcuts leads to poor generalization in out-of-distribution (OOD) scenarios where these priors no longer hold. We argue that this is fundamentally an overfitting problem where models prioritize frequent "head" samples over rare "tail" samples.

The KDAR Framework

To address these challenges, we introduce KDAR (Knowledge Distillation for Addressing Prior-dependency). Our approach consists of two core components designed to move beyond simple one-hot label training.

1. Knowledge Distillation as Regularization

Instead of training with hard "one-hot" labels, KDAR utilizes "soft labels" generated by a well-trained teacher model. These soft labels provide a probability distribution across multiple potential answers, acting as a regularizer that prevents the student model from becoming over-confident in the most frequent answers. Furthermore, these labels provide semantic guidance; for example, they can signal that "red" and "orange" are more related than "red" and "tennis," helping the model narrow the search space logically based on the question type.

The KDAR framework utilizes knowledge distillation from a teacher model (T-VQA) to guide a student model (S-VQA) using soft labels and an adaptive loss function.

2. Adaptive Sample-wise Reweighting

We implement a dynamic loss function that adjusts the importance of each sample during training. If the student model's prediction for a specific sample is significantly more confident than the teacher's, the system identifies a potential shortcut exploitation. The model then reduces the weight of that sample, forcing the student to focus on harder, rarer samples that require genuine multimodal reasoning.

Experimental Results

We evaluated KDAR on two primary benchmarks: VQA-CPv2, the standard for testing OOD generalization where answer distributions differ between training and testing, and VQAv2, used to ensure performance is maintained on in-distribution (IID) data.

Performance on VQA-CPv2

KDAR achieved state-of-the-art (SOTA) performance on the VQA-CPv2 benchmark. Notably, it showed balanced improvements across all question types, including "Number" and "Other" (open-ended) questions, which are historically difficult to debias.

Performance comparison on the VQA-CPv2 test set. KDAR (Ours) achieves state-of-the-art results across all question categories.

Generalization and the "Debiasing Tax"

Many debiasing techniques suffer from a "debiasing tax," where OOD improvements come at the cost of IID performance. KDAR avoids this pitfall, maintaining high accuracy on the standard VQAv2 dataset (65.54%). This demonstrates that our method enhances robustness without sacrificing the model's ability to handle standard distributions.

Conclusion

KDAR effectively mitigates language priors in VQA by treating bias as an overfitting issue. By combining the semantic regularization of soft labels with an adaptive reweighting strategy, we enable models to achieve superior generalization. Our method is model-agnostic and sets a new benchmark for performance on OOD datasets while remaining competitive on standard benchmarks.

Summary updated

Abstract

Previous studies have pointed out that visual question answering (VQA) models are prone to relying on language priors for answer predictions. In this context, predictions often depend on linguistic shortcuts rather than a comprehensive grasp of multimodal knowledge, which diminishes their generalization ability. In this paper, we propose a novel method, namely, KDAR, leveraging knowledge distillation to address the prior-dependency dilemmas within the VQA task. Specifically, the regularization effect facilitated by soft labels from a well-trained teacher is employed to penalize overfitting to the most common answers. The soft labels, which serve a regularization role, also provide semantic guidance that narrows the range of candidate answers. Additionally, we design an adaptive sample-wise reweighting learning strategy to further mitigate bias by dynamically adjusting the importance of each sample. Experimental results demonstrate that our method enhances performance in both OOD and IID settings. Our method achieves state-of-the-art performance on the VQA-CPv2 out-of-distribution (OOD) benchmark, significantly outperforming previous state-of-the-art approaches.
BibTeX
@inproceedings{peng2024overcoming,
  author = {Daowan Peng and Wei Wei},
  title = {{Overcoming Language Priors for Visual Question Answering Based on Knowledge Distillation}},
  booktitle = {ICME},
  year = {2024},
  url = {https://openreview.net/forum?id=v4d45nUtLm},
}
Figure