CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment

Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei

Introduction

Contrastive Language-Image Pre-training (CLIP) has emerged as a cornerstone of modern computer vision, primarily due to its ability to perform zero-shot image classification by aligning visual and textual representations. However, its utility in more complex Vision-Language Understanding (VLU) tasks, such as Visual Question Answering (VQA) and Visual Entailment, has historically been limited. This limitation stems from a fundamental "task gap": CLIP is trained to associate images with descriptive captions, whereas VQA requires reasoning over specific questions. This paper investigates whether CLIP’s massive pre-trained knowledge can be unlocked for VLU tasks through innovative prompting and parameter-efficient fine-tuning.

Bridging the Task Gap

The primary challenge in applying CLIP to VQA is that the model is a discriminator, not a generative engine. Simply concatenating a question with an image often results in performance no better than random guessing because the input format deviates significantly from the image-caption pairs seen during pre-training. To address this, the authors propose two main strategies: TAP-C for zero-shot scenarios and BiNor for few-shot adaptation.

TAP-C: Template-Answer-Prompt then CLIP

TAP-C is a zero-shot method designed to transform VQA into a format CLIP understands: image-text matching. It involves two key steps:

  1. Automatic Template Generation: Using a pre-trained T5 model, the system converts a question (e.g., "What color is the ball?") into a masked statement (e.g., "The color of the ball is [mask]"). This aligns the text input with CLIP's "description" bias.
  2. Answer Filtering: To narrow the search space, a language model filters the standard VQA answer set to identify contextually plausible candidates. CLIP then ranks these candidates by calculating the alignment score between the image and the completed template.

BiNor: Parameter-Efficient Fine-Tuning

In few-shot learning environments, full fine-tuning of CLIP often leads to rapid overfitting due to the high parameter count relative to the small dataset. The authors introduce BiNor (Bias and Normalization), a strategy that updates only the bias terms and normalization layers. By modifying only 0.3 million parameters—a fraction of the total model—BiNor maintains the robustness of the pre-trained weights while adapting the model to the specific nuances of the VLU task.

Cross-Modality Transfer

A significant finding of this research is the strength of CLIP's latent space alignment. The authors tested this by training a classifier for Visual Entailment (SNLI-VE) using only text-text pairs (captions and hypotheses). When evaluated on image-text pairs, the model maintained high accuracy (~67%), suggesting that the logic learned in the textual modality transfers seamlessly to the visual modality. This confirms that CLIP’s representations are truly multi-modal and structurally consistent across domains.

Experimental Results

The proposed methods were evaluated on the VQAv2 and SNLI-VE benchmarks, yielding the following insights:

  • Zero-shot VQA: Utilizing TAP-C, the CLIP (ViT-B/16) model achieved an accuracy of 38.72% on VQAv2. This significantly outperforms the 7-billion parameter Frozen model, which scored 29.50%, despite CLIP being substantially smaller.
  • Few-shot VQA: With only 16 examples per category, the performance improved to 47.42% for the ViT-B/16 architecture and 48.89% for Res50x16.
  • Visual Entailment: The cross-modality transfer experiments demonstrated that CLIP could effectively perform visual reasoning even without explicit visual training for the entailment task.

Discussion and Limitations

While CLIP demonstrates impressive few-shot capabilities, the study identifies specific "blind spots." CLIP struggles with fine-grained counting, particularly for small objects or those in cluttered scenes. This is likely a byproduct of the global contrastive loss used during pre-training, which prioritizes global semantic alignment over local spatial details. Additionally, the model occasionally exhibits semantic confusion between foreground and background elements (e.g., attributing an action performed by a background figure to the primary subject).

Conclusion

This research establishes that CLIP is a highly capable few-shot learner for VLU tasks when the "task gap" is addressed through proper prompting and efficient fine-tuning. By reformulating VQA as a matching problem and focusing on normalization layers during adaptation, CLIP can outperform models many times its size. Future work will likely focus on improving spatial awareness and fine-grained reasoning to overcome current limitations in counting and background-foreground differentiation.

Summary updated

BibTeX
@inproceedings{song2022clip,
  author = {Haoyu Song and Li Dong and Wei-Nan Zhang and Ting Liu and Furu Wei},
  title = {{CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.421},
}
Figure