XtremeCLIP: Extremely Parameter-efficient Tuning for Low-resource Vision Language Understanding

Moming Tang, Chengyu Wang, Jianing Wang, Chuanqi Tan, Songfang Huang, Cen Chen, Weining Qian

Introduction

Pre-trained vision-language models like CLIP have set new benchmarks in multimodal understanding. However, adapting these massive models to specific downstream tasks—such as Visual Question Answering (VQA) or Visual Entailment (VE)—typically requires extensive labeled data and significant computational resources. In "low-resource" scenarios, where only a few hundred or thousand examples are available, standard fine-tuning often leads to overfitting, where the model memorizes the small dataset instead of learning generalizable patterns. While existing parameter-efficient methods like adapters or prompt tuning offer some relief, they often remain task-specific or involve too many trainable parameters for extremely small datasets.

XtremeCLIP addresses these challenges by reformulating Vision-Language Understanding (VLU) as a simple "open-book" matching problem. By tuning as few as 5,000 parameters, it significantly outperforms traditional fine-tuning and adapter-based methods in data-scarce environments.

Methods: Open-Book Affinity Matching

The core innovation of XtremeCLIP is treating VLU tasks as an affinity-matching problem using a Prototype Matrix, rather than building complex task-specific neural network heads.

The XtremeCLIP architecture illustrates the feature fusion process and the use of a tunable prototype matrix for affinity matching.

The framework operates through four key stages:

  1. Feature Fusion: For an input image-text pair, visual and textual features are extracted using frozen CLIP encoders. These features are combined using a fusion function that captures interactions through addition ($v+t$), subtraction ($v-t$), and element-wise multiplication ($v \times t$).
  2. Knowledge-Base Prototypes: The model maintains "prototypes"—representative vectors—for every possible answer or category. These are initialized by averaging the fused features of training samples within each category.
  3. Affinity Matching: Predictions are made by calculating the similarity (affinity) between the input's fused features and the category prototypes.
  4. Training Paradigm: The CLIP encoders remain completely frozen. Only the Prototype Matrix is learned. Training utilizes a combination of standard Cross-Entropy loss and a Contrastive Learning (CL) loss to ensure the model correctly ranks categories.

Experimental Setup

The authors evaluated XtremeCLIP across three VLU task types using five datasets:

  • Visual Entailment (VE): SNLI-VE (determining if an image supports, contradicts, or is neutral toward a text premise).
  • Visual Question Answering (VQA): VQA v2 (yes/no subset).
  • Image Classification (IC): EuroSAT (satellite imagery), DTD (textures), and FGVC (fine-grained aircraft models).

Testing focused on low-resource subsets (2k, 5k, and 10k samples for VE/VQA) and few-shot settings (8 and 16 images per class for IC).

Key Insights

  • Extreme Parameter Efficiency: XtremeCLIP utilizes only 5,000 to 7,000 tunable parameters—a 20,000x reduction compared to the 149 million parameters in full fine-tuning. This efficiency is the primary defense against overfitting in low-data regimes.
  • Unified Architecture: Unlike previous methods requiring different architectures for classification versus entailment, XtremeCLIP uses the same matching logic across all tasks.
  • Rich Supervisory Signals: By incorporating contrastive learning, the model learns the relative ranking of all labels, providing more informative feedback than simple classification when data is limited.

Key Results

XtremeCLIP consistently outperformed significantly larger fine-tuning methods across all benchmarks.

Performance comparison of XtremeCLIP versus full fine-tuning across different training data scales for Visual Entailment and Image Classification.

On the SNLI-VE (10k) task, XtremeCLIP achieved 62.06% accuracy, compared to 51.10% for full fine-tuning. In 16-shot FGVC aircraft classification, it reached 48.30%, nearly doubling the performance of full fine-tuning (28.14%) and surpassing specialized adapter methods like Tip-Adapter (45.12%). As shown in the data-scale studies, XtremeCLIP’s performance scales sharply with additional data, whereas standard fine-tuning struggles to gain traction in low-data regimes.

Limitations and Future Work

XtremeCLIP is currently designed for discriminative tasks (choosing from a fixed set of labels) and cannot perform generative tasks like image captioning. Furthermore, the framework is currently optimized for the CLIP architecture. Future research will focus on extending this matching paradigm to generative tasks and applying it to other large-scale vision-language models.

Summary updated

BibTeX
@inproceedings{tang2023xtremeclip,
  author = {Moming Tang and Chengyu Wang and Jianing Wang and Chuanqi Tan and Songfang Huang and Cen Chen and Weining Qian},
  title = {{XtremeCLIP: Extremely Parameter-efficient Tuning for Low-resource Vision Language Understanding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.397},
}
Figure