A Survey of Low-shot Vision-Language Model Adaptation via Representer Theorem

Kun Ding, Ying Wang, Gaofeng Meng, Shiming XiangarXiv 2024 · DOI · arXiv

Abstract

This paper presents a unified theoretical framework for adapting pre-trained vision-language models (VLMs) like CLIP to low-shot tasks, grounded in the mathematical Representer Theorem. By decomposing diverse Parameter-Efficient Fine-Tuning (PEFT) methods into a single five-component architecture, the authors provide a systematic taxonomy of the current research landscape. Furthermore, they identify a key limitation in existing "cache-based" adapters—the assumption of class independence—and propose a novel improvement using Kernel Ridge Regression (KRR) to model inter-class correlations, leading to consistent performance gains across 11 benchmark datasets.

Introduction

Foundation models such as CLIP have demonstrated remarkable "zero-shot" capabilities across various vision tasks. However, adapting these models to specific downstream tasks with very limited data (low-shot learning) remains a significant challenge. Traditional full-parameter fine-tuning often "breaks" the robust pre-trained features, leading to poor generalization on small datasets.

To address this, researchers have developed a wide array of Parameter-Efficient Fine-Tuning (PEFT) methods, including prompt tuning (modifying input text) and adapters (inserting small bottleneck layers). Despite their success, the field has become a "zoo" of disconnected techniques without a common theoretical framework to compare them or explain their relative performance. This paper seeks to provide that missing theoretical map by reframing these methods through the lens of the Representer Theorem.

Comparison of common VLM adaptation methods: LP-CLIP, Zero-shot CLIP, and CoOp.

Unified Computational Framework

The authors propose a unified framework based on the Representer Theorem, which states that the optimal solution for certain learning problems can be expressed as a linear combination of kernel functions centered on the training data. They break down every PEFT method into five functional blocks:

  1. Input Image Encoding: The process of transforming a raw image into a feature vector.
  2. Anchors Computation: Defining the "keys" or reference points, such as training image features or text prompts.
  3. Kernels Computation: Measuring the similarity between the input and the defined anchors.
  4. Logits Computation: Transforming these similarities into class predictions.
  5. Loss Function: The objective used to update the learnable parameters.
The proposed unified computational framework for VLM adaptation based on the Representer Theorem.

By viewing existing methods like CoOp, Tip-Adapter, and CALIP through this lens, the authors realized that most current "cache-based" models treat classes as independent. To rectify this, they implemented a variant using the closed-form solution of Kernel Ridge Regression (KRR), which mathematically accounts for the correlations between different classes during the prediction step.

Experimental Evaluation

The framework and the proposed KRR variant were evaluated across 11 diverse public datasets, including general object recognition (ImageNet), fine-grained classification (OxfordPets, StanfordCars), and specialized domains like satellite imagery (EuroSAT) and textures (DTD).

Key Results

  • Inter-Class Correlation: The authors demonstrate that the popular "cache model" (e.g., in Tip-Adapter) is a simplified, non-parametric version of the Representer Theorem. By using KRR to model how different class anchors relate to one another, accuracy significantly improved.
  • Performance Gains: Adding KRR to prompt-based methods like CoOp increased average accuracy by 2.14%, while for KgCoOp, the boost was 4.56%. For adapter-based methods like Tip-Adapter, KRR improved accuracy by up to 4.61% at 16-shots.
  • Robustness: On datasets where standard cache models typically underperform (e.g., FGVCAircraft), the KRR solution successfully "rectified" the performance, turning potential drops into significant gains.

Conclusion and Future Work

This survey successfully maps nearly 15 different state-of-the-art methods into a single taxonomy, clarifying the relationships between prompt-based and adapter-based tuning. While the framework currently focuses on contrastive VLMs for image classification, future work includes extending this theoretical grounding to generative models (like Flamingo) and dense tasks such as object detection. The authors also suggest that "Pre-training with Representers"—integrating kernel-based ideas directly into the initial training of foundation models—could be a promising research direction.

Summary updated

Abstract

The advent of pre-trained vision-language foundation models has revolutionized the field of zero/few-shot (i.e., low-shot) image recognition. The key challenge to address under the condition of limited training data is how to fine-tune pre-trained vision-language models in a parameter-efficient manner. Previously, numerous approaches tackling this challenge have been proposed. Meantime, a few survey papers are also published to summarize these works. However, there still lacks a unified computational framework to integrate existing methods together, identify their nature and support in-depth comparison. As such, this survey paper first proposes a unified computational framework from the perspective of Representer Theorem and then derives many of the existing methods by specializing this framework. Thereafter, a comparative analysis is conducted to uncover the differences and relationships between existing methods. Based on the analyses, some possible variants to improve the existing works are presented. As a demonstration, we extend existing methods by modeling inter-class correlation between representers in reproducing kernel Hilbert space (RKHS), which is implemented by exploiting the closed-form solution of kernel ridge regression. Extensive experiments on 11 datasets are conducted to validate the effectiveness of this method. Toward the end of this paper, we discuss the limitations and provide further research directions.
BibTeX
@misc{ding2024survey,
  author = {Kun Ding and Ying Wang and Gaofeng Meng and Shiming Xiang},
  title = {{A Survey of Low-shot Vision-Language Model Adaptation via Representer Theorem}},
  year = {2024},
  eprint = {2410.11686},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.11686},
}
Figure