Abstract
This paper presents a unified theoretical framework for adapting pre-trained vision-language models (VLMs) like CLIP to low-shot tasks, grounded in the mathematical Representer Theorem. By decomposing diverse Parameter-Efficient Fine-Tuning (PEFT) methods into a single five-component architecture, the authors provide a systematic taxonomy of the current research landscape. Furthermore, they identify a key limitation in existing "cache-based" adapters—the assumption of class independence—and propose a novel improvement using Kernel Ridge Regression (KRR) to model inter-class correlations, leading to consistent performance gains across 11 benchmark datasets.
Introduction
Foundation models such as CLIP have demonstrated remarkable "zero-shot" capabilities across various vision tasks. However, adapting these models to specific downstream tasks with very limited data (low-shot learning) remains a significant challenge. Traditional full-parameter fine-tuning often "breaks" the robust pre-trained features, leading to poor generalization on small datasets.
To address this, researchers have developed a wide array of Parameter-Efficient Fine-Tuning (PEFT) methods, including prompt tuning (modifying input text) and adapters (inserting small bottleneck layers). Despite their success, the field has become a "zoo" of disconnected techniques without a common theoretical framework to compare them or explain their relative performance. This paper seeks to provide that missing theoretical map by reframing these methods through the lens of the Representer Theorem.
Unified Computational Framework
The authors propose a unified framework based on the Representer Theorem, which states that the optimal solution for certain learning problems can be expressed as a linear combination of kernel functions centered on the training data. They break down every PEFT method into five functional blocks:
- Input Image Encoding: The process of transforming a raw image into a feature vector.
- Anchors Computation: Defining the "keys" or reference points, such as training image features or text prompts.
- Kernels Computation: Measuring the similarity between the input and the defined anchors.
- Logits Computation: Transforming these similarities into class predictions.
- Loss Function: The objective used to update the learnable parameters.
By viewing existing methods like CoOp, Tip-Adapter, and CALIP through this lens, the authors realized that most current "cache-based" models treat classes as independent. To rectify this, they implemented a variant using the closed-form solution of Kernel Ridge Regression (KRR), which mathematically accounts for the correlations between different classes during the prediction step.
Experimental Evaluation
The framework and the proposed KRR variant were evaluated across 11 diverse public datasets, including general object recognition (ImageNet), fine-grained classification (OxfordPets, StanfordCars), and specialized domains like satellite imagery (EuroSAT) and textures (DTD).
Key Results
- Inter-Class Correlation: The authors demonstrate that the popular "cache model" (e.g., in Tip-Adapter) is a simplified, non-parametric version of the Representer Theorem. By using KRR to model how different class anchors relate to one another, accuracy significantly improved.
- Performance Gains: Adding KRR to prompt-based methods like CoOp increased average accuracy by 2.14%, while for KgCoOp, the boost was 4.56%. For adapter-based methods like Tip-Adapter, KRR improved accuracy by up to 4.61% at 16-shots.
- Robustness: On datasets where standard cache models typically underperform (e.g., FGVCAircraft), the KRR solution successfully "rectified" the performance, turning potential drops into significant gains.
Conclusion and Future Work
This survey successfully maps nearly 15 different state-of-the-art methods into a single taxonomy, clarifying the relationships between prompt-based and adapter-based tuning. While the framework currently focuses on contrastive VLMs for image classification, future work includes extending this theoretical grounding to generative models (like Flamingo) and dense tasks such as object detection. The authors also suggest that "Pre-training with Representers"—integrating kernel-based ideas directly into the initial training of foundation models—could be a promising research direction.
Summary updated