Abstract
Adapting large-scale pre-trained vision models to downstream tasks often involves a choice between Linear Probing (LP) and Visual Prompting (VP). While VP is effective for out-of-distribution (OOD) data, identifying the optimal adaptation strategy typically requires exhaustive training. We propose a training-free Log-Likelihood Ratio (LLR) metric to predict the performance gain of VP over LP. By approximating visual prompts through efficient simulations, our method achieves a 100x speedup in model selection with high predictive accuracy.
Introduction
The emergence of massive pre-trained models like CLIP has shifted the focus of computer vision toward efficient adaptation. Two primary paradigms exist: Linear Probing (LP), which adjusts the final classification layer, and Visual Prompting (VP), which prepends a trainable pixel "frame" to the input while keeping the model weights frozen.
While VP excels on out-of-distribution (OOD) data, it often underperforms on in-distribution (ID) tasks where the model already possesses strong feature representations. Currently, practitioners must train both methods to determine the superior approach—a process that is computationally prohibitive. This work introduces a training-free method to estimate a dataset's compatibility with VP versus LP before full-scale training.
Methodology: The Log-Likelihood Ratio
We extend the LogME framework, which uses maximum likelihood to estimate model transferability, into a Log-Likelihood Ratio (LLR):
$$LLR(x) = \log p_{VP}(x) - \log p_{LP}(x)$$
A positive LLR indicates that the dataset is likely OOD and will benefit from VP, whereas a negative LLR suggests an ID dataset better suited for LP.
Visual Prompt Approximations
Calculating the VP likelihood typically requires a trained prompt. To avoid this cost, we propose three approximations:
- Gaussian Prompts: Adding random noise to the input (data-free).
- Gradient Approximation: Estimating the prompt direction via a Taylor expansion on a small data subset (~1k samples).
- Mini-Finetuning: Training a prompt for a minimal number of epochs (1–5) on a subset.
These simulations allow us to rank datasets by their expected accuracy gain from VP without full optimization.
Key Insights and Results
The Dominant Feature Theory
A core finding of this study is the "Dominant Feature" theory. In OOD datasets, the model's embeddings are primarily driven by the prompt, which acts as a bridge to the pre-trained feature space. Conversely, in ID datasets, the original image features dominate, and the prompt acts as harmful noise.
As shown in Figure 1, the Pearson Correlation Coefficient (PCC) between the prompt and the embedding is significantly higher for OOD data (SVHN, PCC: 0.90) than for ID data (DTD, PCC: 0.53). In contrast, the final output for ID data is dominated by the clean image (PCC: 0.91).
Performance and Efficiency
The LLR approach demonstrates remarkable efficiency and predictive power:
- Speed: Achieves up to a 100x reduction in runtime compared to standard training.
- Accuracy: Correctly selects the better adaptation method with up to 91.7% accuracy for Transformer-based models.
- Reliability: LLR scores show strong correlation (Kendall’s $\tau$ and Spearman’s $\rho$) with actual accuracy gains, outperforming standard OOD detection baselines like Mahalanobis distance.
Conclusion
This work provides a theoretical and practical framework for choosing between Visual Prompting and Linear Probing. By leveraging the Log-Likelihood Ratio and prompt approximations, researchers can significantly reduce the computational overhead of model adaptation. Future work will explore the scalability of this approach to more complex prompt architectures and even larger vision-language models.
Summary updated