When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspective

Hsi-Ai Tsao, Lei Hsiung, Pin-Yu Chen, Tsung-Yi Ho

TL;DR

Choosing between Visual Prompting (VP) and Linear Probing (LP) for transfer learning usually requires expensive trial-and-error training. This paper introduces a Log-Likelihood Ratio (LLR) metric that predicts which method will perform better for a specific dataset with a 100x speedup over traditional training. By approximating visual prompts through clever simulations, the authors show that VP excels when a dataset is "out-of-distribution" (OOD), while LP remains king for "in-distribution" (ID) tasks.

What Problem Does This Paper Address?

When adapting massive pre-trained models (like CLIP) to new tasks, researchers typically use Linear Probing (LP)—adjusting only the final classification layer—or Visual Prompting (VP)—adding a trainable "frame" of pixels around the input image while keeping the model frozen.

The challenge is that there is no "one-size-fits-all" method. VP is known to shine on out-of-distribution (OOD) data where the model struggles, but it can actually hurt performance on data the model already understands well (ID). Currently, the only way to know which to use is to train both and compare results, which is computationally expensive and time-consuming. The authors sought a "training-free" (or low-cost) way to estimate a dataset's compatibility with these methods before committing to full training.

Methods

The researchers built their approach on LogME, an existing method for estimating how well a pre-trained model will adapt to a new task using maximum likelihood. They extended this into a Log-Likelihood Ratio (LLR):

$$LLR(x) = \log p_{VP}(x) - \log p_{LP}(x)$$

  • Positive LLR: Suggests the dataset is OOD and will benefit more from Visual Prompting.
  • Negative LLR: Suggests the dataset is ID and better suited for Linear Probing.

To calculate the VP likelihood without actually training a full prompt, they proposed four Visual Prompt Approximations:

  1. Gaussian Prompts: Adding random noise (completely data-free).
  2. Gradient Approximation: Using a Taylor expansion on a tiny subset of data (~1k samples) to estimate the optimal prompt direction.
  3. Mini-Finetuning: Training a prompt for just 1 to 5 epochs on a small subset.

By comparing the likelihood of these "simulated" prompts against the likelihood of the clean image (LP), they can rank datasets by their expected "accuracy gain" from VP.

Datasets

The authors evaluated their method across 12 diverse downstream datasets and 5 pre-trained models (ResNet18, ResNext-IG, ViT-B-16, Swin-T, and CLIP ViT-B/32). The datasets included:

  • General/Natural: CIFAR10, CIFAR100, Food101, OxfordIIITPet, Flowers102.
  • Specialized/OOD: SVHN (Street View House Numbers), EuroSAT (Satellite imagery), GTSRB (Traffic signs), ISIC (Skin lesions), DTD (Textures), FMoW (Functional Map of the World), and UCF101 (Action recognition).

They also created a "Combined Dataset" by mixing SVHN (OOD) and DTD (ID) in varying proportions to test how the LLR score shifts as a dataset becomes more or less "in-distribution."

Key Insights and Ideas

  • The "Dominant Feature" Theory: Through Pearson Correlation analysis, the authors found that in OOD datasets, the model's embeddings are dominated by the prompt. In ID datasets, the embeddings are dominated by the original image. This explains why VP works: it provides a "bridge" for OOD data, but acts as "noise" for ID data.
  • Simulation is Sufficient: You don't need a perfectly trained prompt to know if a prompt will work. Even simple Gaussian noise or a single gradient step provides enough signal to calculate a meaningful LLR score.
  • LLR vs. OOD Detection: The LLR score proved to be a more reliable predictor of transfer learning success than standard OOD detection baselines like Mahalanobis distance or ODIN confidence scores.

Key Results

  • Efficiency: The LLR approach achieves up to a 100-fold reduction in runtime compared to standard linear probing training.
  • Predictive Accuracy: The method achieved a "selection accuracy" (correctly choosing VP vs. LP) of up to 91.7% on models like Swin-T and ViT-B-16.
  • Strong Correlation: The LLR scores showed high Kendall’s $\tau$ and Spearman’s $\rho$ correlation coefficients with actual accuracy gains, meaning the ranking of datasets was highly consistent with real-world performance.

Limitations and Future Work

  • Model Sensitivity: The authors noted that Vision Transformers (ViT) and Swin Transformers are much more sensitive to prompt quality than convolutional networks (ResNet). If the simulated prompt is too "noisy," the LLR score for these models can become less reliable.
  • Prompt Design: This study focused on the standard "frame" padding prompt. Future work could explore whether LLR remains an effective predictor for more complex prompt architectures (like deep prompting or internal patches).
  • Scaling: While tested on 12 datasets, the "vastness of prompt space" remains a challenge. The authors suggest further refining the approximation methods to handle even larger-scale vision-language models.

Summary updated

Abstract

Adapting pre-trained models to new tasks can exhibit varying effectiveness across datasets. Visual prompting, a state-of-the-art parameter-efficient transfer learning method, can significantly improve the performance of out-of-distribution tasks. On the other hand, linear probing, a standard transfer learning method, can sometimes become the best approach. We propose a log-likelihood ratio (LLR) approach to analyze the comparative benefits of visual prompting and linear probing. By employing the LLR score alongside resource-efficient visual prompts approximations, our cost-effective measure attains up to a 100-fold reduction in run time compared to full training, while achieving prediction accuracies up to 91%. The source code is available at VP-LLR.
BibTeX
@inproceedings{tsao2025when,
  author = {Hsi-Ai Tsao and Lei Hsiung and Pin-Yu Chen and Tsung-Yi Ho},
  title = {{When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspective}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888693},
}
Figure