Introduction
Visual Prompting (VP) is an emerging technique for "reprogramming" fixed, pre-trained models to perform new tasks without modifying their weights. By learning a small "visual prompt"—a pattern of pixels added to the input—researchers can nudge a model like ResNet (trained on ImageNet) to classify entirely different data, such as flower species or medical scans.
A significant challenge in VP is Label Mapping (LM): determining which source class (e.g., "Cardoon") should correspond to a target class (e.g., "Spear Thistle"). Previous approaches used static or random mappings, which are often sub-optimal. This paper introduces ILM-VP (Iterative Label Mapping-based Visual Prompting) to solve this by co-optimizing the prompt and the mapping.
Methodology: The ILM-VP Framework
The core of ILM-VP is framing visual prompting as a Bi-Level Optimization (BLO) problem. Instead of a fixed assignment, the label mapping is treated as a dynamic variable that evolves alongside the visual prompt:
- Upper-Level Optimization: The visual prompt (pixel perturbation) is updated to minimize classification loss on the target task.
- Lower-Level Optimization: The label mapping is re-evaluated and updated based on which source classes the model naturally predicts for the target data as the prompt changes.
This alternating optimization cycle ensures that the prompt and the mapping "co-evolve," leading to a more stable and effective reprogramming process.
Experimental Setup
The researchers evaluated ILM-VP across 13 diverse target datasets, including:
- Natural Images: CIFAR-10/100, OxfordPets, Flowers102.
- Specialized Domains: GTSRB (traffic signs), EuroSAT (satellite imagery), DTD (textures).
- Medical Data: ABIDE (brain-regional correlation graphs).
They utilized source models pre-trained on ImageNet-1K (ResNet-18, ResNet-50) and Instagram (ResNeXt-101).
Key Insights: Semantic Emergence
One of the most compelling findings is that ILM-VP discovers semantically meaningful mappings without explicit supervision. For instance, it might map "Spear Thistle" to "Cardoon" or "Oxeye Daisy" to "Daisy" because these classes share visual features. This "semantic emergence" suggests that visual prompting works by identifying "visual synonyms" within the source model's pre-trained vocabulary.
Key Results
- Performance Gains: ILM-VP consistently outperformed existing baselines. On ResNet-18, it achieved a 7.9% accuracy boost on Flowers102 and 6.7% on CIFAR-100 compared to the previous state-of-the-art.
- Parameter Efficiency: ILM-VP is highly efficient, requiring only ~0.05M trainable parameters—a fraction of the 11.7M+ parameters needed for full fine-tuning.
- Robustness in Data-Scarce Regimes: In medical datasets like ABIDE, visual prompting actually outperformed full fine-tuning, likely because the smaller parameter count prevents overfitting on limited data.
Limitations and Future Work
While ILM-VP significantly narrows the gap, visual prompting still generally lags behind full-model fine-tuning in absolute accuracy for standard natural image tasks. Additionally, the bi-level optimization process introduces some computational overhead compared to simpler one-shot mapping methods. Future research may explore more complex output transformations and the theoretical limits of model reprogrammability.
Summary updated