Understanding and Improving Visual Prompting: A Label-Mapping Perspective

Aochuan Chen, Yuguang Yao, Pin-Yu Chen, Yihua Zhang, Sijia Liu

Introduction

Visual Prompting (VP) is an emerging technique for "reprogramming" fixed, pre-trained models to perform new tasks without modifying their weights. By learning a small "visual prompt"—a pattern of pixels added to the input—researchers can nudge a model like ResNet (trained on ImageNet) to classify entirely different data, such as flower species or medical scans.

A significant challenge in VP is Label Mapping (LM): determining which source class (e.g., "Cardoon") should correspond to a target class (e.g., "Spear Thistle"). Previous approaches used static or random mappings, which are often sub-optimal. This paper introduces ILM-VP (Iterative Label Mapping-based Visual Prompting) to solve this by co-optimizing the prompt and the mapping.

Overview of the ILM-VP framework compared to prior art, highlighting the iterative co-optimization of visual prompts and label mappings alongside performance gains.

Methodology: The ILM-VP Framework

The core of ILM-VP is framing visual prompting as a Bi-Level Optimization (BLO) problem. Instead of a fixed assignment, the label mapping is treated as a dynamic variable that evolves alongside the visual prompt:

  1. Upper-Level Optimization: The visual prompt (pixel perturbation) is updated to minimize classification loss on the target task.
  2. Lower-Level Optimization: The label mapping is re-evaluated and updated based on which source classes the model naturally predicts for the target data as the prompt changes.

This alternating optimization cycle ensures that the prompt and the mapping "co-evolve," leading to a more stable and effective reprogramming process.

Experimental Setup

The researchers evaluated ILM-VP across 13 diverse target datasets, including:

  • Natural Images: CIFAR-10/100, OxfordPets, Flowers102.
  • Specialized Domains: GTSRB (traffic signs), EuroSAT (satellite imagery), DTD (textures).
  • Medical Data: ABIDE (brain-regional correlation graphs).

They utilized source models pre-trained on ImageNet-1K (ResNet-18, ResNet-50) and Instagram (ResNeXt-101).

Key Insights: Semantic Emergence

One of the most compelling findings is that ILM-VP discovers semantically meaningful mappings without explicit supervision. For instance, it might map "Spear Thistle" to "Cardoon" or "Oxeye Daisy" to "Daisy" because these classes share visual features. This "semantic emergence" suggests that visual prompting works by identifying "visual synonyms" within the source model's pre-trained vocabulary.

Examples of semantically meaningful label mappings discovered by ILM-VP, showing how target classes are mapped to visually similar source classes.

Key Results

  • Performance Gains: ILM-VP consistently outperformed existing baselines. On ResNet-18, it achieved a 7.9% accuracy boost on Flowers102 and 6.7% on CIFAR-100 compared to the previous state-of-the-art.
  • Parameter Efficiency: ILM-VP is highly efficient, requiring only ~0.05M trainable parameters—a fraction of the 11.7M+ parameters needed for full fine-tuning.
  • Robustness in Data-Scarce Regimes: In medical datasets like ABIDE, visual prompting actually outperformed full fine-tuning, likely because the smaller parameter count prevents overfitting on limited data.

Limitations and Future Work

While ILM-VP significantly narrows the gap, visual prompting still generally lags behind full-model fine-tuning in absolute accuracy for standard natural image tasks. Additionally, the bi-level optimization process introduces some computational overhead compared to simpler one-shot mapping methods. Future research may explore more complex output transformations and the theoretical limits of model reprogrammability.

Summary updated

Abstract

We revisit and advance visual prompting (VP), an input prompting technique for vision tasks. VP can reprogram a fixed, pre-trained source model to accomplish downstream tasks in the target domain by simply incorporating universal prompts (in terms of input perturbation patterns) into downstream data points. Yet, it remains elusive why VP stays effective even given a ruleless label mapping (LM) between the source classes and the target classes. Inspired by the above, we ask: How is LM interrelated with VP? And how to exploit such a relationship to improve its accuracy on target tasks? We peer into the influence of LM on VP and provide an affirmative answer that a better ‘quality’ of LM (assessed by mapping precision and explanation) can consistently improve the effectiveness of VP. This is in contrast to the prior art where the factor of LM was missing. To optimize LM, we propose a new VP framework, termed ILM-VP (iterative label mapping-based visual prompting), which automatically re-maps the source labels to the target labels and progressively improves the target task accuracy of VP. Further, when using a contrastive language-image pretrained (CLIP) model for VP, we propose to integrate an LM process to assist the text prompt selection of CLIP and to improve the target task accuracy. Extensive experiments demonstrate that our proposal significantly outperforms state-of-the-art VP methods. As highlighted below, we show that when reprogramming an ImageNet-pretrained ResNet-18 to 13 target tasks, ILM-VP outperforms baselines by a substantial margin, e.g., 7.9% and 6.7% accuracy improvements in transfer learning to the target Flowers102 and CIFAR100 datasets. Besides, our proposal on CLIP-based VP provides 13.7% and 7.1% accuracy improvements on Flowers102 and DTD respectively. Code is available at https://github.com/OPTML-Group/ILM-VP.
BibTeX
@inproceedings{chen2022understanding,
  author = {Aochuan Chen and Yuguang Yao and Pin-Yu Chen and Yihua Zhang and Sijia Liu},
  title = {{Understanding and Improving Visual Prompting: A Label-Mapping Perspective}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52729.2023.01834},
}
Figure