Towards Robust and Accurate Visual Prompting

Qi Li, Liangzhi Li, Zhouqiang Jiang, Bowen Wang

Abstract

Visual prompting, an efficient method for transfer learning, has shown its potential in vision tasks. However, previous works focus exclusively on VP from standard source models, it is still unknown how it performs under the scenario of a robust source model: Whether a visual prompt derived from a robust model can inherit the robustness while suffering from the generalization performance decline, albeit for a downstream dataset that is different from the source dataset? In this work, we get an affirmative answer of the above question and give an explanation on the visual representation level. Moreover, we introduce a novel technique named Prompt Boundary Loose (PBL) to effectively mitigates the suboptimal results of visual prompt on standard accuracy without losing (or even significantly improving) its adversarial robustness when using a robust model as source model. Extensive experiments across various datasets show that our findings are universal and demonstrate the significant benefits of our proposed method.

Introduction

Transfer learning typically involves fine-tuning large-scale pre-trained models, a process that is often computationally prohibitive. Visual Prompting (VP) has emerged as a lightweight alternative, where the source model remains frozen, and only a small set of learnable parameters—the "prompt"—is optimized to adapt the model to a downstream task.

While VP is efficient, most existing research utilizes standard source models that are vulnerable to adversarial attacks. This paper investigates Robust Source Visual Prompting (RSVP), exploring whether prompts can inherit the adversarial robustness of a robust source model and how to mitigate the inherent accuracy penalties associated with such models.

Methodology: Robust Source Visual Prompting and PBL

The authors confirm that RSVP prompts successfully inherit the robustness of their source models. However, robust models often possess complex and "tight" decision boundaries, which can hinder the optimization of a simple visual prompt, leading to a decline in standard accuracy.

To address this, the researchers propose Prompt Boundary Loose (PBL). Instead of a rigid one-to-one mapping between source and downstream labels, PBL partitions the source model's output vector. It utilizes the maximum confidence score within each partition to represent a downstream class. This effectively "loosens" the decision boundaries, providing a larger target area for the prompt to map into during training.

Schematic of the RSVP pipeline and the Prompt Boundary Loose (PBL) mechanism.

Visual Representation and Human Alignment

A key discovery of this work is the qualitative difference between prompts. Standard source visual prompts (SSVP) typically appear as unstructured noise. In contrast, RSVP prompts exhibit "human-aligned" features, often containing ghost-like visual structures that resemble the target objects (e.g., petal shapes for flower datasets). This suggests that robust models learn representations that are more closely aligned with human visual perception.

Comparison of visual prompts generated from standard models (SSVP) versus robust models (RSVP) across different datasets.

Experimental Evaluation

The method was evaluated using ResNet-18, ResNet-50, and WideResNet-50-2 backbones pre-trained on ImageNet-1K. Testing spanned eight diverse datasets, including Flowers102, SVHN, EuroSAT, and StanfordCars.

Key results include:

  • Accuracy Improvement: PBL consistently boosts standard accuracy. For instance, on EuroSAT with a ResNet-50 backbone, accuracy improved by approximately 5%.
  • Robustness Retention: PBL does not compromise robustness; in fact, it often enhances it. On the DTD dataset, adversarial accuracy rose from 26.47% to 39.36%.
  • Efficiency: Unlike traditional adversarial training, PBL introduces no additional computational overhead during the training phase.

Conclusion

This research demonstrates that visual prompts can effectively inherit robustness from source models. By introducing the PBL technique, the authors provide a way to bypass the traditional robustness-accuracy trade-off in visual prompting. Future work will focus on automating the tuning of the temperature parameter in PBL and bridging the performance gap in extreme cross-domain scenarios.

Summary updated

Abstract

Visual prompting, an efficient method for transfer learning, has shown its potential in vision tasks. However, previous works focus exclusively on VP from standard source models, it is still unknown how it performs under the scenario of a robust source model: Whether a visual prompt derived from a robust model can inherit the robustness while suffering from the generalization performance decline, albeit for a downstream dataset that is different from the source dataset? In this work, we get an affirmative answer of the above question and give an explanation on the visual representation level. Moreover, we introduce a novel technique named Prompt Boundary Loose (PBL) to effectively mitigates the suboptimal results of visual prompt on standard accuracy without losing (or even significantly improving) its adversarial robustness when using a robust model as source model. Extensive experiments across various datasets show that our findings are universal and demonstrate the significant benefits of our proposed method.
BibTeX
@misc{li2023towards,
  author = {Qi Li and Liangzhi Li and Zhouqiang Jiang and Bowen Wang},
  title = {{Towards Robust and Accurate Visual Prompting}},
  year = {2023},
  eprint = {2311.10992},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.10992},
}
Figure