Bridging attack and prompting: An Enhanced Visual Prompting at the pixel level

Junyang Wu, Xianhang Li, Chen Wei, Huiyu Wang, Alan Yuille, Yuyin Zhou, Cihang XieOpenReview 2022 · OpenReview

Abstract

In this paper, we study the problem of the visual prompt at the pixel level. Recent works demonstrate flexibility and generalization of visual-only prompt. However, it still cannot achieve superior results compared with linear probe in terms of accuracy and parameter efficiency. We believe that the full power of visual prompt remains to be harnessed through a novel perspective, which bridges adversarial attack and visual prompt considering the high similarity in both formats and objective functions. Bringing in the “old ideas” in adversarial attacks to enhance visual prompt is promising since there are extensive theoretical and empirical solutions to improve the performance of adversarial attack. Therefore, we propose a novel and concise visual prompting method incorporating simple and effective training strategies inspired by ideas from adversarial attack. Specifically, we introduce the input diversity and gradient normalization into visual prompt learning to obtain better generalization ability. Moreover, to avoid disruptions to the original image caused by perturbation without changing the spatial size of inputs, we separate the prompt and image by shrinking and then padding the image with learnable visual prompts, which can significantly improve the performance further without increasing FLOPs. Extensive experiments are conducted on various large-scale pre-trained models across several downstream datasets under different scenarios. We show that with a CLIP-based model, our enhanced visual prompt can successfully outperform linear probe by 1.9% across 12 datasets on average with a comparable number of parameters, and can even match fully fine-tuning paradigm in some settings training only 0.04% parameters.
BibTeX
@misc{wu2022bridging,
  author = {Junyang Wu and Xianhang Li and Chen Wei and Huiyu Wang and Alan Yuille and Yuyin Zhou and Cihang Xie},
  title = {{Bridging attack and prompting: An Enhanced Visual Prompting at the pixel level}},
  year = {2022},
  url = {https://openreview.net/forum?id=RjsiAoZqN6},
  howpublished = {OpenReview},
}

Loading related content...

Figure