Visual Prompting for Adversarial Robustness

Aochuan Chen, Peter Lorenz, Yuguang Yao, Pin-Yu Chen, Sijia Liu

Abstract

In this work, we leverage visual prompting (VP) to improve adversarial robustness of a fixed, pre-trained model at testing time. Compared to conventional adversarial defenses, VP allows us to design universal (i.e., data-agnostic) input prompting templates, which have plug-and-play capabilities at testing time to achieve desired model performance without introducing much computation overhead. Although VP has been successfully applied to improving model generalization, it remains elusive whether and how it can be used to defend against adversarial attacks. We investigate this problem and show that the vanilla VP approach is not effective in adversarial defense since a universal input prompt lacks the capacity for robust learning against sample-specific adversarial perturbations. To circumvent it, we propose a new VP method, termed Class-wise Adversarial Visual Prompting (C-AVP), to generate class-wise visual prompts so as to not only leverage the strengths of ensemble prompts but also optimize their interrelations to improve model robustness. Our experiments show that C-AVP outperforms the conventional VP method, with 2.1X standard accuracy gain and 2X robust accuracy gain. Compared to classical test-time defenses, C-AVP also yields a 42X inference time speedup.

Introduction

Deep learning models are notoriously fragile; small, intentional perturbations to an image (adversarial attacks) can trick a model into misclassifying a "dog" as a "cat." While Adversarial Training (AT) is the gold standard for defense, it is computationally expensive and requires retraining the entire model from scratch. To avoid this, researchers look for "test-time" defenses that work with fixed, off-the-shelf models.

Comparison of test accuracy under PGD attacks for vanilla models and universal visual prompting (U-AVP).

One promising idea is Visual Prompting (VP)—adding a static frame of pixels around an image to help the model generalize. However, as shown in Figure 1, "universal" prompts (U-AVP) fail to provide a robust defense, significantly degrading accuracy on clean images while offering only marginal protection against attacks.

Methodology: Class-wise Adversarial Visual Prompting (C-AVP)

The authors propose Class-wise Adversarial Visual Prompting (C-AVP). Instead of one universal prompt, they create a unique prompt for every class (e.g., one for "planes," one for "birds").

Architectural comparison between (a) No Prompt, (b) Universal Visual Prompting (U-AVP), and (c) the proposed Class-wise Adversarial Visual Prompting (C-AVP).

The core challenge is identifying which prompt to apply to an unlabeled image at test time. If a mismatched prompt is applied, it might act as a "backdoor trigger," forcing a misclassification. To solve this, C-AVP uses a joint optimization strategy with three specialized loss functions:

  1. Correct Selection Loss: Maximizes model confidence when the correct prompt is applied.
  2. Backdoor Mitigation Loss: Penalizes the "backdoor effect" to ensure mismatched prompts do not trick the model.
  3. Coupling Loss: Ensures the correct prompt's influence significantly outweighs incorrect ones.

Experimental Results

The study utilizes the CIFAR-10 dataset and a pre-trained ResNet18 model, evaluating robustness against PGD (Projected Gradient Descent) attacks.

Key Performance Metrics

  • Superior Accuracy: C-AVP achieved 57.57% standard accuracy and ~34.75% robust accuracy, a massive leap over Universal AVP (27.75% standard and 16.9% robust).
  • Efficiency: C-AVP is 42x to 518x faster during inference compared to state-of-the-art test-time defenses like Denoised Smoothing or Input Purification.

Insights

  • Failure of Universality: A single, data-agnostic prompt lacks the capacity to defend against sample-specific adversarial noise. Robustness requires class-specific information.
  • Prompt-as-Backdoor Risk: Class-wise prompts can unintentionally behave like backdoor attacks. The optimization strategy specifically mitigates this "cross-talk" between prompts.

Conclusion and Future Work

C-AVP represents a significant step toward efficient test-time defense. However, a "robustness gap" remains; while 34.75% robust accuracy is impressive for a plug-and-play method, it still lags behind full adversarial training. Furthermore, there is a trade-off between robustness and standard accuracy (dropping from ~94% to ~57%). Future research will focus on closing this gap and scaling the method to larger datasets like ImageNet.

Summary updated

Abstract

In this work, we leverage visual prompting (VP) to improve adversarial robustness of a fixed, pre-trained model at testing time. Compared to conventional adversarial defenses, VP allows us to design universal (i.e., data-agnostic) input prompting templates, which have plug-and-play capabilities at testing time to achieve desired model performance without introducing much computation overhead. Although VP has been successfully applied to improving model generalization, it remains elusive whether and how it can be used to defend against adversarial attacks. We investigate this problem and show that the vanilla VP approach is not effective in adversarial defense since a universal input prompt lacks the capacity for robust learning against sample-specific adversarial perturbations. To circumvent it, we propose a new VP method, termed Class-wise Adversarial Visual Prompting (C-AVP), to generate class-wise visual prompts so as to not only leverage the strengths of ensemble prompts but also optimize their interrelations to improve model robustness. Our experiments show that C-AVP outperforms the conventional VP method, with 2.1X standard accuracy gain and 2X robust accuracy gain. Compared to classical test-time defenses, C-AVP also yields a 42X inference time speedup.
BibTeX
@inproceedings{chen2023visual,
  author = {Aochuan Chen and Peter Lorenz and Yuguang Yao and Pin-Yu Chen and Sijia Liu},
  title = {{Visual Prompting for Adversarial Robustness}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10097245},
}
Figure