PromptAttack: Prompt-Based Attack for Language Models via Gradient Search

Yundi Shi, Piji Li, Changchun Yin, Zhaoyang Han, Lu Zhou, Zhe Liu

Abstract

As Large Language Models (LLMs) shift toward the "prompt learning" paradigm, the security of the prompt template itself becomes a critical concern. This paper introduces PromptAttack, a novel adversarial method designed to automatically construct malicious prompt templates that deceive Pre-trained Language Models (PLMs). By leveraging gradient-based search to identify influential "trigger tokens," PromptAttack can reduce a model's classification accuracy by over 50%. Our findings reveal that prompt-based systems are highly vulnerable to targeted template manipulation, with larger, more robust models often exhibiting the greatest susceptibility.

Introduction

Traditional fine-tuning of Large Language Models involves updating all model parameters, a process that has become computationally prohibitive as model sizes scale. In response, "Prompt Learning" has emerged as a lightweight alternative. In this paradigm, models are provided with a "cloze-style" template—such as [Sentence] The movie is [MASK]—to perform downstream tasks without extensive retraining.

While research has focused on optimizing these templates for accuracy, their security implications have remained largely unexamined. Most practitioners assume templates are either hand-crafted or benignly optimized. PromptAttack challenges this assumption by asking: Can an attacker craft a malicious template that appears benign but forces the model to output an incorrect label? This study demonstrates that the template represents a significant and previously unstudied attack surface in NLP.

The PromptAttack Framework

PromptAttack is a white-box adversarial method that requires access to the target model's gradients. The attack follows a structured three-stage pipeline to transform a standard prompt into a malicious one.

1. Automatic Label Mapping

Before initiating the search for trigger tokens, the system must identify which words in the model's vocabulary best represent the target labels. For instance, in a sentiment analysis task, the system might determine that the word "extraordinary" has the highest correlation with a "Positive" sentiment label within the model's internal representation.

Inspired by the concept of Universal Adversarial Triggers, PromptAttack uses the model's gradients to identify tokens that minimize the probability of the correct label. By calculating a first-order approximation of how changing a specific token affects the loss function, the method can efficiently scan the entire vocabulary to find "unfriendly" candidate words that are most likely to cause a misclassification.

3. Sequence Selection Strategies

To organize these candidate words into a functional and effective template, the authors evaluated three distinct strategies:

  • Random Replacement: An iterative approach that swaps tokens in the template to find the most damaging combination.
  • Beam Search: A more sophisticated, step-by-step search that maintains the top-$k$ most effective partial templates at each stage, leading to more stable and potent attacks.
  • GPT-2 Refinement: To ensure the attack remains "stealthy," the authors use GPT-2 to generate sequences with lower perplexity. This ensures the malicious template does not look like random gibberish, though it often comes at a slight cost to attack effectiveness.

Experimental Results

The effectiveness of PromptAttack was evaluated on three sentiment classification benchmarks: SST-2, IMDB, and Amazon Video_Games. The authors tested the attack against BERT-base, RoBERTa-base, and RoBERTa-large.

Massive Accuracy Degradation

The results were striking. On the SST-2 dataset using RoBERTa-large, PromptAttack (via Beam Search) reduced the model's accuracy from 89.79% to 33.02%—a drop of 56.77%. Across all three datasets, the attack consistently achieved accuracy degradation of over 40%.

The Vulnerability of Scale

One of the paper's most surprising insights is that larger models are not necessarily more robust against prompt-based attacks. In many cases, RoBERTa-large was more susceptible to PromptAttack than the smaller BERT-base. This suggests that as models become more sensitive to the nuances of prompt instructions, they also become more vulnerable to the precise manipulation of those instructions.

The Stealth-Effectiveness Trade-off

There is a clear tension between the potency of an attack and its readability. Templates discovered via Beam Search were the most effective at breaking the models but resulted in high perplexity (nonsense text). Conversely, templates refined by GPT-2 were much more natural and readable but slightly less effective as an attack vector. This highlights a "stealth-effectiveness trade-off" that attackers must navigate.

Discussion and Future Work

PromptAttack demonstrates that the instructions provided to a model are just as dangerous an attack vector as the input data itself. Even in few-shot scenarios—where the model is only shown 100 samples per category—the attack remained highly effective, dropping accuracy by 20-40%.

However, the current research has limitations. As a white-box attack, it requires internal access to model gradients, which may not always be available to an adversary. Future research should explore black-box variants of this attack. Additionally, this study focused on "hard" prompts (discrete text tokens); the vulnerability of "soft" prompts (continuous vector embeddings) remains an open question. Finally, the development of robust defense mechanisms against malicious templates is now a priority for the NLP community to ensure the safe deployment of prompt-based systems.

Summary updated

Abstract

As the pre-trained language models (PLMs) continue to grow, so do the hardware and data requirements for fine-tuning PLMs. Therefore, the researchers have come up with a lighter method called Prompt Learning. However, during the investigations, we observe that the prompt learning methods are vulnerable and can easily be attacked by some illegally constructed prompts, resulting in classification errors, and serious security problems for PLMs. Most of the current research ignores the security issue of prompt-based methods. Therefore, in this paper, we propose a malicious prompt template construction method (PromptAttack) to probe the security performance of PLMs. Several unfriendly template construction approaches are investigated to guide the model to misclassify the task. Extensive experiments on three datasets and three PLMs prove the effectiveness of our proposed approach PromptAttack. We also conduct experiments to verify that our method is applicable in few-shot scenarios.
BibTeX
@inproceedings{shi2022promptattack,
  author = {Yundi Shi and Piji Li and Changchun Yin and Zhaoyang Han and Lu Zhou and Zhe Liu},
  title = {{PromptAttack: Prompt-Based Attack for Language Models via Gradient Search}},
  booktitle = {NLPCC},
  year = {2022},
  url = {https://openreview.net/forum?id=qJ60uFnJ3Bk},
}
Figure