Abstract
As Large Language Models (LLMs) shift toward the "prompt learning" paradigm, the security of the prompt template itself becomes a critical concern. This paper introduces PromptAttack, a novel adversarial method designed to automatically construct malicious prompt templates that deceive Pre-trained Language Models (PLMs). By leveraging gradient-based search to identify influential "trigger tokens," PromptAttack can reduce a model's classification accuracy by over 50%. Our findings reveal that prompt-based systems are highly vulnerable to targeted template manipulation, with larger, more robust models often exhibiting the greatest susceptibility.
Introduction
Traditional fine-tuning of Large Language Models involves updating all model parameters, a process that has become computationally prohibitive as model sizes scale. In response, "Prompt Learning" has emerged as a lightweight alternative. In this paradigm, models are provided with a "cloze-style" template—such as [Sentence] The movie is [MASK]—to perform downstream tasks without extensive retraining.
While research has focused on optimizing these templates for accuracy, their security implications have remained largely unexamined. Most practitioners assume templates are either hand-crafted or benignly optimized. PromptAttack challenges this assumption by asking: Can an attacker craft a malicious template that appears benign but forces the model to output an incorrect label? This study demonstrates that the template represents a significant and previously unstudied attack surface in NLP.
The PromptAttack Framework
PromptAttack is a white-box adversarial method that requires access to the target model's gradients. The attack follows a structured three-stage pipeline to transform a standard prompt into a malicious one.
1. Automatic Label Mapping
Before initiating the search for trigger tokens, the system must identify which words in the model's vocabulary best represent the target labels. For instance, in a sentiment analysis task, the system might determine that the word "extraordinary" has the highest correlation with a "Positive" sentiment label within the model's internal representation.
2. Gradient-Based Candidate Search
Inspired by the concept of Universal Adversarial Triggers, PromptAttack uses the model's gradients to identify tokens that minimize the probability of the correct label. By calculating a first-order approximation of how changing a specific token affects the loss function, the method can efficiently scan the entire vocabulary to find "unfriendly" candidate words that are most likely to cause a misclassification.
3. Sequence Selection Strategies
To organize these candidate words into a functional and effective template, the authors evaluated three distinct strategies:
- Random Replacement: An iterative approach that swaps tokens in the template to find the most damaging combination.
- Beam Search: A more sophisticated, step-by-step search that maintains the top-$k$ most effective partial templates at each stage, leading to more stable and potent attacks.
- GPT-2 Refinement: To ensure the attack remains "stealthy," the authors use GPT-2 to generate sequences with lower perplexity. This ensures the malicious template does not look like random gibberish, though it often comes at a slight cost to attack effectiveness.
Experimental Results
The effectiveness of PromptAttack was evaluated on three sentiment classification benchmarks: SST-2, IMDB, and Amazon Video_Games. The authors tested the attack against BERT-base, RoBERTa-base, and RoBERTa-large.
Massive Accuracy Degradation
The results were striking. On the SST-2 dataset using RoBERTa-large, PromptAttack (via Beam Search) reduced the model's accuracy from 89.79% to 33.02%—a drop of 56.77%. Across all three datasets, the attack consistently achieved accuracy degradation of over 40%.
The Vulnerability of Scale
One of the paper's most surprising insights is that larger models are not necessarily more robust against prompt-based attacks. In many cases, RoBERTa-large was more susceptible to PromptAttack than the smaller BERT-base. This suggests that as models become more sensitive to the nuances of prompt instructions, they also become more vulnerable to the precise manipulation of those instructions.
The Stealth-Effectiveness Trade-off
There is a clear tension between the potency of an attack and its readability. Templates discovered via Beam Search were the most effective at breaking the models but resulted in high perplexity (nonsense text). Conversely, templates refined by GPT-2 were much more natural and readable but slightly less effective as an attack vector. This highlights a "stealth-effectiveness trade-off" that attackers must navigate.
Discussion and Future Work
PromptAttack demonstrates that the instructions provided to a model are just as dangerous an attack vector as the input data itself. Even in few-shot scenarios—where the model is only shown 100 samples per category—the attack remained highly effective, dropping accuracy by 20-40%.
However, the current research has limitations. As a white-box attack, it requires internal access to model gradients, which may not always be available to an adversary. Future research should explore black-box variants of this attack. Additionally, this study focused on "hard" prompts (discrete text tokens); the vulnerability of "soft" prompts (continuous vector embeddings) remains an open question. Finally, the development of robust defense mechanisms against malicious templates is now a priority for the NLP community to ensure the safe deployment of prompt-based systems.
Summary updated