Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet their sensitivity to instruction phrasing remains a critical vulnerability. This paper introduces PromptBench, a comprehensive benchmark designed to evaluate the robustness of LLMs against "adversarial prompts"—small, meaning-preserving perturbations in instructions. By analyzing nine state-of-the-art models across 13 datasets, the study reveals a widespread lack of robustness. Notably, word-level attacks can cause an average performance drop of 39%. The authors identify an "attention shift" phenomenon where adversarial noise distracts models from core task data, and they find that few-shot prompting serves as a significant defense mechanism.
Introduction: The Fragility of LLM Instructions
While LLMs are increasingly integrated into complex workflows, they often exhibit "brittle" behavior. Previous research has largely focused on adversarial samples—corrupted data inputs. However, this paper shifts the focus to adversarial prompts: the instructions themselves (e.g., "As a mathematics instructor, calculate...").
Because a single prompt is often used to process thousands of data points, a "broken" or suboptimal prompt has a significantly higher impact than a single corrupted data sample. PromptBench addresses the gap in our understanding of how slight deviations in user input, such as typos or synonym replacements, affect a model's ability to follow instructions while maintaining the original semantic meaning.
Methodology and Benchmarking Framework
The researchers developed a systematic framework to "attack" prompts at four distinct levels of granularity:
- Character-level: Introducing typos or misspellings (e.g., using
TextBuggerorDeepWordBug). - Word-level: Replacing words with synonyms or contextually similar terms (e.g.,
BertAttack). - Sentence-level: Appending irrelevant or distracting sentences to the instruction (e.g.,
StressTest). - Semantic-level: Simulating linguistic nuances by translating prompts into other languages and back to English.
To measure impact, the authors introduced the Performance Drop Rate (PDR), a unified metric quantifying the relative decline in accuracy compared to a "clean" prompt. The evaluation spanned 9 popular LLMs (including GPT-4, ChatGPT, Llama2, and Vicuna) across 13 datasets covering tasks like sentiment analysis (SST-2), grammar correctness (CoLA), reading comprehension (SQuAD V2), and mathematical reasoning.
Key Insights: The Attention Shift Phenomenon
A primary contribution of this work is the identification of the "Attention Shift" phenomenon. Using attention visualization, the authors observed that adversarial prompts cause LLMs to "lose focus." Instead of attending to the core data provided in the sample, the model's attention weights shift toward the perturbed characters or words in the prompt. This distraction leads to incorrect reasoning and failure to follow the original instruction.
Beyond attention shifts, the study yielded several critical findings:
- Word-Level Attacks are Most Potent: Synonym replacement proved more effective at confusing models than simple typos, leading to the highest PDR across most tasks.
- Few-Shot as a Shield: Providing a few examples (Few-shot prompting) significantly increases resilience compared to Zero-shot instructions. The examples act as a contextual anchor, helping the model ignore prompt noise.
- Robustness vs. Scale: Larger models are not universally more robust. While GPT-4 and UL2 performed well, some smaller models occasionally outperformed larger counterparts, suggesting that fine-tuning data quality may be more vital for robustness than raw parameter count.
Results and Model Performance
The benchmark results confirm a widespread vulnerability across the LLM landscape. Word-level attacks caused an average 39% performance drop across all evaluated tasks.
In terms of model rankings, GPT-4 and UL2 emerged as the most robust, while Vicuna demonstrated the least resilience. Furthermore, adversarial prompts were found to be "transferable"; a prompt crafted to fool one model (like ChatGPT) often successfully fooled others (like T5-large), though the effect was less pronounced than direct attacks. Human validation confirmed that over 85% of these adversarial prompts remained semantically equivalent to the originals, highlighting a significant gap between human and machine understanding.
Conclusion and Future Directions
PromptBench highlights the urgent need for more robust LLM training and prompt engineering practices. While the study focused on black-box attacks, future work may explore "white-box" optimizations where attackers have access to model weights. Additionally, while input preprocessing (like spell-check) and adversarial training are potential defenses, their full implementation within a standardized benchmark remains an open area for research. As LLMs become more ubiquitous, ensuring they can reliably interpret instructions despite minor linguistic variations is paramount for safety and usability.
Summary updated