TL;DR
This paper introduces a novel, automated method for generating adversarial prompts that can reliably trick aligned Large Language Models (LLMs) into producing harmful content. Crucially, these attacks are transferable across different LLMs, including proprietary ones, raising significant concerns about the current state of LLM safety.
What Problem Does This Paper Address?
The rapid advancement of LLMs has led to their widespread adoption, but also to concerns about their potential to generate harmful or objectionable content. To combat this, developers have focused on "aligning" these models, aiming to prevent them from producing such outputs. While "jailbreak" attacks have existed, they often require significant human ingenuity and are brittle. Existing automated methods have had limited success. This paper tackles the challenge of creating a robust, automated attack that can consistently bypass LLM alignment mechanisms and induce harmful behavior across a variety of models and prompts.
Methods
The authors propose a three-pronged approach to generate adversarial suffixes:
- Initial Affirmative Responses: The attack aims to make the LLM start its response with an affirmative phrase (e.g., "Sure, here is...") to a harmful query. This primes the model to continue with the objectionable content.
- Combined Greedy and Gradient-Based Discrete Optimization (GCG): This is the core of their automated attack. It iteratively refines an adversarial suffix by:
- Using gradients to identify promising token substitutions.
- Evaluating these candidates to find the one that maximally decreases the loss (i.e., increases the probability of an affirmative, harmful response).
- Unlike previous methods like AutoPrompt, GCG considers all possible token replacements at each step, leading to more effective optimization.
- Robust Multi-Prompt and Multi-Model Attacks: To ensure the attack is transferable and effective across various scenarios, the optimization process is performed across multiple prompts and multiple LLMs simultaneously. This results in a single adversarial suffix that works broadly.
Datasets
The authors introduce a new benchmark called AdvBench, designed to evaluate adversarial attacks on LLMs for eliciting harmful content. AdvBench consists of two settings:
- Harmful Strings: 500 strings reflecting harmful content (profanity, graphic depictions, etc.) where the goal is to prompt the model to output these exact strings.
- Harmful Behaviors: 500 harmful behaviors formulated as instructions, where the goal is to prompt the model to attempt to comply with the instruction.
The models used for training and evaluation include open-source LLMs like Vicuna-7B, Vicuna-13B, LLaMA-2-Chat, Pythia, Falcon, and Guanaco, as well as proprietary models such as ChatGPT (GPT-3.5 and GPT-4), Bard (PaLM-2), and Claude.
Key Insights and Ideas
- Automated attacks are highly effective: The paper demonstrates that a well-designed automated attack can outperform manual "jailbreaks" in terms of consistency and success rate.
- Transferability is pervasive: Adversarial prompts generated on one model can reliably attack others, even those with different architectures and tokenizers. This suggests a fundamental vulnerability in current LLM alignment.
- GCG's advantage: The greedy coordinate gradient approach, by considering all token substitutions at each step, significantly outperforms prior discrete optimization methods like AutoPrompt.
- "Arms race" concerns: The findings echo the challenges faced in computer vision, where adversarial attacks remain a persistent threat, raising questions about the long-term viability of current LLM alignment strategies.
- Interpretability of adversarial prompts: Unlike in image attacks, some adversarial prompts in LLMs show a degree of interpretability, hinting at the underlying mechanisms of the attack.
Key Results
- The proposed GCG attack achieves an 88% success rate on Vicuna-7B for eliciting harmful strings, significantly outperforming AutoPrompt (25%).
- For harmful behaviors, GCG achieves 100% success rate on Vicuna-7B and 88% on LLaMA-2-7B-Chat, compared to prior work's 96% and 36% respectively.
- The generated adversarial prompts show remarkable transferability, achieving up to 87.9% ASR on GPT-3.5 and 53.6% on GPT-4, and 2.1% on Claude-2 (though this is still a significant increase over baselines).
- Ensembling multiple GCG prompts further boosts success rates, achieving 86.6% ASR on GPT-3.5 and 46.9% on GPT-4.
Limitations and Future Work
- Overfitting: The authors note that optimizing for too many steps can lead to overfitting to source models, potentially decreasing transferability.
- Manual Fine-tuning: For extreme harms, manual rephrasing of prompts can significantly improve success rates, suggesting that fully automated attacks may still have limitations for the most sensitive content.
- Claude-2 Robustness: Claude-2 appears to be more robust to the automated attack compared to other commercial models, though manual fine-tuning can still improve results.
- Future Work: The authors suggest exploring whether adversarial training can lead to truly robust models without sacrificing generative capabilities, investigating other alignment mechanisms, and understanding the factors that influence transferability across different LLMs. They also highlight the need for more reliable alignment strategies beyond current post-hoc "repair" methods.
Summary updated