TL;DR
This paper introduces JailPO, a novel black-box framework that automates the creation of jailbreak prompts for aligned Large Language Models (LLMs). By leveraging preference optimization and two specialized attack models, JailPO efficiently generates prompts that bypass safety mechanisms, offering a scalable way to probe LLM vulnerabilities.
What Problem Does This Paper Address?
Large Language Models (LLMs) have become incredibly powerful, but their alignment with human values and safety guidelines is a critical concern. Despite advancements, these "aligned" LLMs can still be tricked into generating harmful or undesirable content through "jailbreak" attacks. Existing methods for creating these attacks are often manual, labor-intensive, or require white-box access to the model, limiting their scalability and practicality. This paper addresses the need for an efficient, universal, and black-box jailbreaking framework to systematically evaluate the security and alignment of LLMs.
Methods
JailPO employs a two-pronged approach:
- Enhanced Model Construction: Two specialized LLMs are trained:
- Question Enhanced Model (QEM): This model learns to generate covert jailbreak questions that are more complex and nuanced, aiming to bypass alignment filters. It's trained using a self-instruction methodology and then fine-tuned with preference optimization.
- Template Enhanced Model (TEM): This model learns to generate complex scenarios or templates that can trick LLMs into responding positively to harmful prompts. Similar to QEM, it uses self-instruction and preference optimization.
- Preference Optimization: This is the core innovation. Instead of just generating prompts, JailPO uses a preference-based approach. It trains attack models to understand what makes a jailbreak prompt "effective" by learning from pairs of preferred (successful jailbreak) and dispreferred (failed jailbreak) responses. This is achieved by:
- Using a detector (ClassJudge) to score the success of generated jailbreak prompts.
- Constructing a preference dataset based on these scores.
- Fine-tuning the attack models using SimPO (Simple Preference Optimization) on this dataset.
JailPO also introduces three jailbreak patterns:
- QEPrompt: Uses prompts generated by QEM for covert expression.
- TemplatePrompt: Combines QEPrompt with templates from TEM for complex scenarios.
- MixAsking: A pattern-matching approach that first uses QEPrompt and then TemplatePrompt if the initial response is non-compliant, aiming for higher efficiency.
Datasets
The paper utilizes the AdvBench dataset for evaluation, which contains 520 questions requesting harmful content like misinformation and dangerous suggestions. The authors note that their test questions are distinct from the training set. For training their attack models, they employ a self-instruction methodology for data augmentation, leveraging existing question sets and templates.
Key Insights and Ideas
- LLMs can learn to jailbreak: Preliminary experiments show that LLMs themselves have the potential to generate effective jailbreak prompts, a key motivation for JailPO.
- Preference optimization is key: Applying preference optimization to attack models significantly improves their ability to generate effective jailbreak prompts, going beyond simple generation.
- Templates vs. Covert Questions: Attacks based on complex templates are more effective at bypassing model alignment, but covert question transformations elicit riskier responses and are more likely to evade defenses.
- Universality is achievable: JailPO demonstrates impressive universality, working effectively across different LLMs (including commercial ones) even when trained on a single base model.
- Alignment vs. Model Size: The paper suggests that as LLMs increase in size, their alignment performance might deteriorate, leading to greater vulnerability to attacks like those generated by JailPO.
Key Results
- High Attack Effectiveness (ASR): JailPO consistently achieves or approaches optimal Attack Success Rate (ASR) across various LLMs, often outperforming baselines like GCG and TemplateJailbreak by orders of magnitude fewer queries. For example, on Llama2, the TemplatePrompt pattern showed a 6-8x increase in ASR compared to the original TemplateJailbreak.
- Efficient Question Targeting (QSR): JailPO demonstrates strong Question Success Rate (QSR) with minimal queries, indicating its ability to rapidly generate effective jailbreak prompts. MixAsking showed a significant superiority in QSR.
- Robustness Against Defenses: JailPO, particularly QEPrompt, shows outstanding performance against defense mechanisms like Perplexity and LLM-Guard, often achieving nearly 100% Defense Rassing Rate (DPR) on certain models.
- Advanced LLM Performance: JailPO remains effective even on the latest LLMs like GPT-4 and Command-R, achieving comparable attack performance to Llama2 and demonstrating its universality. Command-R was found to be particularly vulnerable.
Limitations and Future Work
- Base Model Dependency: While JailPO shows universality, the performance can vary slightly depending on the base model used for training the attack models. Llama2-based results showed better performance against GPT-3.5 than other base models.
- Query Budget: While efficient, JailPO still requires a certain number of queries. The paper explores the trade-off between query iterations and QSR, suggesting that an appropriate number of queries can achieve satisfactory attack coverage at an acceptable cost.
- Future Work: The authors suggest that their findings aid in exploring LLM vulnerabilities and provide insights for developing better alignment methods to ensure safe LLM usage. Further research could explore more sophisticated preference learning techniques or investigate the inherent conflict between alignment objectives and generative capabilities in larger models.
Summary updated