TL;DR
This paper introduces WILDTEAMING, an automated framework for discovering and composing novel jailbreak tactics from real-world user interactions. It also presents WILDJAILBREAK, a large-scale, diverse safety dataset, enabling more robust LLM safety training that balances safeguarding with general capabilities.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly powerful but remain vulnerable to "jailbreaks" – adversarial prompts designed to bypass safety guardrails and elicit harmful content. Existing red-teaming methods are often manual, computationally expensive, or limited in the diversity of attacks they can discover. Furthermore, there's a scarcity of open-source, large-scale datasets for training LLMs to be robust against these sophisticated adversarial attacks, especially those that mimic real-world user behavior. This gap hinders the development of truly safe and reliable LLMs.
Methods
WILDTEAMING employs a two-stage approach:
- MINE (Mining In-the-Wild Tactics): This stage automatically extracts jailbreak tactics from real-world user-chatbot interactions (LMSYS-CHAT-1M and WILDCHAT). It uses GPT-4 to identify and cluster these tactics, uncovering over 5.7K unique clusters of novel jailbreak strategies.
- COMPOSE (Composing Tactics into Attacks): In this stage, WILDTEAMING combines mined tactics with vanilla harmful prompts to generate diverse and challenging adversarial attacks. It leverages off-the-shelf LLMs (like Mixtral-8x7B and GPT-4) for composition and then applies lightweight "off-topic" and "low-risk" pruning to refine the attacks, ensuring they remain harmful and relevant.
Datasets
- WILDJAILBREAK: A newly introduced, large-scale, open-source synthetic safety dataset comprising 262K prompt-response pairs. It includes four types of data:
- Vanilla Harmful (H): Direct requests for harmful content.
- Vanilla Benign (B): Harmless prompts designed to resemble harmful ones, used to mitigate over-refusal.
- Adversarial Harmful (H): Jailbreaks created by WILDTEAMING, transforming vanilla harmful prompts into complex, stealthy attacks.
- Adversarial Benign (B): Adversarial prompts that mimic jailbreaks but contain no harmful intent, used to counteract exaggerated safety behaviors.
- HARMBENCH: Used for evaluation, this benchmark provides a standardized framework for assessing jailbreaking methods across various unsafe scenarios.
Key Insights and Ideas
- In-the-Wild Tactics are Diverse and Effective: Real-world user interactions reveal a surprisingly rich and diverse set of jailbreak tactics that are often more sophisticated than those documented in existing datasets.
- Composition is Key to Adversarial Strength: Combining multiple jailbreak tactics significantly increases the diversity and success rate of adversarial attacks, highlighting the need for comprehensive red-teaming.
- Data Scale and Diversity Matter for Safety: The WILDJAILBREAK dataset demonstrates that training with a large and diverse mix of both vanilla and adversarial harmful/benign queries is crucial for achieving robust safety without sacrificing general capabilities or causing over-refusal.
- Balancing Safety and Capabilities is Achievable: The research shows that a carefully constructed safety dataset, like WILDJAILBREAK, can lead to models that are both safe and generally capable, achieving a Pareto optimality.
Key Results
- WILDTEAMING discovered 5.7K unique clusters of novel jailbreak tactics.
- It generated up to 4.6x more diverse and successful adversarial attacks compared to state-of-the-art methods.
- The WILDJAILBREAK dataset contains 262K prompt-response pairs.
- Models trained with WILDJAILBREAK showed substantial safety improvements with minimal impact on general capabilities. For instance, a model trained with WILDJAILBREAK showed an 85.1% relative improvement on HARMBENCH compared to a baseline trained on original Tulu2Mix.
- The study found that all four components of WILDJAILBREAK (vanilla/adversarial x harmful/benign) are indispensable for achieving balanced safety.
Limitations and Future Work
- Tactics Reflect Source Data: The mined jailbreak tactics are limited by the scope of the source datasets (LMSYS-1M and WILDCHAT) and may not capture the full spectrum of real-world jailbreak tactics.
- Synthetic Data Nuances: While WILDJAILBREAK is synthetic, the adversarial prompts generated by LLMs might inherit inherent styles and filtering heuristics from their source models, not perfectly mirroring in-the-wild user queries.
- Context-Dependent Harm: Certain harmful behaviors are inherently context-dependent and might require external layers of defense beyond the model itself.
Future work could involve exploring synthetic data generation that more closely aligns with human-written attacks and developing more sophisticated safety alignment approaches. The authors also call for more open engagement and sharing of safety resources within the research community.
Summary updated