TL;DR
This paper introduces AIDSAFE, a novel multi-agent deliberation framework for generating high-quality, policy-embedded Chain-of-Thought (CoT) data to improve LLM safety reasoning. AIDSAFE significantly enhances LLM safety generalization and jailbreak robustness by creating better training data, and a novel "ear-whisperer" agent recipe improves preference data for alignment.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly being trained to "reason" about safety policies before generating responses, a paradigm known as safety reasoning. This approach aims to mitigate issues like over-refusal and jailbreak vulnerabilities. However, creating the necessary high-quality, policy-embedded Chain-of-Thought (CoT) datasets for this training is a major bottleneck. Human annotation is prohibitively expensive, and LLM-generated CoTs can be inaccurate, deceptive, or conflict with policies due to their inherent fuzziness. This paper tackles the challenge of efficiently generating reliable and comprehensive policy-embedded CoT data for LLM safety training.
Methods
The core of the paper is the AIDSAFE framework, which uses a multi-agent, iterative deliberation process for data generation.
- Initialization: An LLM agent decomposes the user query's intents and generates an initial CoT and response based on a given set of safety policies.
- Deliberation Stage: Multiple agents iteratively refine the CoT and response. Each agent evaluates the current reasoning, suggests corrections or additions, and updates the response. This continues until a consensus is reached or a deliberation budget is exhausted. This stage is inspired by multi-agent debate frameworks.
- Refinement Stage: A dedicated "refiner" agent acts as an impartial evaluator, filtering out redundant, deceptive, or policy-inconsistent thoughts from the deliberation process to ensure high-quality outputs.
For preference data creation (needed for alignment stages like DPO), the authors introduce a supplemental recipe using an "ear-whisperer" agent. This agent generates "bad beliefs" or deceptive prefixes to prepend to prompts when sampling rejected CoTs. This creates a clearer distinction between selected and rejected responses, improving the effectiveness of preference learning.
Datasets
- BeaverTails dataset: Used for generating safety reasoning CoTs. This dataset is a benchmark for safety training and alignment research, covering 14 harm categories. 5,000 prompts were subsampled for this study.
- DataAdvisor datasets: An additional 8,000 CoT and response pairs were generated from these datasets for safety training.
- Alpaca dataset (Alpagsus): 5,000 general prompts were used to generate CoTs for balancing safety with general utility. For these, only the "Respectfulness and Helpfulness" policy was applied.
- XSTest dataset: Used for evaluating over-refusal.
- StrongREJECT dataset: Used for evaluating jailbreak robustness.
- MMLU dataset: A subset of five categories was used for evaluating model utility.
Key Insights and Ideas
- Multi-agent deliberation is key: The iterative refinement process involving multiple agents significantly improves the quality and policy adherence of generated CoTs compared to single-LLM generation.
- Refinement is crucial: A dedicated refiner agent is essential for filtering out noise and ensuring the final CoTs are concise, coherent, and strictly aligned with safety policies.
- Distinguishing "rejected" CoTs is hard: Standard sampling methods struggle to create meaningfully distinct rejected responses for preference learning. The "ear-whisperer" agent provides a novel solution by injecting adversarial beliefs.
- Safety reasoning data improves generalization: Fine-tuning models with AIDSAFE-generated CoTs leads to better safety performance on both in-domain and out-of-domain datasets, especially for models not initially trained for safety.
- Balancing safety and utility: While safety training can sometimes lead to over-refusal or utility degradation, AIDSAFE's approach, particularly with the ear-whisperer for preference data, helps strike a better balance.
Key Results
- AIDSAFE-generated CoTs achieved superior policy adherence and reasoning quality compared to single LLM generations.
- Fine-tuning open-source LLMs (Mixtral and Qwen 2.5) with AIDSAFE data significantly improved:
- In-domain safety: 20% increase for Mixtral (76% to 96%).
- Out-of-domain safety: 54.95% increase for Mixtral (31.00% to 85.95%).
- Jailbreak robustness: High safety rates achieved (94.04% for Mixtral, 95.39% for Qwen).
- The "ear-whisperer" recipe for preference data creation showed a substantial distribution shift between selected and rejected responses, improving alignment effectiveness.
- DPO training with this data further improved safety performance and jailbreak robustness, while mitigating some of the over-refusal issues seen with standard sampling.
Limitations and Future Work
- Policy Coverage: The framework currently uses a predefined set of five safety policies. Expanding this to include more policies could enhance reasoning capabilities for a broader spectrum of safety challenges.
- Model and Agent Constraints: Experiments were limited to Mixtral 8x22B as the agent and a two-agent deliberation process. Exploring other LLMs and more complex multi-agent setups could yield further improvements.
- SFT Setup: The SFT experiments did not include an initial warm-up phase with general CoTs, which could have further optimized performance.
- Potential Interruptions: Highly safety-trained LLMs with strict guardrails might refuse to engage with prompts requiring reasoning about harmful content, potentially interrupting the deliberation process.
- Ear-Whisperer Effectiveness: The ear-whisperer agent's effectiveness diminishes for already highly safety-trained LLMs, making it harder to generate distinct "rejected" CoTs.
- Ethical Considerations: The potential for misuse of the ear-whisperer agent to generate malicious outputs is acknowledged. Careful design and ethical considerations are crucial for policy development.
Future work could involve exploring more sophisticated multi-agent architectures, expanding policy coverage, and investigating methods to further enhance the ear-whisperer's effectiveness on highly robust models.
Summary updated