Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet ensuring their safety remains a critical challenge. Current safety mechanisms often rely on surface-level pattern matching, which frequently leads to two undesirable outcomes: over-refusal of harmless prompts (e.g., refusing to explain how to "kill a process" in computing) and jailbreak vulnerability, where adversarial prompts bypass safety filters.
A promising approach to mitigate these issues is "safety reasoning," where models are trained to explicitly deliberate on safety policies before generating a response. However, the primary bottleneck for this approach is the scarcity of high-quality training data. Human-authored Chain-of-Thought (CoT) reasoning is expensive to scale, and single-prompt LLM generations often fail to capture the nuances of complex safety policies. This paper introduces AIDSAFE (Agentic Iterative Deliberation for SAFEty reasoning), a multi-agent framework designed to generate high-quality, policy-embedded synthetic data to bridge this gap.
Methodology: The AIDSAFE Framework
The AIDSAFE framework employs a three-stage pipeline to generate synthetic safety reasoning data, moving beyond simple single-shot generation to a more robust deliberative process.
- Initialization: The process begins with an agent decomposing the user's intent—distinguishing between explicit and implicit requests—and generating an initial "seed" CoT and response based on provided safety policies.
- Agent Deliberation: Multiple agents engage in iterative rounds of debate. These agents critique the initial reasoning, suggesting corrections or additions to ensure comprehensive coverage of all relevant safety guidelines. This iterative process significantly reduces hallucinations and improves adherence to nuanced policies.
- Refinement: A final "refiner" agent acts as an impartial judge. It filters out redundant, deceptive, or inconsistent thoughts to produce a concise, high-quality final CoT that is both faithful to the policies and helpful to the user.
To further enhance preference alignment via Direct Preference Optimization (DPO), the authors introduce a "supplemental ear-whisperer" recipe. Instead of standard sampling, an adversarial agent is used to inject "bad beliefs" into the model's input. This creates intentionally flawed "rejected" CoT samples, providing a much stronger training signal for the model to distinguish between sound and deceptive reasoning.
Experimental Evaluation
The framework was evaluated using a diverse set of benchmarks and datasets:
- Training Data: 5,000 unique prompts from BeaverTails (covering 14 harm categories) and 5,000 general utility prompts from Alpagsus.
- Safety Benchmarks: WildChat-1M for out-of-domain safety and StrongREJECT for evaluating robustness against 12 distinct jailbreak techniques.
- Utility Benchmarks: XSTest for measuring over-refusal accuracy and MMLU to ensure general knowledge retention.
Results and Discussion
The impact of AIDSAFE-generated data is substantial across all key metrics. The multi-agent deliberation process resulted in a 10.9% improvement in policy adherence (faithfulness) compared to standard zero-shot LLM generations.
One of the most significant findings is the model's improved generalization. For Mixtral, out-of-domain safety on the WildChat dataset skyrocketed from 31% to 85.95%. Furthermore, robustness against jailbreak attacks improved from 51% to 94%. Crucially, these safety gains did not come at the cost of utility; the models maintained high over-refusal accuracy, demonstrating that they learned to apply principles rather than simply memorizing "bad words."
Pairwise evaluations by auto-graders (Claude-3 and Command) further confirmed these results, with AIDSAFE-generated reasoning being preferred over standard LLM reasoning by a margin of over 64%.
Conclusion
AIDSAFE demonstrates that multi-agent deliberation is a powerful tool for overcoming the data bottleneck in LLM safety reasoning. By generating high-quality, policy-embedded CoT data, the framework enables models to achieve superior safety generalization and jailbreak robustness while mitigating the common safety-utility trade-off. Future work will focus on expanding policy coverage and exploring more dynamic agent configurations to further refine reasoning quality.
Summary updated