Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation

Tharindu Kumarage, Ninareh Mehrabi, Anil Ramakrishna, Xinyan Zhao, Richard S. Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Charith Peris

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet ensuring their safety remains a critical challenge. Current safety mechanisms often rely on surface-level pattern matching, which frequently leads to two undesirable outcomes: over-refusal of harmless prompts (e.g., refusing to explain how to "kill a process" in computing) and jailbreak vulnerability, where adversarial prompts bypass safety filters.

A promising approach to mitigate these issues is "safety reasoning," where models are trained to explicitly deliberate on safety policies before generating a response. However, the primary bottleneck for this approach is the scarcity of high-quality training data. Human-authored Chain-of-Thought (CoT) reasoning is expensive to scale, and single-prompt LLM generations often fail to capture the nuances of complex safety policies. This paper introduces AIDSAFE (Agentic Iterative Deliberation for SAFEty reasoning), a multi-agent framework designed to generate high-quality, policy-embedded synthetic data to bridge this gap.

Methodology: The AIDSAFE Framework

The AIDSAFE framework employs a three-stage pipeline to generate synthetic safety reasoning data, moving beyond simple single-shot generation to a more robust deliberative process.

  1. Initialization: The process begins with an agent decomposing the user's intent—distinguishing between explicit and implicit requests—and generating an initial "seed" CoT and response based on provided safety policies.
  2. Agent Deliberation: Multiple agents engage in iterative rounds of debate. These agents critique the initial reasoning, suggesting corrections or additions to ensure comprehensive coverage of all relevant safety guidelines. This iterative process significantly reduces hallucinations and improves adherence to nuanced policies.
  3. Refinement: A final "refiner" agent acts as an impartial judge. It filters out redundant, deceptive, or inconsistent thoughts to produce a concise, high-quality final CoT that is both faithful to the policies and helpful to the user.
The AIDSAFE pipeline: Initialization, Agent Deliberation, and Refinement stages for generating safety-embedded CoT data.

To further enhance preference alignment via Direct Preference Optimization (DPO), the authors introduce a "supplemental ear-whisperer" recipe. Instead of standard sampling, an adversarial agent is used to inject "bad beliefs" into the model's input. This creates intentionally flawed "rejected" CoT samples, providing a much stronger training signal for the model to distinguish between sound and deceptive reasoning.

Experimental Evaluation

The framework was evaluated using a diverse set of benchmarks and datasets:

  • Training Data: 5,000 unique prompts from BeaverTails (covering 14 harm categories) and 5,000 general utility prompts from Alpagsus.
  • Safety Benchmarks: WildChat-1M for out-of-domain safety and StrongREJECT for evaluating robustness against 12 distinct jailbreak techniques.
  • Utility Benchmarks: XSTest for measuring over-refusal accuracy and MMLU to ensure general knowledge retention.

Results and Discussion

The impact of AIDSAFE-generated data is substantial across all key metrics. The multi-agent deliberation process resulted in a 10.9% improvement in policy adherence (faithfulness) compared to standard zero-shot LLM generations.

One of the most significant findings is the model's improved generalization. For Mixtral, out-of-domain safety on the WildChat dataset skyrocketed from 31% to 85.95%. Furthermore, robustness against jailbreak attacks improved from 51% to 94%. Crucially, these safety gains did not come at the cost of utility; the models maintained high over-refusal accuracy, demonstrating that they learned to apply principles rather than simply memorizing "bad words."

Safety Rate vs. Overrefusal Accuracy. AIDSAFE-trained models (DB) show significant improvements in safety while maintaining high overrefusal accuracy compared to base and zero-shot models.

Pairwise evaluations by auto-graders (Claude-3 and Command) further confirmed these results, with AIDSAFE-generated reasoning being preferred over standard LLM reasoning by a margin of over 64%.

Conclusion

AIDSAFE demonstrates that multi-agent deliberation is a powerful tool for overcoming the data bottleneck in LLM safety reasoning. By generating high-quality, policy-embedded CoT data, the framework enables models to achieve superior safety generalization and jailbreak robustness while mitigating the common safety-utility trade-off. Future work will focus on expanding policy coverage and exploring more dynamic agent configurations to further refine reasoning quality.

Summary updated

BibTeX
@inproceedings{kumarage2025towards,
  author = {Tharindu Kumarage and Ninareh Mehrabi and Anil Ramakrishna and Xinyan Zhao and Richard S. Zemel and Kai-Wei Chang and Aram Galstyan and Rahul Gupta and Charith Peris},
  title = {{Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure