Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation

Tharindu Kumarage, Ninareh Mehrabi, Anil Ramakrishna, Xinyan Zhao, Richard S. Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Charith Peris

TL;DR

This paper introduces AIDSAFE, a novel multi-agent deliberation framework for generating high-quality, policy-embedded Chain-of-Thought (CoT) data to improve LLM safety reasoning. AIDSAFE significantly enhances LLM safety generalization and jailbreak robustness by creating better training data, and a novel "ear-whisperer" agent recipe improves preference data for alignment.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly being trained to "reason" about safety policies before generating responses, a paradigm known as safety reasoning. This approach aims to mitigate issues like over-refusal and jailbreak vulnerabilities. However, creating the necessary high-quality, policy-embedded Chain-of-Thought (CoT) datasets for this training is a major bottleneck. Human annotation is prohibitively expensive, and LLM-generated CoTs can be inaccurate, deceptive, or conflict with policies due to their inherent fuzziness. This paper tackles the challenge of efficiently generating reliable and comprehensive policy-embedded CoT data for LLM safety training.

Methods

The core of the paper is the AIDSAFE framework, which uses a multi-agent, iterative deliberation process for data generation.

  1. Initialization: An LLM agent decomposes the user query's intents and generates an initial CoT and response based on a given set of safety policies.
  2. Deliberation Stage: Multiple agents iteratively refine the CoT and response. Each agent evaluates the current reasoning, suggests corrections or additions, and updates the response. This continues until a consensus is reached or a deliberation budget is exhausted. This stage is inspired by multi-agent debate frameworks.
  3. Refinement Stage: A dedicated "refiner" agent acts as an impartial evaluator, filtering out redundant, deceptive, or policy-inconsistent thoughts from the deliberation process to ensure high-quality outputs.

For preference data creation (needed for alignment stages like DPO), the authors introduce a supplemental recipe using an "ear-whisperer" agent. This agent generates "bad beliefs" or deceptive prefixes to prepend to prompts when sampling rejected CoTs. This creates a clearer distinction between selected and rejected responses, improving the effectiveness of preference learning.

Datasets

  • BeaverTails dataset: Used for generating safety reasoning CoTs. This dataset is a benchmark for safety training and alignment research, covering 14 harm categories. 5,000 prompts were subsampled for this study.
  • DataAdvisor datasets: An additional 8,000 CoT and response pairs were generated from these datasets for safety training.
  • Alpaca dataset (Alpagsus): 5,000 general prompts were used to generate CoTs for balancing safety with general utility. For these, only the "Respectfulness and Helpfulness" policy was applied.
  • XSTest dataset: Used for evaluating over-refusal.
  • StrongREJECT dataset: Used for evaluating jailbreak robustness.
  • MMLU dataset: A subset of five categories was used for evaluating model utility.

Key Insights and Ideas

  • Multi-agent deliberation is key: The iterative refinement process involving multiple agents significantly improves the quality and policy adherence of generated CoTs compared to single-LLM generation.
  • Refinement is crucial: A dedicated refiner agent is essential for filtering out noise and ensuring the final CoTs are concise, coherent, and strictly aligned with safety policies.
  • Distinguishing "rejected" CoTs is hard: Standard sampling methods struggle to create meaningfully distinct rejected responses for preference learning. The "ear-whisperer" agent provides a novel solution by injecting adversarial beliefs.
  • Safety reasoning data improves generalization: Fine-tuning models with AIDSAFE-generated CoTs leads to better safety performance on both in-domain and out-of-domain datasets, especially for models not initially trained for safety.
  • Balancing safety and utility: While safety training can sometimes lead to over-refusal or utility degradation, AIDSAFE's approach, particularly with the ear-whisperer for preference data, helps strike a better balance.

Key Results

  • AIDSAFE-generated CoTs achieved superior policy adherence and reasoning quality compared to single LLM generations.
  • Fine-tuning open-source LLMs (Mixtral and Qwen 2.5) with AIDSAFE data significantly improved:
  • In-domain safety: 20% increase for Mixtral (76% to 96%).
  • Out-of-domain safety: 54.95% increase for Mixtral (31.00% to 85.95%).
  • Jailbreak robustness: High safety rates achieved (94.04% for Mixtral, 95.39% for Qwen).
  • The "ear-whisperer" recipe for preference data creation showed a substantial distribution shift between selected and rejected responses, improving alignment effectiveness.
  • DPO training with this data further improved safety performance and jailbreak robustness, while mitigating some of the over-refusal issues seen with standard sampling.

Limitations and Future Work

  • Policy Coverage: The framework currently uses a predefined set of five safety policies. Expanding this to include more policies could enhance reasoning capabilities for a broader spectrum of safety challenges.
  • Model and Agent Constraints: Experiments were limited to Mixtral 8x22B as the agent and a two-agent deliberation process. Exploring other LLMs and more complex multi-agent setups could yield further improvements.
  • SFT Setup: The SFT experiments did not include an initial warm-up phase with general CoTs, which could have further optimized performance.
  • Potential Interruptions: Highly safety-trained LLMs with strict guardrails might refuse to engage with prompts requiring reasoning about harmful content, potentially interrupting the deliberation process.
  • Ear-Whisperer Effectiveness: The ear-whisperer agent's effectiveness diminishes for already highly safety-trained LLMs, making it harder to generate distinct "rejected" CoTs.
  • Ethical Considerations: The potential for misuse of the ear-whisperer agent to generate malicious outputs is acknowledged. Careful design and ethical considerations are crucial for policy development.

Future work could involve exploring more sophisticated multi-agent architectures, expanding policy coverage, and investigating methods to further enhance the ear-whisperer's effectiveness on highly robust models.

Summary updated

Abstract

Safety reasoning is a recent paradigm where LLMs reason over safety policies before generating responses, thereby mitigating limitations in existing safety measures such as over-refusal and jailbreak vulnerabilities. However, implementing this paradigm is challenging due to the resource-intensive process of creating high-quality policy-embedded chain-of-thought (CoT) datasets while ensuring reasoning remains accurate and free from hallucinations or policy conflicts. To tackle this, we propose AIDSAFE: Agentic Iterative Deliberation for Safety Reasoning, a novel data generation recipe that leverages multi-agent deliberation to iteratively expand reasoning on safety policies. A data refiner stage in AIDSAFE ensures high-quality outputs by eliminating repetitive, redundant, and deceptive thoughts. AIDSAFE-generated CoTs provide a strong foundation for supervised fine-tuning (SFT)-based safety training. Additionally, to address the need of preference data in alignment stages, such as DPO training, we introduce a supplemental recipe that uses belief augmentation to create distinct selected and rejected CoT samples. Our evaluations demonstrate that AIDSAFE-generated CoTs achieve superior policy adherence and reasoning quality. Consequently, we show that fine-tuning open-source LLMs on these CoTs can significantly improve safety generalization and jailbreak robustness while maintaining acceptable utility and over-refusal accuracy.
BibTeX
@inproceedings{kumarage2025towards,
  author = {Tharindu Kumarage and Ninareh Mehrabi and Anil Ramakrishna and Xinyan Zhao and Richard S. Zemel and Kai-Wei Chang and Aram Galstyan and Rahul Gupta and Charith Peris},
  title = {{Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure