Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking

Jianjun Zhu, Lingyong Yan, Shuaiqiang Wang, Dawei Yin, Lei Sha

Abstract

Large Language Models (LLMs) are increasingly vulnerable to "jailbreaking"—adversarial prompts designed to bypass safety filters and elicit harmful content. Traditional defenses, such as external guardrails or supervised fine-tuning for hard refusals, often suffer from high system complexity or excessive "over-refusal" of benign prompts. This paper introduces Reasoning-to-Defend (R2D), a novel training paradigm that leverages the inherent reasoning capabilities of LLMs to evaluate prompt safety. By distilling safety-aware reasoning trajectories and utilizing "pivot tokens" to guide internal deliberation, R2D significantly reduces the success rate of jailbreak attacks without compromising general model performance or helpfulness.

Introduction

As Large Language Models (LLMs) become more integrated into daily workflows, ensuring their safety against adversarial attacks is paramount. Jailbreaking remains a persistent threat, where attackers craft prompts to force models into generating prohibited content. Current defense strategies generally fall into two categories: external guardrails (separate models that monitor inputs and outputs) and supervised fine-tuning (SFT) for direct refusal. However, guardrails introduce significant latency and complexity, while SFT often leads to "over-refusal," where models reject safe queries (e.g., "How do I kill a Python process?") due to a lack of contextual understanding.

This research proposes a shift from reactive refusal to proactive reasoning. Instead of a knee-jerk response, can a model "think" through the intent and safety of a prompt before answering? Reasoning-to-Defend (R2D) addresses this by training models to use internal reasoning trajectories to assess safety, allowing for more nuanced and robust defenses.

Methodology

The R2D framework moves the defense mechanism from the final output level to the internal reasoning level. It is built upon two primary technical pillars:

1. Safety-Aware Reasoning Distillation (SwaRD)

To teach smaller LLMs how to deliberate on safety, the authors utilize a powerful teacher model, DeepSeek-R1-70B. This model generates detailed, step-by-step "thought processes" for both benign and malicious prompts. These trajectories are then distilled into smaller, non-reasoning models (such as Llama-3 or Mistral). This process trains the student models to open a <think> block and evaluate the context and potential risks of a prompt before generating a final response.

2. Contrastive Pivot Optimization (CPO)

To ensure the model's internal monologue remains aligned with safety goals, R2D introduces "pivot tokens": [SAFE], [UNSAFE], and [RETHINK]. These tokens act as checkpoints within the reasoning chain.

  • [SAFE] / [UNSAFE]: The model predicts these to categorize its current assessment of the prompt.
  • [RETHINK]: If the model identifies a potential safety violation during its reasoning, this token triggers a self-correction loop, forcing the model to adjust its strategy before the user sees the output.

CPO uses a novel loss function to explicitly train the model to predict the correct pivot token by contrasting it against incorrect alternatives, sharpening the model's real-time safety perception.

Experimental Evaluation

The authors evaluated R2D across several dimensions, including robustness, over-refusal, and general capabilities.

  • Training Data: The models were trained on 52,000 helpful instructions from the Alpaca dataset and 520 jailbreak instructions from AdvBench, with reasoning trajectories synthesized by DeepSeek-R1-70B.
  • Jailbreak Benchmarks: Robustness was tested using JailbreakBench (100 prompts) and HarmBench (400 harmful behaviors) against various attack vectors, including PAIR, GCG, and AutoDAN.
  • Over-Refusal Benchmarks: The XSTest dataset (250 safe but sensitive prompts) was used to measure whether the defense caused the model to become overly restrictive.
  • General Capabilities: Performance was monitored on standard benchmarks like MMLU, ARC, and BoolQ to ensure no "capability tax" was paid for the added safety.

Results and Key Insights

Significant Reduction in Attack Success

R2D demonstrated exceptional performance in neutralizing jailbreaks. On JailbreakBench, the defense reduced the Attack Success Rate (ASR) by an average of 56% compared to undefended models. On HarmBench, the ASR for ZeroShot and FewShot attacks dropped to nearly 0% across multiple architectures, including Llama-3 and Qwen.

Superiority Over External Guardrails

When compared to "Erase-and-Check"—a strong baseline utilizing an external Llama-Guard model—R2D achieved a 17% lower ASR. This suggests that internal reasoning is a more effective filter than external monitoring for complex adversarial prompts.

Mitigating Over-Refusal

By reasoning through the context of a prompt, R2D avoids the pitfalls of keyword-based refusal. The model can distinguish between a violent act and a technical term (like "killing a process"), allowing it to remain helpful where traditional SFT-based models might fail.

Performance and Latency

Crucially, R2D maintains the model's general intelligence, with no significant drop in performance on MMLU or ARC. The system also exhibits "Smart Latency": benign queries are processed quickly with minimal reasoning, while harmful or ambiguous queries trigger deeper safety deliberation and [RETHINK] loops, concentrating computational resources where they are most needed.

Conclusion and Future Work

Reasoning-to-Defend proves that explicit, step-by-step reasoning is a powerful tool for safety alignment. By allowing models to reflect on their own thought processes, R2D creates a defense that is both more robust against attacks and more helpful to users.

While highly effective against single-turn jailbreaks, the authors note that multi-turn attacks (such as the Crescendo attack) remain a challenge. Future research will explore integrating safety-aware reasoning with Reinforcement Learning (RL) and extending these techniques to multi-modal models to further harden LLMs against evolving adversarial threats.

Summary updated

Abstract

Large Reasoning Models (LRMs) have recently demonstrated impressive performances across diverse domains.However, how the safety of Large Language Models (LLMs) benefits from enhanced reasoning capabilities against jailbreak queries remains unexplored.To bridge this gap, in this paper, we propose Reasoningto-Defend (R2D), a novel training paradigm that integrates a safety-aware reasoning mechanism into LLMs' generation process.This enables self-evaluation at each step of the reasoning process, forming safety PIVOT TOKENS as indicators of the safety status of responses.Furthermore, in order to improve the accuracy of predicting PIVOT TOKENS, we propose Contrastive Pivot Optimization (CPO), which enhances the model's perception of the safety status of given dialogues.LLMs dynamically adjust their response strategies during reasoning, significantly enhancing their safety capabilities defending jailbreak attacks.Extensive experiments demonstrate that R2D effectively mitigates various attacks and improves overall safety, while maintaining the original performances.This highlights the substantial potential of safety-aware reasoning in improving robustness of LRMs and LLMs against various jailbreaks.
BibTeX
@misc{zhu2025reasoning,
  author = {Jianjun Zhu and Lingyong Yan and Shuaiqiang Wang and Dawei Yin and Lei Sha},
  title = {{Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking}},
  year = {2025},
  eprint = {2502.12970},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.12970},
}
Figure