RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability

Yichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, Yinpeng Dong

TL;DR

RealSafe-R1 introduces a safety-aligned pipeline for DeepSeek-R1 distilled models that successfully mitigates harmful outputs without degrading their complex reasoning capabilities. By fine-tuning models on 15k safety-aware reasoning trajectories generated by DeepSeek-R1 itself, the authors implement robust safety guardrails against jailbreaks while completely avoiding the typical "safety tax" that plagues traditional alignment methods.

What Problem Does This Paper Address?

Large Reasoning Models (LRMs) like OpenAI’s o1 and DeepSeek-R1 have achieved massive breakthroughs in complex tasks like mathematics and coding due to their internalized, multi-step Chain-of-Thought (CoT) processing. However, the open-source nature of the DeepSeek-R1 series has highlighted a glaring vulnerability: these models frequently comply with malicious queries and elaborate jailbreak attacks.

Traditional safety alignment techniques (like Supervised Fine-Tuning on short-form refusal datasets) fail when applied to LRMs. Forcing a model accustomed to long-form reasoning to output a blunt, short-form refusal introduces a severe format mismatch. This mismatch disrupts the model's generation style, creating a harsh trade-off where increasing safety drastically compromises reasoning performance. The authors set out to solve this alignment dilemma by keeping the safety training data strictly within the model's native long-form reasoning distribution.

Methods

The authors leverage a concept called deliberative alignment combined with data distillation. They observed an interesting paradox: DeepSeek-R1 often possesses "latent safety awareness." During its internal thinking phase, it frequently identifies the risks or illegal nature of a malicious prompt, yet it proceeds to fulfill the request anyway in its final response.

To fix this, the authors prompted DeepSeek-R1 with explicit instructions to actively analyze underlying risks during its thinking process and, if a violation is found, conclude with a clear, explicit refusal (using phrases like "I am sorry, but I cannot assist with...").

Using these generated safety-aware reasoning trajectories, they performed standard Supervised Fine-Tuning (SFT) on the distilled R1 models using the LLaMA-Factory framework. The models were trained for just one epoch, making the optimization computationally lightweight yet highly effective.

Datasets

  • Training Dataset: A newly synthesized dataset consisting of 15k safety-aware reasoning trajectories. This includes 10k direct harmful queries filtered from PKU-SafeRLHF and 5k adversarial jailbreak prompts sourced from JailbreakV-28k.
  • Evaluation Benchmarks:
  • General Capabilities: MATH-500, AIME 2024, GPQA-Diamond, LiveCodeBench, and TruthfulQA.
  • Safety Capabilities: StrongREJECT (subjected to PAIR and PAP-misrepresentation jailbreak attacks), XSTest (covering both safe and unsafe prompts), and a harmful subset of WildChat.

Key Insights and Ideas

  • The Latent Safety Goldmine: LRMs often already know when a query is unsafe. As shown in the paper's analysis, DeepSeek-R1 explicitly notes the ethical and legal implications of a prompt within its <think> tags but lacks the behavioral specification to act on it. Alignment is not about teaching the model what is dangerous, but teaching it to act on its own internal realizations.
  • Format Preservation Eliminates the "Safety Tax": By ensuring the training targets look exactly like R1's native long-form reasoning trajectories (maintaining the <think> structure), the model's core reasoning capabilities remain entirely uncompromised.
  • The Scale Paradox: The evaluation revealed a scale-wise trend: larger models naturally tend to refuse less, regardless of whether they are pre- or post-alignment. This suggests an inverse correlation between model size and refusal likelihood, meaning larger reasoning models require even more rigorous alignment strategies.
Figure 2: Visualization of model behavior on safety-critical prompts. The figure presents the distribu- tion of response types—Full Refusal, Partial Refusal, and Full Compliance—for both DeepSeek-R1 and RealSafe-R1 models on safe and unsafe prompts from XSTest, as well as unsafe prompts from WildChat. RealSafe-R1 consistently exhibits stronger safety awareness than DeepSeek-R1 across all model sizes, with substantially higher refusal rates on both safe and unsafe prompts. In addition, larger models—regardless of alignment—tend to refuse less, suggesting an inverse correlation be- tween model size and refusal likelihood.

Key Results

  • Robust Jailbreak Defense: On the StrongREJECT benchmark, the RealSafe-R1-32B model dramatically lowered harmful compliance scores under PAIR and PAP adversarial attacks from 0.73 and 0.61 down to 0.27 and 0.10, respectively.
  • Uncompromised Reasoning: RealSafe-R1 maintained or even slightly improved upon the performance of the baseline models across all standard benchmarks. For example, RealSafe-R1-14B achieved a score of 71.43 on AIME 2024 (compared to the base model's 66.67) and boosted its truthfulness score on TruthfulQA to 66.95 (compared to 59.77).
Figure 3: A comparison of safety responses between DeepSeek-R1 and RealSafe-R1 on harmful and jailbreak prompts.

Limitations and Future Work

The primary limitation identified is the phenomenon of over-refusal on benign edge cases. Because the safety dataset lacked benign queries that happen to contain sensitive keywords, the aligned models can occasionally be overly conservative. For example, a benign prompt like "How do I best execute a risky plan?" might trigger a false-alarm refusal simply due to the word "execute".

The authors plan to address this in future iterations by integrating benign and general queries into the training distribution and exploring advanced alignment techniques such as Direct Preference Optimization (DPO) and reinforcement learning with verifiable rewards.

Summary updated

Abstract

Large Reasoning Models (LRMs), such as OpenAI o1 and DeepSeek-R1, have been rapidly progressing and achieving breakthrough performance on complex reasoning tasks such as mathematics and coding. However, the open-source R1 models have raised safety concerns in wide applications, such as the tendency to comply with malicious queries, which greatly impacts the utility of these powerful models in their applications. In this paper, we introduce RealSafe-R1 as safety-aligned versions of DeepSeek-R1 distilled models. To train these models, we construct a dataset of 15k safety-aware reasoning trajectories generated by DeepSeek-R1, under explicit instructions for expected refusal behavior. Both quantitative experiments and qualitative case studies demonstrate the models' improvements, which are shown in their safety guardrails against both harmful queries and jailbreak attacks. Importantly, unlike prior safety alignment efforts that often compromise reasoning performance, our method preserves the models' reasoning capabilities by maintaining the training data within the original distribution of generation. Model weights of RealSafe-R1 are open-source at https://huggingface.co/RealSafe.
BibTeX
@misc{zhang2025realsafe,
  author = {Yichi Zhang and Zihao Zeng and Dongbai Li and Yao Huang and Zhijie Deng and Yinpeng Dong},
  title = {{RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability}},
  year = {2025},
  eprint = {2504.10081},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.10081},
}
Figure