TL;DR
This paper tackles the critical safety shortcomings of the popular DeepSeek-R1 large language model, particularly in Chinese contexts. The authors conduct a thorough safety evaluation of its distilled versions, identify vulnerabilities, and then implement targeted safety enhancements, resulting in models that are significantly safer without sacrificing reasoning abilities.
What Problem Does This Paper Address?
The DeepSeek-R1 model, lauded for its impressive reasoning and open-source nature, has been found to have significant safety issues. Previous research has shown a 100% attack success rate on harmful prompts, and security firms have identified critical vulnerabilities. While some work has addressed these issues in Chinese contexts, the safety of the distilled models within the R1 series, which are crucial for practical deployment due to their efficiency, remained largely unevaluated. This paper aims to fill that gap by comprehensively assessing the safety of these distilled models in Chinese contexts and then developing methods to enhance their safety.
Methods
The researchers employed a two-pronged approach:
- Safety Evaluation: They used the comprehensive Chinese safety benchmark, CHiSafetyBench, to evaluate the safety capabilities of six distilled DeepSeek-R1 models. This involved assessing their ability to identify and refuse harmful content across various dimensions like discrimination, violation of values, and infringement of rights.
- Safety Enhancement: Based on the identified vulnerabilities, they implemented supervised fine-tuning. This involved creating a custom fine-tuning dataset of approximately 50,000 samples, including instructions for critical safety areas and chain-of-thought reasoning samples to balance safety with performance. This fine-tuning was applied to the entire DeepSeek-R1 model series.
Datasets
- CHiSafetyBench: This is a comprehensive Chinese safety benchmark used for evaluating the models' safety capabilities. It includes tasks for risk content identification (multiple-choice questions) and risky question refusal.
- Custom Fine-tuning Dataset: Approximately 50,000 samples were created, comprising instructions for critical safety areas (discrimination, infringement) and chain-of-thought reasoning samples.
Key Insights and Ideas
- Distillation's Double-Edged Sword: While distillation makes models more efficient, it can inadvertently degrade their safety performance, especially in specific linguistic contexts like Chinese.
- Chinese Context Matters: Safety vulnerabilities can be amplified or manifest differently in non-English contexts, necessitating specialized evaluation and enhancement.
- Targeted Enhancement is Effective: Supervised fine-tuning with carefully curated safety data can significantly improve model safety without a substantial hit to reasoning capabilities.
- Open-Sourcing for Community Benefit: Releasing the safety-enhanced models provides a valuable resource for the research community to build upon.
Key Results
- Safety Degradation Post-Distillation: The study found that distilled models generally showed a declining trend in safety performance compared to their base models, particularly in risk content identification and refusal of risky questions.
- Significant Safety Improvements After Enhancement: The safety-enhanced models demonstrated substantial improvements. For instance, in the risk content identification task, accuracy improved by 10.72%. In the refusal-to-answer task, RR-1 and RR-2 increased by 18.39% and 14.72% respectively, while HR decreased by 1.45%.
- Reasoning Capabilities Maintained: Crucially, the safety enhancements did not significantly impair the models' reasoning abilities. In some cases, reasoning performance even saw marginal improvements.
- Open-Sourced Models: The authors have open-sourced their safety-enhanced DeepSeek-R1 models to foster further research.
Limitations and Future Work
- Model Size and Data Influence: The paper notes that improvements in safety performance for some smaller models after distillation might be related to their size and the specific Chinese instructions used in the SFT data.
- Continued Vigilance: While significant improvements were made, the paper implies that ongoing research and optimization of DeepSeek models are necessary.
- Future Research: The open-sourced models are intended to serve as a valuable resource for future research and collaborative optimization.
Summary updated