Safety Evaluation of DeepSeek Models in Chinese Contexts

Wenjing Zhang, Xuejiao Lei, Zhaoxiang Liu, Ning Wang, Zhenhong Long, Peijun Yang, Jiaojiao Zhao, Minjie Hua, Chaoyang Ma, Kai Wang, Shiguo LianarXiv 2025 · DOI · OpenReview

TL;DR

This paper introduces CHiSafetyBench, a novel Chinese-specific benchmark for evaluating the safety of large language models. It reveals significant safety deficiencies in DeepSeek-R1 and DeepSeek-V3 within Chinese contexts, highlighting a critical gap in current safety research which has largely focused on English.

What Problem Does This Paper Address?

The rapid advancement and widespread adoption of powerful large language models (LLMs) like the DeepSeek series have brought immense benefits, but also significant safety concerns. While DeepSeek models boast impressive reasoning capabilities and an open-source approach, previous research has indicated critical vulnerabilities, with one study showing a 100% attack success rate for DeepSeek-R1 on harmful prompts. However, the vast majority of safety evaluations have been conducted in English, leaving a crucial gap in understanding how these models perform in non-English contexts, particularly in Chinese, a language with unique cultural nuances and societal norms. This paper aims to bridge this gap by providing a comprehensive safety evaluation of DeepSeek models specifically within the Chinese linguistic and cultural landscape.

Methods

The core of this research is the development and application of CHiSafetyBench, a new benchmark designed to systematically evaluate LLM safety in Chinese contexts. This benchmark is built upon the "Basic Safety Requirements for Generative Artificial Intelligence Services" standard issued by the Chinese government. CHiSafetyBench employs two primary evaluation tasks:

  1. Multiple-choice questions for risk content identification: This task assesses the model's ability to correctly identify and flag harmful or inappropriate content. The primary metric used is Accuracy (ACC).
  2. Risky questions for refusal to answer: This task evaluates the model's capability to refuse to answer harmful queries and, ideally, provide responsible guidance. Metrics include Rejection Rate (RR-1, RR-2) and Harm Rate (HR).

The study also includes a comparative analysis, evaluating DeepSeek-R1 and DeepSeek-V3 against a selection of other well-regarded Chinese-capable LLMs, including models from the Baichuan, ChatGLM, Qwen, and Yi series.

Datasets

The paper introduces CHiSafetyBench as its primary evaluation dataset. This benchmark is specifically designed for Chinese contexts and covers five major safety areas: discrimination, violation of values, commercial violations, infringement of rights, and security requirements for specific services. The paper does not detail training datasets for the models evaluated, as the focus is on their performance on the newly introduced safety benchmark.

Key Insights and Ideas

  • Critical Safety Gap in Chinese Contexts: The most significant insight is the confirmation of substantial safety deficiencies in DeepSeek models when evaluated in Chinese, a domain largely overlooked by prior research.
  • CHiSafetyBench's Importance: The introduction of CHiSafetyBench is a key contribution, providing a much-needed standardized tool for assessing LLM safety in a non-English, culturally specific environment.
  • DeepSeek's Persistent Weaknesses: Both DeepSeek-R1 and DeepSeek-V3 exhibit notable weaknesses, particularly in categories related to discrimination and value violations, suggesting these are persistent issues not fully addressed by their current safety mechanisms.
  • Comparative Performance: The study highlights that while DeepSeek models are powerful, they lag behind top-performing models like the Qwen series in several safety metrics within the Chinese context.
  • Refusal Capabilities Need Improvement: Beyond simply identifying harmful content, the models struggle with effectively refusing risky queries and providing appropriate guidance, indicating room for improvement in their refusal mechanisms.

Key Results

The experimental results reveal that:

  • Risk Content Identification (ACC):
    • DeepSeek-R1 achieved an overall ACC of 71.14%, which is 19.72% lower than the top-performing Qwen1.5-72B-Chat.
  • DeepSeek-V3 achieved an overall ACC of 84.17%, still 6.96% lower than Qwen1.5-72B-Chat.
  • Both models showed particular weakness in the discrimination category, with DeepSeek-R1 scoring 50.22% and DeepSeek-V3 scoring 66.96%.
  • Refusal to Answer (RR-1, RR-2, HR):
    • DeepSeek-R1 and DeepSeek-V3 showed relatively low rejection rates (RR-1 and RR-2) for risky questions, with DeepSeek-R1 at 67.60% and DeepSeek-V3 at 59.83%. These are significantly lower than the best-performing Qwen models.
  • While their harm rates (HR) were relatively low (0.65% for R1, 0.43% for V3), indicating they don't frequently generate harmful outputs, their ability to refuse and guide is lacking.
  • DeepSeek-R1 demonstrated a slight improvement over DeepSeek-V3 in overall refusal capabilities (RR-1 and RR-2), but with a minor increase in harm rate.

Qualitative examples provided in the paper (Figures 5-7) illustrate specific instances where DeepSeek models misjudge sensitive topics, fail to refuse harmful prompts, or even suggest problematic actions, contrasting with the more robust and safe responses of models like Qwen1.5-32B-Chat.

Limitations and Future Work

The authors acknowledge that any evaluation benchmark inherently has limitations, including biases introduced by the selection of test samples, data distribution characteristics, and the setting of evaluation criteria. They state their intention to continuously optimize CHiSafetyBench to enhance its comprehensiveness and reliability. Future work will involve promptly updating the evaluation results and the benchmark itself to provide the community with the most accurate and up-to-date safety assessments of LLMs in Chinese contexts.

Summary updated

Abstract

Recently, the DeepSeek series of models, leveraging their exceptional reasoning capabilities and open-source strategy, is reshaping the global AI landscape. Despite these advantages, they exhibit significant safety deficiencies. Research conducted by Robust Intelligence, a subsidiary of Cisco, in collaboration with the University of Pennsylvania, revealed that DeepSeek-R1 has a 100\% attack success rate when processing harmful prompts. Additionally, multiple safety companies and research institutions have confirmed critical safety vulnerabilities in this model. As models demonstrating robust performance in Chinese and English, DeepSeek models require equally crucial safety assessments in both language contexts. However, current research has predominantly focused on safety evaluations in English environments, leaving a gap in comprehensive assessments of their safety performance in Chinese contexts. In response to this gap, this study introduces CHiSafetyBench, a Chinese-specific safety evaluation benchmark. This benchmark systematically evaluates the safety of DeepSeek-R1 and DeepSeek-V3 in Chinese contexts, revealing their performance across safety categories. The experimental results quantify the deficiencies of these two models in Chinese contexts, providing key insights for subsequent improvements. It should be noted that, despite our efforts to establish a comprehensive, objective, and authoritative evaluation benchmark, the selection of test samples, characteristics of data distribution, and the setting of evaluation criteria may inevitably introduce certain biases into the evaluation results. We will continuously optimize the evaluation benchmark and periodically update this report to provide more comprehensive and accurate assessment outcomes. Please refer to the latest version of the paper for the most recent evaluation results and conclusions.
BibTeX
@misc{zhang2025safety,
  author = {Wenjing Zhang and Xuejiao Lei and Zhaoxiang Liu and Ning Wang and Zhenhong Long and Peijun Yang and Jiaojiao Zhao and Minjie Hua and Chaoyang Ma and Kai Wang and Shiguo Lian},
  title = {{Safety Evaluation of DeepSeek Models in Chinese Contexts}},
  year = {2025},
  eprint = {2502.11137},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.11137},
}
Figure