Abstract
This paper presents a systematic security evaluation of the DeepSeek model series against jailbreak attacks, comparing them to OpenAI’s GPT-3.5 and GPT-4. We identify a unique architectural trade-off: while DeepSeek’s Mixture-of-Experts (MoE) design provides inherent robustness against optimization-heavy attacks, it remains significantly more vulnerable to simple, human-engineered prompts than its proprietary counterparts. Our findings suggest that the sparse activation in MoE models creates a discontinuous optimization landscape for attackers but also leads to an "alignment generalization gap" where safety tuning may not uniformly cover all expert modules.
Introduction
As open-source Large Language Models (LLMs) like DeepSeek R1 begin to rival proprietary giants like GPT-4 in performance, a critical question emerges: are they equally safe? Most "jailbreak" research—the study of prompts designed to bypass safety filters—has focused on dense models like GPT. This study addresses the security gap created by DeepSeek’s unique architecture, specifically investigating whether the sparse Mixture-of-Experts (MoE) approach changes the "attack surface" compared to the Dense Transformer architecture used by the GPT series.
Methodology
The researchers conducted a comprehensive "red-teaming" evaluation using the HarmBench framework, a standardized benchmark containing 510 harmful behaviors across categories such as direct instructions (Standard), persona-embedded requests (Contextual), and copyright violations.
Model Architecture
DeepSeek models utilize a sparse MoE design where only a fraction of the model's parameters are activated for any given input. This is contrasted with the dense architectures of the GPT series. The architecture includes Multi-Head Latent Attention (MLA) and a routing mechanism for experts.
Attack Strategies
The evaluation tested two model families across seven distinct jailbreak strategies:
- Zero-Shot: Direct harmful queries.
- Optimization-based (GCG-T, AutoDAN): Using gradients to find suffixes that trigger harmful outputs.
- LLM-Assisted (PAIR, TAP): Iterative refinement of jailbreak prompts using a helper AI.
- Human-Engineered/Persuasive (PAP, HumanJailbreaks): Psychological tactics and complex roleplay.
The study compared several versions of DeepSeek-Distill (1.5B to 32B parameters) against GPT-3.5 Turbo and GPT-4 Turbo, measuring the Attack Success Rate (ASR).
Results and Key Insights
The MoE "Routing" Defense
A significant finding is that DeepSeek is remarkably robust against optimization-based attacks like TAP-T and GCG-T. DeepSeek models achieved an ASR of below 1% on TAP-T attacks, whereas GPT-4 Turbo was vulnerable at 57.7%. The authors hypothesize that because MoE models "route" tokens to different experts, the optimization process becomes "discontinuous." Attackers struggle to find a smooth mathematical path to bypass safety filters because the model dynamically switches active expert modules.
The "Under-Aligned Expert" Problem
Conversely, DeepSeek is much more vulnerable to Zero-Shot and Human-engineered attacks. In Zero-Shot scenarios, DeepSeek-7B reached a peak ASR of 40.4%, significantly higher than GPT-4 Turbo’s 12.7%. For human-crafted jailbreaks, GPT-4 Turbo demonstrated elite robustness (ASR as low as 2.6%), while DeepSeek models hovered around 41%. This suggests that adversarial prompts can trick the "router" into sending queries to expert modules that have not been sufficiently safety-tuned, a phenomenon termed the "alignment generalization gap."
Scale vs. Safety
Interestingly, as DeepSeek models increased in size (from 1.5B to 32B), they often became more vulnerable to certain attacks. This suggests that in open-source MoE frameworks, increased model capability can sometimes outpace the effectiveness of safety alignment.
Discussion and Conclusion
The study concludes that while MoE architectures offer exciting efficiency gains and accidental robustness to gradient-based attacks, they require a more targeted approach to safety tuning. The "patchy" safety observed—where a model might refuse a chemical formula but comply with misinformation requests—indicates that alignment in MoE models is harder to spread uniformly. Future work should focus on "modular alignment" to ensure every expert module is individually secured against exploitation.
Summary updated