Abstract
Mixture-of-Experts (MoE) architectures are a promising solution for scaling large language models (LLMs) while managing computational costs. However, conventional MoE designs often struggle with limited expert specialization, where experts fail to acquire distinct, non-overlapping knowledge. We introduce DeepSeekMoE, an architecture designed for "ultimate expert specialization" through two key strategies: fine-grained expert segmentation and shared expert isolation. Our results demonstrate that DeepSeekMoE 16B achieves performance parity with top-tier 7B dense models, such as LLaMA2, while utilizing only 40% of the computation. Furthermore, at a 2B parameter scale, DeepSeekMoE nearly reaches the performance of its dense counterpart with the same total parameters, setting a new benchmark for MoE efficiency.
The Challenge of Expert Specialization
Scaling LLMs is essential for performance but is traditionally constrained by linear increases in compute requirements. MoE architectures, like GShard, address this by activating only a fraction of the model's parameters for any given input. Despite their efficiency, standard MoEs face two primary hurdles:
- Knowledge Hybridity: Large experts are often forced to learn disparate types of information, making them "jacks-of-all-trades" and difficult to optimize.
- Knowledge Redundancy: Multiple experts frequently converge on the same foundational or "common sense" knowledge, wasting parameter capacity that could otherwise be used for specialized expertise.
DeepSeekMoE aims to overcome these issues by ensuring each expert captures highly focused, unique knowledge.
DeepSeekMoE Architecture
DeepSeekMoE introduces two pivotal architectural innovations to drive specialization:
- Fine-Grained Expert Segmentation: Instead of a few large experts, the Feed-Forward Network (FFN) intermediate hidden dimension is split into $m$ smaller pieces. By activating $m$ times more experts, the total FLOPs remain constant, but the number of possible expert combinations increases exponentially. This combinatorial flexibility allows the model to precisely route tokens to the most relevant knowledge units.
- Shared Expert Isolation: The architecture designates a specific number of experts as "shared," which are always activated regardless of the router's decision. These experts act as a sink for common knowledge and basic patterns, freeing the "routed" experts to focus on niche, specialized information.
Experimental Results
The models were trained on a massive bilingual (English and Chinese) corpus. A 2B parameter version was used for architectural validation (trained on 100B tokens), while a 16B version was trained on 2T tokens for full-scale evaluation across benchmarks like HellaSwag, PIQA, and HumanEval.
DeepSeekMoE demonstrates significant efficiency gains. At the 2B scale, it significantly outperforms GShard and even matches a GShard model with 1.5x more parameters and compute. Most notably, DeepSeekMoE 16B (with only ~2.8B activated parameters) matches the performance of LLaMA2 7B and DeepSeek 7B while using only ~40% of the FLOPs.
Key Insights and Analysis
Analysis of the model's internal dynamics confirms that DeepSeekMoE achieves higher specialization than previous architectures. The model is significantly more sensitive to the disabling of its top routed experts compared to GShard, suggesting that each expert has acquired more unique and irreplaceable knowledge.
Furthermore, the combinatorial explosion of expert selections (e.g., picking 8 out of 64 instead of 2 out of 16) allows the model to synthesize a much more tailored "virtual expert" for every token. While this granularity increases communication overhead in distributed settings, the performance gains and computational savings make it a compelling path forward for scaling LLMs.
Conclusion
DeepSeekMoE represents a significant step toward the theoretical upper bound of MoE performance. By isolating common knowledge and increasing routing flexibility, it achieves state-of-the-art efficiency. Future work will focus on optimizing the communication overhead and exploring the scaling laws governing expert granularity as total model size continues to grow.
Summary updated