Abstract
In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-$K$ out of $N$ experts, face challenges in ensuring expert specialization, i.e., each expert acquiring non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into $mN$ ones and activating $mK$ from them, allowing for a more flexible combination of activated experts; (2) isolating $K_s$ experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, showing performance comparable with DeepSeek 67B using only 28.5% of computations.
Introduction
Scaling Large Language Models (LLMs) typically requires massive computational budgets. Mixture-of-Experts (MoE) architectures, such as GShard, attempt to mitigate this by activating only a subset of "experts" (parameters) for each token, keeping inference costs manageable while increasing total model capacity. However, conventional MoEs face two major hurdles: knowledge hybridity and knowledge redundancy. Because there are typically few experts (e.g., 8 or 16), each expert is forced to learn a wide, overlapping range of topics, preventing true specialization. Furthermore, common knowledge—such as basic grammar or universal facts—ends up being redundantly learned across multiple experts, wasting parameter capacity.
DeepSeekMoE is a novel architecture designed to achieve "ultimate expert specialization" by addressing these inefficiencies through finer granularity and strategic isolation of common knowledge.
Architectural Innovations
DeepSeekMoE introduces two core architectural strategies to push the Pareto frontier of LLM efficiency:
1. Fine-Grained Expert Segmentation
Instead of a few large experts, DeepSeekMoE splits the Feed-Forward Network (FFN) intermediate dimension into many smaller experts ($mN$). To keep computation constant, it activates a larger number of these smaller experts ($mK$). This dramatically increases the combinatorial flexibility—the number of ways the model can combine experts to process a specific token. For example, by splitting 16 experts into 64, the number of possible expert combinations increases exponentially, allowing the model to route tokens with much higher precision.
2. Shared Expert Isolation
The model designates a specific set of experts as "shared." These experts are always activated regardless of the router's decision. These shared experts act as a "sink" for common, cross-domain knowledge. By capturing fundamental information in these shared units, the "routed" experts are freed to focus purely on specialized, non-overlapping information. Research indicates that replacing shared experts with additional routed experts leads to a significant performance drop, proving that shared experts capture fundamental knowledge that routed experts cannot easily replicate.
To ensure hardware efficiency and training stability, DeepSeekMoE employs Expert-Level and Device-Level Load Balancing losses, preventing "routing collapse" where only a few experts are utilized.
Experimental Evaluation
The models were trained on a high-quality multilingual corpus developed by DeepSeek-AI. Evaluation spanned a wide battery of benchmarks, including MMLU, C-Eval, and CMMLU for multi-subject knowledge; HumanEval and MBPP for coding; and GSM8K and MATH for reasoning.
Key Results
- Efficiency Gains: DeepSeekMoE 16B matches the performance of LLaMA2 7B and DeepSeek 7B (dense) while using only ~40% of the FLOPs.
- Approaching the Upper Bound: DeepSeekMoE 2B performs nearly as well as a dense model with the same number of total parameters. This is significant because a dense model represents the theoretical performance ceiling for any MoE architecture.
- Superior Scaling: At the 145B parameter scale, the architecture matches the performance of the DeepSeek 67B dense model using only 28.5% (and potentially as low as 18.2%) of the computation.
- Deployment Advantages: Due to its sparse activation, the 16B model can be deployed on a single 40GB GPU and achieves nearly 2.5x the inference speed of a standard 7B dense model.
Discussion and Limitations
While DeepSeekMoE significantly improves FFN efficiency, it faces an attention bottleneck. Because MoE primarily scales FFN parameters, the attention parameters remain constant. DeepSeekMoE 16B (with ~0.5B attention parameters) occasionally struggled on specific multiple-choice tasks compared to 7B dense models (which have ~2.5B attention parameters).
Furthermore, there is a practical limit to expert granularity; experts must remain large enough for matrix multiplication to stay efficient on modern GPU hardware. Future work will explore even finer granularity for models exceeding the 16B scale and continue the training of the 145B variant to uncover its full potential beyond the preliminary 245B tokens.
Conclusion
DeepSeekMoE represents a significant step toward ultimate expert specialization. By combining fine-grained segmentation with shared expert isolation, the architecture achieves state-of-the-art efficiency, allowing smaller computational budgets to reach the performance levels of much larger dense models.
Summary updated