DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, Wenfeng Liang

Abstract

Mixture-of-Experts (MoE) architectures are a promising solution for scaling large language models (LLMs) while managing computational costs. However, conventional MoE designs often struggle with limited expert specialization, where experts fail to acquire distinct, non-overlapping knowledge. We introduce DeepSeekMoE, an architecture designed for "ultimate expert specialization" through two key strategies: fine-grained expert segmentation and shared expert isolation. Our results demonstrate that DeepSeekMoE 16B achieves performance parity with top-tier 7B dense models, such as LLaMA2, while utilizing only 40% of the computation. Furthermore, at a 2B parameter scale, DeepSeekMoE nearly reaches the performance of its dense counterpart with the same total parameters, setting a new benchmark for MoE efficiency.

The Challenge of Expert Specialization

Scaling LLMs is essential for performance but is traditionally constrained by linear increases in compute requirements. MoE architectures, like GShard, address this by activating only a fraction of the model's parameters for any given input. Despite their efficiency, standard MoEs face two primary hurdles:

  1. Knowledge Hybridity: Large experts are often forced to learn disparate types of information, making them "jacks-of-all-trades" and difficult to optimize.
  2. Knowledge Redundancy: Multiple experts frequently converge on the same foundational or "common sense" knowledge, wasting parameter capacity that could otherwise be used for specialized expertise.

DeepSeekMoE aims to overcome these issues by ensuring each expert captures highly focused, unique knowledge.

DeepSeekMoE Architecture

DeepSeekMoE introduces two pivotal architectural innovations to drive specialization:

  • Fine-Grained Expert Segmentation: Instead of a few large experts, the Feed-Forward Network (FFN) intermediate hidden dimension is split into $m$ smaller pieces. By activating $m$ times more experts, the total FLOPs remain constant, but the number of possible expert combinations increases exponentially. This combinatorial flexibility allows the model to precisely route tokens to the most relevant knowledge units.
  • Shared Expert Isolation: The architecture designates a specific number of experts as "shared," which are always activated regardless of the router's decision. These experts act as a sink for common knowledge and basic patterns, freeing the "routed" experts to focus on niche, specialized information.
The evolution of MoE architectures from conventional top-2 routing to DeepSeekMoE's fine-grained segmentation and shared expert isolation.

Experimental Results

The models were trained on a massive bilingual (English and Chinese) corpus. A 2B parameter version was used for architectural validation (trained on 100B tokens), while a 16B version was trained on 2T tokens for full-scale evaluation across benchmarks like HellaSwag, PIQA, and HumanEval.

DeepSeekMoE demonstrates significant efficiency gains. At the 2B scale, it significantly outperforms GShard and even matches a GShard model with 1.5x more parameters and compute. Most notably, DeepSeekMoE 16B (with only ~2.8B activated parameters) matches the performance of LLaMA2 7B and DeepSeek 7B while using only ~40% of the FLOPs.

Performance comparison of DeepSeekMoE against various baselines, including Dense, Hash Layer, Switch Transformer, and GShard models.

Key Insights and Analysis

Analysis of the model's internal dynamics confirms that DeepSeekMoE achieves higher specialization than previous architectures. The model is significantly more sensitive to the disabling of its top routed experts compared to GShard, suggesting that each expert has acquired more unique and irreplaceable knowledge.

Furthermore, the combinatorial explosion of expert selections (e.g., picking 8 out of 64 instead of 2 out of 16) allows the model to synthesize a much more tailored "virtual expert" for every token. While this granularity increases communication overhead in distributed settings, the performance gains and computational savings make it a compelling path forward for scaling LLMs.

Conclusion

DeepSeekMoE represents a significant step toward the theoretical upper bound of MoE performance. By isolating common knowledge and increasing routing flexibility, it achieves state-of-the-art efficiency. Future work will focus on optimizing the communication overhead and exploring the scaling laws governing expert granularity as total model size continues to grow.

Summary updated

Abstract

In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-K out of N experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into mN ones and activating mK from them, allowing for a more flexible combination of activated experts; (2) isolating Ks experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 × expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which sets the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations.
BibTeX
@inproceedings{dai2024deepseekmoe,
  author = {Damai Dai and Chengqi Deng and Chenggang Zhao and R. X. Xu and Huazuo Gao and Deli Chen and Jiashi Li and Wangding Zeng and Xingkai Yu and Y. Wu and Zhenda Xie and Y. K. Li and Panpan Huang and Fuli Luo and Chong Ruan and Zhifang Sui and Wenfeng Liang},
  title = {{DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=EmUsC2FogT},
}
Figure