Mixture of Diverse Size Experts

Manxi Sun, Wei Liu, Jian Luan, Pengzhi Gao, Bin Wang

Abstract

Traditional Mixture-of-Experts (MoE) architectures typically employ experts of uniform size, which may lead to computational inefficiencies when processing tokens of varying difficulty. We present Mixture of Diverse Size Experts (MoDSE), a novel approach that utilizes experts with heterogeneous parameter counts. By routing complex tokens to larger experts and simpler tokens to smaller ones, MoDSE achieves significant performance improvements across a wide range of benchmarks without increasing the total parameter budget or compromising inference speed.

Introduction

In standard MoE architectures, such as Mixtral or DeepSeekMoE, every expert is structurally identical. However, language modeling involves tokens with vastly different predictive difficulties. "Easy" tokens, like common idioms, require minimal computation, whereas "hard" tokens involving complex reasoning or specialized knowledge demand more capacity. Using a large expert for simple tokens is wasteful, while a small expert may be insufficient for difficult ones. MoDSE addresses this by creating a heterogeneous architecture that adapts to token difficulty while maintaining hardware efficiency.

Methods

MoDSE introduces two key innovations to enable effective heterogeneous expert modeling:

  1. Diverse Size Experts: Instead of uniform Feed-Forward Network (FFN) layers, MoDSE assigns experts a range of hidden layer dimensions. To maintain the same total parameter count as a baseline MoE, experts are grouped into pairs. For every "over-sized" expert, there is a corresponding "under-sized" expert such that their combined size matches the standard expert dimension.
  2. Expert-Pair Allocation Strategy: To prevent hardware load imbalance—where a GPU node holding larger experts might become a bottleneck—MoDSE co-locates each pair (one large and one small expert) on the same GPU node. This ensures that memory and computational workloads are distributed evenly across the cluster.
Architecture of the MoDSE layer showing expert pairs of diverse sizes co-located on GPU nodes to maintain load balance.

The routing mechanism utilizes a standard Top-K router which, through the training process, naturally learns to triage tokens based on their inherent difficulty.

Datasets and Evaluation

The model was pre-trained on a 100-billion token dataset comprising English and Chinese text from diverse sources, including CommonCrawl, code, academic papers, and books. Evaluation was performed across nine benchmarks covering general knowledge (MMLU), math and logic (GSM8K, MATH), common sense (PIQA, SIQA), and reading comprehension (TriviaQA).

Key Insights

  • The "Amount of Thought" Analogy: Similar to how humans dedicate more cognitive effort to complex problems, MoDSE allocates greater parameter space to difficult tokens.
  • Emergent Specialization: The router autonomously learns to send difficult tokens to larger experts. Analysis revealed that tokens with high cross-entropy loss in baseline models were twice as likely to be routed to the largest experts in MoDSE.
  • Efficiency through Distribution: MoDSE demonstrates that the distribution of the parameter budget is as critical as its total scale. Varying expert sizes yielded significant gains, such as an 18-point improvement in SIQA, without adding extra parameters.

Key Results

MoDSE was evaluated in 300M × 8 and 700M × 8 configurations. The 700M MoDSE model outperformed the baseline MoE on every benchmark tested. Notable improvements include MMLU (26.5 to 29.9 accuracy) and SIQA (42.9 to 60.9 EM).

Training and validation loss curves for 300M and 700M MoDSE models compared to baseline MoE, demonstrating consistent performance gains.

Importantly, inference duration remained nearly identical to the baseline. For the MMLU task, the baseline took 3m 26s while MoDSE took 3m 27s, confirming that the expert-pairing strategy effectively mitigates hardware bottlenecks.

Limitations and Future Work

While promising, MoDSE's benefits at "frontier-scale" (tens or hundreds of billions of parameters) remain to be explored. Additionally, the expert size ratios were manually set; future research could automate the discovery of optimal size distributions. Finally, the specific 100B token dataset used is not currently open-source.

Summary updated

Abstract

The Sparsely-Activated Mixture-of-Experts (MoE) architecture has gained popularity for scaling large language models (LLMs) due to the sub-linearly increasing computational costs. Despite its success, most of the current structure designs face the challenge that the experts share the same size such that tokens have no chance to choose the experts with the most appropriate size to generate the next token. To migrate this defect, we propose Mixture of Diverse Size Experts (MoDSE), a new MoE architecture with designed layers where experts have different sizes. Analysis on difficult token generation tasks shows that experts with different sizes give better predictions, and the routing path of the experts tends to be stable after a period of training. The diversity of experts’ size will lead to load unbalancing. To tackle this limitation, we introduce an expert-pair allocation strategy to distribute the workload evenly across the GPUs. Comprehensive evaluations across multiple benchmarks demonstrate the effectiveness of MoDSE, surpassing existing MoEs by adaptively assigning the parameter budget to experts while maintaining the same total parameter size and number of experts.
BibTeX
@inproceedings{sun2024mixture,
  author = {Manxi Sun and Wei Liu and Jian Luan and Pengzhi Gao and Bin Wang},
  title = {{Mixture of Diverse Size Experts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=nJxAl2saCp},
}
Figure