Abstract
Traditional Mixture-of-Experts (MoE) architectures typically employ experts of uniform size, which may lead to computational inefficiencies when processing tokens of varying difficulty. We present Mixture of Diverse Size Experts (MoDSE), a novel approach that utilizes experts with heterogeneous parameter counts. By routing complex tokens to larger experts and simpler tokens to smaller ones, MoDSE achieves significant performance improvements across a wide range of benchmarks without increasing the total parameter budget or compromising inference speed.
Introduction
In standard MoE architectures, such as Mixtral or DeepSeekMoE, every expert is structurally identical. However, language modeling involves tokens with vastly different predictive difficulties. "Easy" tokens, like common idioms, require minimal computation, whereas "hard" tokens involving complex reasoning or specialized knowledge demand more capacity. Using a large expert for simple tokens is wasteful, while a small expert may be insufficient for difficult ones. MoDSE addresses this by creating a heterogeneous architecture that adapts to token difficulty while maintaining hardware efficiency.
Methods
MoDSE introduces two key innovations to enable effective heterogeneous expert modeling:
- Diverse Size Experts: Instead of uniform Feed-Forward Network (FFN) layers, MoDSE assigns experts a range of hidden layer dimensions. To maintain the same total parameter count as a baseline MoE, experts are grouped into pairs. For every "over-sized" expert, there is a corresponding "under-sized" expert such that their combined size matches the standard expert dimension.
- Expert-Pair Allocation Strategy: To prevent hardware load imbalance—where a GPU node holding larger experts might become a bottleneck—MoDSE co-locates each pair (one large and one small expert) on the same GPU node. This ensures that memory and computational workloads are distributed evenly across the cluster.
The routing mechanism utilizes a standard Top-K router which, through the training process, naturally learns to triage tokens based on their inherent difficulty.
Datasets and Evaluation
The model was pre-trained on a 100-billion token dataset comprising English and Chinese text from diverse sources, including CommonCrawl, code, academic papers, and books. Evaluation was performed across nine benchmarks covering general knowledge (MMLU), math and logic (GSM8K, MATH), common sense (PIQA, SIQA), and reading comprehension (TriviaQA).
Key Insights
- The "Amount of Thought" Analogy: Similar to how humans dedicate more cognitive effort to complex problems, MoDSE allocates greater parameter space to difficult tokens.
- Emergent Specialization: The router autonomously learns to send difficult tokens to larger experts. Analysis revealed that tokens with high cross-entropy loss in baseline models were twice as likely to be routed to the largest experts in MoDSE.
- Efficiency through Distribution: MoDSE demonstrates that the distribution of the parameter budget is as critical as its total scale. Varying expert sizes yielded significant gains, such as an 18-point improvement in SIQA, without adding extra parameters.
Key Results
MoDSE was evaluated in 300M × 8 and 700M × 8 configurations. The 700M MoDSE model outperformed the baseline MoE on every benchmark tested. Notable improvements include MMLU (26.5 to 29.9 accuracy) and SIQA (42.9 to 60.9 EM).
Importantly, inference duration remained nearly identical to the baseline. For the MMLU task, the baseline took 3m 26s while MoDSE took 3m 27s, confirming that the expert-pairing strategy effectively mitigates hardware bottlenecks.
Limitations and Future Work
While promising, MoDSE's benefits at "frontier-scale" (tens or hundreds of billions of parameters) remain to be explored. Additionally, the expert size ratios were manually set; future research could automate the discovery of optimal size distributions. Finally, the specific 100B token dataset used is not currently open-source.
Summary updated