Abstract
Sparsely Mixture of Experts (MoE) has received great interest due to its promising scaling capability with affordable computational overhead. MoE models convert dense layers into sparse experts, and utilize a gated routing network to make experts conditionally activated. However, as the number of experts grows, MoE with outrageous parameters suffers from overfitting and sparse data allocation. Such problems are especially severe on tasks with limited data, thus hindering the progress towards improving performance by scaling up. We verify that there exists a performance upper bound of scaling up sparse MoE. In this work, we propose Mixture of Expert Clusters — a general approach to enable expert layers to learn more diverse and appropriate knowledge by imposing variance-based constraints on the routing stage. Given this, we could further propose a cluster-level expert dropout strategy specifically designed for the expert cluster structure. Our experiments reveal that MoEC could improve performance on machine translation and natural language understanding tasks. MoEC plays a positive role in mitigating overfitting and sparse data allocation problems, thus fully releasing the potential of large-scale sparse models.
The Scaling Paradox in Sparse MoE
Sparse Mixture of Experts (MoE) models are designed to scale capacity efficiently by activating only a subset of parameters (experts) for each input token. However, empirical evidence suggests a performance ceiling when scaling these models on tasks with limited data.
As illustrated in Figure 1, increasing the expert count from 32 to 128 leads to higher validation loss. This phenomenon, termed the "Too Many Experts" paradox, stems from sparse data allocation. As the number of experts grows, the training data is sliced too thin; each individual expert receives a narrow, less diverse set of tokens, which prevents adequate parameter training and exacerbates overfitting. In some cases, a 64-expert model can even exhibit higher training loss than a 32-expert model, indicating that the routing mechanism is starving parameters of diverse data.
Mixture of Expert Clusters (MoEC)
To break this scaling ceiling, the authors introduce Mixture of Expert Clusters (MoEC). This approach modifies the routing stage to group experts into clusters, ensuring that each expert is exposed to a richer, more diverse set of tokens.
MoEC implements two primary mechanisms:
- Variance-Based Clustering Loss: During training, a clustering loss is added to the objective function. This forces a group of "neighbor" experts to have similar routing probabilities for the same tokens, effectively forming a static cluster. Instead of a token being routed to a single hyper-specific expert, it is shared among experts within a cluster, providing each expert with more varied data.
- Cluster-Level Expert Dropout: Standard global expert dropout can route tokens to irrelevant experts, damaging semantic mapping. MoEC introduces cluster-level dropout, which randomly disables experts within a specific cluster. If a token's primary expert is dropped, it is forced to visit an alternative expert within the same relevant cluster, providing regularization without sacrificing routing sanity.
Experimental Results
The effectiveness of MoEC was evaluated across machine translation and natural language understanding (NLU) benchmarks.
Machine Translation
On the WMT-14 English-to-German task, a 64-expert MoEC model achieved a BLEU score of 32.50, a significant +1.91 improvement over the standard 64-expert MoE baseline (30.59). While the baseline MoE model peaked at 32 experts and degraded thereafter, MoEC successfully pushed the performance upper bound higher, maintaining strong results even at 128 experts.
Natural Language Understanding
On the GLUE benchmark, MoEC achieved an average score of 88.41, outperforming both the MoE baseline (87.78) and the dense baseline (85.16). These results confirm that MoEC effectively mitigates overfitting across diverse linguistic tasks.
Conclusion
MoEC addresses the fundamental bottleneck of sparse data allocation in large-scale MoE models. By grouping experts into clusters and employing targeted dropout, MoEC allows sparse models to fully realize their scaling potential, even when trained on datasets of limited size.
Summary updated