MoEC: Mixture of Expert Clusters

Yuan Xie, Shaohan Huang, Tianyu Chen, Furu Wei

Abstract

Sparsely Mixture of Experts (MoE) has received great interest due to its promising scaling capability with affordable computational overhead. MoE converts dense layers into sparse experts, and utilizes a gated routing network to make experts conditionally activated. However, as the number of experts grows, MoE with outrageous parameters suffers from overfitting and sparse data allocation. Such problems are especially severe on tasks with limited data, thus hindering the progress for MoE models to improve performance by scaling up. In this work, we propose Mixture of Expert Clusters - a general approach to enable expert layers to learn more diverse and appropriate knowledge by imposing variance-based constraints on the routing stage. We further propose a cluster-level expert dropout strategy specifically designed for the expert cluster structure. Our experiments reveal that MoEC could improve performance on machine translation and natural language understanding tasks, and raise the performance upper bound for scaling up experts under limited data. We also verify that MoEC plays a positive role in mitigating overfitting and sparse data allocation.

Introduction

Sparse Mixture of Experts (MoE) models have become a cornerstone of the "scaling laws" era, allowing researchers to massively increase a model's parameter count (capacity) without a proportional increase in computational cost. This is achieved by activating only a small subset of "experts" for any given input token.

However, researchers have identified a significant "performance upper bound" in traditional MoE. As the number of experts increases, each individual expert receives fewer and less diverse training samples—a phenomenon known as sparse data allocation. This leads to a paradox where a 64-expert model might exhibit higher training loss than a 32-expert model because the experts lack sufficient data to learn effectively. This results in severe overfitting, particularly on downstream tasks where data is limited.

Methodology: Mixture of Expert Clusters (MoEC)

To address these limitations, we propose MoEC, which introduces two primary innovations to the standard MoE architecture: Expert Clustering and Cluster-level Expert Dropout.

Expert Clustering via Clustering Loss

Instead of allowing the router to assign tokens to any expert in an unconstrained manner, MoEC imposes a variance-based constraint. This "Clustering Loss" encourages "neighbor" experts to have similar routing probabilities. This creates an inductive bias where experts within a cluster are prone to tokens with similar hidden states, effectively allowing them to "share" the workload for specific data types.

Comparison between conventional MoE and the proposed MoEC architecture.

Cluster-level Expert Dropout

Traditional dropout randomly zeroes out neurons or weights. MoEC's version operates at the routing stage. During training, some experts within a cluster are randomly "dropped" (ignored by the router). This forces the model to dispatch tokens to other experts in the same cluster, ensuring that every expert receives a diverse set of samples and prevents the model from over-relying on a single "star" expert.

Experimental Setup

We evaluated MoEC across two major domains:

  • Machine Translation: WMT 2014 English-to-German (3.96M sentence pairs).
  • Natural Language Understanding (NLU): The GLUE benchmark, including tasks like MNLI, CoLA, and SST-2.
  • Pre-training: For NLU tasks, models were pre-trained on the BooksCorpus and English Wikipedia.

Key Results and Insights

Mitigating Overfitting and Raising the Scaling Ceiling

In standard MoE, performance often plateaus or drops after 32 experts when training on limited data. MoEC successfully "raises the ceiling," showing consistent gains up to 128 experts. Visualization of validation loss curves demonstrates that while standard MoE validation loss starts to climb (indicating overfitting) relatively early, MoEC's validation loss remains stable and continues to decrease.

Validation loss curves comparing MoE baselines with MoEC, showing improved stability and lower loss at higher expert counts.

Performance Gains

  • Translation: On the WMT14 En-De task, MoEC achieved a 32.50 BLEU score, a significant +1.91 boost over the MoE baseline (30.59).
  • NLU: MoEC outperformed the MoE baseline on every single GLUE task, raising the average score from 87.78 to 88.41.
  • Scaling Efficiency: When scaling from 64 to 128 experts, the standard MoE baseline's performance dropped (from 30.59 to 30.21 BLEU), while MoEC continued to improve.

Conclusion

MoEC offers a robust strategy for scaling sparse models when data is scarce. By forcing experts to share similar tokens within a cluster, the model ensures that no single expert is starved for data. While MoEC converges slightly slower than standard MoE due to the increased complexity of samples each expert must learn, it provides a significantly higher performance ceiling and better generalization for large-scale sparse models.

Summary updated

Abstract

Sparsely Mixture of Experts (MoE) has received great interest due to its promising scaling capability with affordable computational overhead. MoE converts dense layers into sparse experts, and utilizes a gated routing network to make experts conditionally activated. However, as the number of experts grows, MoE with outrageous parameters suffers from overfitting and sparse data allocation. Such problems are especially severe on tasks with limited data, thus hindering the progress for MoE models to improve performance by scaling up. In this work, we propose Mixture of Expert Clusters - a general approach to enable expert layers to learn more diverse and appropriate knowledge by imposing variance-based constraints on the routing stage. We further propose a cluster-level expert dropout strategy specifically designed for the expert cluster structure. Our experiments reveal that MoEC could improve performance on machine translation and natural language understanding tasks, and raise the performance upper bound for scaling up experts under limited data. We also verify that MoEC plays a positive role in mitigating overfitting and sparse data allocation.
BibTeX
@inproceedings{xie2023moec,
  author = {Yuan Xie and Shaohan Huang and Tianyu Chen and Furu Wei},
  title = {{MoEC: Mixture of Expert Clusters}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=HD5AbizZip},
}
Figure