Abstract
Sparse Mixture of Experts (SMoE) architectures enable scaling Large Language Models (LLMs) with constant computational costs by activating only a subset of parameters per token. However, they suffer from "representational collapse," where experts become redundant or idle. We present SimSMoE, a framework that introduces a Similarity Learning module to directly minimize representational similarity between experts using Centered Kernel Alignment (CKA). Our results demonstrate that SimSMoE effectively solves feature collapse, leading to superior performance, faster convergence, and better scaling across various benchmarks.
Introduction
Sparse Mixture of Experts (SMoE) is a popular architecture for scaling LLMs because it allows a model to have a massive number of parameters while only activating a small subset (experts) for any given input token. This keeps computational costs constant while increasing capacity. However, SMoEs suffer from a persistent "representational collapse" problem. This happens in two ways: Routing Collapse, where the router becomes biased towards a few experts, and Feature Collapse, where different experts learn nearly identical representations. While previous work focused on improving the router, SimSMoE focuses on the experts themselves to ensure they actually specialize.
The SimSMoE Framework
The core of the paper is the SimSMoE framework, which adds a diversification objective to the training process.
The Similarity Learning module acts as a "quality control" check. It projects expert outputs into a shared space to calculate a Similarity Loss ($L_{similarity}$) using Centered Kernel Alignment (CKA). The training objective is modified as follows: $$L = L_{task} + \alpha L_{balancing} + \beta L_{similarity}$$ where $\beta$ controls the strength of the diversification push. To maintain efficiency, the authors introduce a hyperparameter ($f^*$) to control the frequency of these similarity checks, avoiding the $O(N^2)$ cost of comparing every expert pair.
Key Insights
- Token Sharing Correlation: There is a strong empirical correlation between token sharing and representational collapse. Experts that receive the same tokens are significantly more likely to become redundant.
- Layer-wise Sensitivity: Representational collapse is more severe in the earlier layers of a model. Deeper layers naturally exhibit more diversity, suggesting that diversification efforts should be concentrated at the start of the network.
- Direct Intervention: Unlike previous methods that try to fix collapse indirectly via routing, SimSMoE shows that directly penalizing similarity in the representation space is more effective and provides a theoretical guarantee of diversity.
Experimental Results
The authors evaluated SimSMoE across a wide range of benchmarks, including pre-training (enwik8, WikiText-103), classification (GLUE, SST-2), and generative tasks (XSum, SQuAD).
SimSMoE consistently outperformed standard SMoE, XMoE, and StableMoE. For a 135M parameter Brainformer, SimSMoE achieved a 1.08 BPC on enwik8, compared to 1.11 for the baseline. Furthermore, the performance gap widened as model size increased, particularly for models over 1 billion parameters. Validation curves show that SimSMoE reaches lower loss levels much earlier in the training process. In fine-tuning, SimSMoE showed massive improvements, such as a 4.8% accuracy gain on the BANKING77 dataset.
Conclusion
SimSMoE addresses the critical issue of representational collapse in SMoE architectures by directly optimizing for expert diversity. By ensuring that experts learn distinct features, the framework allows LLMs to more effectively utilize their expanded parameter capacity. Future work includes scaling these evaluations to models with over 100 billion parameters and exploring similarity constraints for non-MLP expert architectures.
Summary updated