TL;DR
This paper introduces L2R, a novel routing framework for Mixture-of-Experts (MoE) models that significantly improves expert specialization and overall performance. By employing a low-rank latent routing space and a Lipschitz-controlled scoring mechanism (SIPS), L2R addresses key limitations of existing MoE routers, leading to more stable training and better generalization in both language and vision tasks.
What Problem Does This Paper Address?
Mixture-of-Experts (MoE) models are a powerful technique for scaling deep neural networks by selectively activating a subset of "experts" (e.g., feed-forward networks) for each input token. This conditional computation allows for massive model capacity without a proportional increase in inference cost. The crucial component that determines which experts are activated is the "router." However, current MoE systems often face two main challenges:
- Representation Mismatch and High-Dimensional Concentration: Routers typically operate in the raw, high-dimensional representation space of the backbone network. This space is optimized for representation learning, not necessarily for distinguishing between experts. In high dimensions, similarities tend to concentrate, making it difficult for routers to create clear boundaries between experts, leading to ambiguous assignments and hindering stable specialization.
- Scale-Sensitive Dot-Product Scoring: Standard dot-product scoring for routing logits is sensitive to the magnitude (scale) of the input representations and expert anchors. This means that variations in feature scales can disproportionately influence routing decisions, leading to "norm-dominated competition" among experts and further weakening discrimination.
These issues collectively limit the effectiveness of expert specialization, which is central to the success of MoE models.
Methods
L2R tackles these problems with a three-pronged approach:
- Low-Rank Latent Routing Space: Instead of routing in the high-dimensional raw feature space, L2R first projects the input representations into a lower-dimensional "latent routing space" using a shared low-rank projection matrix ($W_q$). Experts are then represented by learnable "anchors" in this compressed space. This decouples routing from raw representation learning, reduces the search space for expert positioning, and mitigates the concentration problem.
- Saturated Inner-Product Scoring (SIPS): L2R introduces SIPS, a novel scoring function that explicitly controls the Lipschitz behavior of the routing mechanism. SIPS rebalances the influence of magnitude and angle in scoring. It uses a bounded query magnitude transform and a compressed anchor magnitude transform. This ensures that routing scores are less sensitive to extreme scales, leading to a smoother, more stable, and controllable scoring geometry that promotes robust expert discrimination.
- Multi-Anchor Heads: To enhance expert expressiveness without significant overhead, L2R associates each expert with multiple anchors in the latent routing space. This allows a single expert to capture a wider range of semantic variations, improving its ability to represent diverse data patterns.
Datasets
The paper evaluates L2R on:
- OLMoE: A large-scale language model pretraining suite. Experiments use a 10B-token subset of the OLMoE pretraining mixture, which is distribution-matched to the full dataset.
- ImageNet-1K: A standard dataset for vision tasks, used to evaluate L2R on a vision MoE system with a ViT-S backbone.
Key Insights and Ideas
- Low-rank routing is sufficient: The paper demonstrates that a surprisingly low rank (e.g., r=2) for the latent routing space is enough to achieve significant improvements, suggesting that the exact dimensionality is less critical than the decoupling from raw high-dimensional space.
- SIPS is key to stability: The SIPS scoring function, by bounding magnitude sensitivity, is shown to be crucial for stable routing, even outperforming pure cosine scoring in some aspects by retaining a controlled magnitude signal.
- Multi-anchor heads boost expressiveness: While a moderate number of anchors per expert is beneficial, excessively many anchors do not consistently improve performance and can even degrade it, suggesting a trade-off between expressiveness and potential overfitting.
- Routing geometry matters across domains: L2R's improvements are observed in both language and vision tasks, highlighting that better routing geometry is a general principle for MoE models, not specific to a particular domain.
Key Results
- LLM Experiments (OLMoE): L2R consistently achieved higher MMLU accuracy (improving from 22.7% for Linear to 27.6%) and competitive performance on other benchmarks compared to baselines like Linear and X-MoE, after training for 10B tokens. It also showed faster convergence and more balanced expert utilization.
- Vision Experiments (ImageNet-1K): L2R (SIPS) achieved the best Top-1 accuracy (76.58%) and Top-5 accuracy (91.50%) on ImageNet-1K using a ViT-S backbone, outperforming Linear and X-MoE.
- Robustness: L2R (SIPS) demonstrated significantly improved robustness to Gaussian noise perturbations compared to unconstrained dot-product routing in a low-rank space.
- Parameter Efficiency: The L2R router adds only a minor overhead in terms of parameters and computation compared to standard linear routers, especially when considering the reduction in the dimensionality of the routing space.
Limitations and Future Work
- Early-stage pretraining: The language experiments focused on an early-stage pretraining regime (10B tokens). Further scaling to longer pretraining budgets and larger MoE configurations is suggested for future work.
- Generalization to other architectures: While L2R was tested on Transformer and ViT backbones, its performance on other architectures and diverse downstream tasks could be further explored.
- Optimal hyperparameters: Investigating how optimal routing rank and the number of anchors vary with model size, expert count, and training duration is a promising direction.
- Hardware efficiency: While L2R is theoretically efficient, practical inference speedups might require further kernel-level optimizations to fully leverage hardware.
Summary updated