L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts

Minghao Yang, Ren Togo, Guang Li, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper introduces L2R, a novel routing framework for Mixture-of-Experts (MoE) models that significantly improves expert specialization and overall performance. By employing a low-rank latent routing space and a Lipschitz-controlled scoring mechanism (SIPS), L2R addresses key limitations of existing MoE routers, leading to more stable training and better generalization in both language and vision tasks.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) models are a powerful technique for scaling deep neural networks by selectively activating a subset of "experts" (e.g., feed-forward networks) for each input token. This conditional computation allows for massive model capacity without a proportional increase in inference cost. The crucial component that determines which experts are activated is the "router." However, current MoE systems often face two main challenges:

  1. Representation Mismatch and High-Dimensional Concentration: Routers typically operate in the raw, high-dimensional representation space of the backbone network. This space is optimized for representation learning, not necessarily for distinguishing between experts. In high dimensions, similarities tend to concentrate, making it difficult for routers to create clear boundaries between experts, leading to ambiguous assignments and hindering stable specialization.
  2. Scale-Sensitive Dot-Product Scoring: Standard dot-product scoring for routing logits is sensitive to the magnitude (scale) of the input representations and expert anchors. This means that variations in feature scales can disproportionately influence routing decisions, leading to "norm-dominated competition" among experts and further weakening discrimination.

These issues collectively limit the effectiveness of expert specialization, which is central to the success of MoE models.

Methods

L2R tackles these problems with a three-pronged approach:

  1. Low-Rank Latent Routing Space: Instead of routing in the high-dimensional raw feature space, L2R first projects the input representations into a lower-dimensional "latent routing space" using a shared low-rank projection matrix ($W_q$). Experts are then represented by learnable "anchors" in this compressed space. This decouples routing from raw representation learning, reduces the search space for expert positioning, and mitigates the concentration problem.
  2. Saturated Inner-Product Scoring (SIPS): L2R introduces SIPS, a novel scoring function that explicitly controls the Lipschitz behavior of the routing mechanism. SIPS rebalances the influence of magnitude and angle in scoring. It uses a bounded query magnitude transform and a compressed anchor magnitude transform. This ensures that routing scores are less sensitive to extreme scales, leading to a smoother, more stable, and controllable scoring geometry that promotes robust expert discrimination.
  3. Multi-Anchor Heads: To enhance expert expressiveness without significant overhead, L2R associates each expert with multiple anchors in the latent routing space. This allows a single expert to capture a wider range of semantic variations, improving its ability to represent diverse data patterns.

Datasets

The paper evaluates L2R on:

  • OLMoE: A large-scale language model pretraining suite. Experiments use a 10B-token subset of the OLMoE pretraining mixture, which is distribution-matched to the full dataset.
  • ImageNet-1K: A standard dataset for vision tasks, used to evaluate L2R on a vision MoE system with a ViT-S backbone.

Key Insights and Ideas

  • Low-rank routing is sufficient: The paper demonstrates that a surprisingly low rank (e.g., r=2) for the latent routing space is enough to achieve significant improvements, suggesting that the exact dimensionality is less critical than the decoupling from raw high-dimensional space.
  • SIPS is key to stability: The SIPS scoring function, by bounding magnitude sensitivity, is shown to be crucial for stable routing, even outperforming pure cosine scoring in some aspects by retaining a controlled magnitude signal.
  • Multi-anchor heads boost expressiveness: While a moderate number of anchors per expert is beneficial, excessively many anchors do not consistently improve performance and can even degrade it, suggesting a trade-off between expressiveness and potential overfitting.
  • Routing geometry matters across domains: L2R's improvements are observed in both language and vision tasks, highlighting that better routing geometry is a general principle for MoE models, not specific to a particular domain.

Key Results

  • LLM Experiments (OLMoE): L2R consistently achieved higher MMLU accuracy (improving from 22.7% for Linear to 27.6%) and competitive performance on other benchmarks compared to baselines like Linear and X-MoE, after training for 10B tokens. It also showed faster convergence and more balanced expert utilization.
  • Vision Experiments (ImageNet-1K): L2R (SIPS) achieved the best Top-1 accuracy (76.58%) and Top-5 accuracy (91.50%) on ImageNet-1K using a ViT-S backbone, outperforming Linear and X-MoE.
  • Robustness: L2R (SIPS) demonstrated significantly improved robustness to Gaussian noise perturbations compared to unconstrained dot-product routing in a low-rank space.
  • Parameter Efficiency: The L2R router adds only a minor overhead in terms of parameters and computation compared to standard linear routers, especially when considering the reduction in the dimensionality of the routing space.

Limitations and Future Work

  • Early-stage pretraining: The language experiments focused on an early-stage pretraining regime (10B tokens). Further scaling to longer pretraining budgets and larger MoE configurations is suggested for future work.
  • Generalization to other architectures: While L2R was tested on Transformer and ViT backbones, its performance on other architectures and diverse downstream tasks could be further explored.
  • Optimal hyperparameters: Investigating how optimal routing rank and the number of anchors vary with model size, expert count, and training duration is a promising direction.
  • Hardware efficiency: While L2R is theoretically efficient, practical inference speedups might require further kernel-level optimizations to fully leverage hardware.

Summary updated

Abstract

Mixture-of-Experts (MoE) models scale neural networks by conditionally activating a small subset of experts, where the router plays a central role in determining expert specialization and overall model performance. However, many modern MoE systems still adopt linear routers in raw high-dimensional representation spaces, where representation mismatch, angular concentration, and scale-sensitive scoring can jointly undermine routing discriminability and stable expert specialization. In this work, we propose Low-rank & Lipschitz-controlled Routing (L2R), a unified routing framework that reshapes both the routing space and scoring geometry. L2R performs expert assignment in a shared low-rank latent routing space and introduces Saturated Inner-Product Scoring (SIPS) to explicitly control the Lipschitz behavior of routing functions, yielding smoother and more stable routing geometry. In addition, L2R incorporates a parameter-efficient multi-anchor routing mechanism to enhance expert expressiveness. Extensive experiments on an OLMoE-based language MoE model and a vision MoE setting on ImageNet demonstrate that L2R consistently improves routing geometry, expert discrimination, and overall model performance. Code will be released.
BibTeX
@misc{yang2026l,
  author = {Minghao Yang and Ren Togo and Guang Li and Takahiro Ogawa and Miki Haseyama},
  title = {{L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts}},
  year = {2026},
  eprint = {2601.21349},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.21349},
}
Figure