CompeteSMoE - Statistically Guaranteed Mixture of Experts Training via Competition

Nam V. Nguyen, Huy Nguyen, Quang Pham, Van Nguyen, Savitha Ramasamy, Nhat HoSubmission to ICLR 2026 · Rejected

TL;DR

Standard Sparse Mixture of Experts (SMoE) relies on a lightweight router that guesses which experts to pick before running them, leading to suboptimal routing policies and sluggish sample efficiency. CompeteSMoE introduces an expert-driven "competition" mechanism—inspired by biological winner-take-all dynamics—where experts route based on their actual neural activations, and a student router periodically distills this competitive policy to maintain standard sparse-inference efficiency.

What Problem Does This Paper Address?

In standard SMoE models (like Switch Transformers or Mixtral), routing is fundamentally detached from expert computation: an external gating projection takes the input token, calculates dot-product affinities, and routes tokens without knowing how the experts would actually respond.

This causes two major issues:

  1. Suboptimal routing policies: The gating network frequently misallocates tokens because it operates blind to the internal feature representations of the expert MLPs.
  2. Poor statistical convergence: Prior theoretical analyses show that estimating parameters under vanilla softmax gating converges very slowly (often requiring sample sizes on the order of $\mathcal{O}(\epsilon^{-12})$ to estimate over-specified experts within error $\epsilon$).

While activating all experts to let them "compete" would yield superior routing, doing so naively destroys the computational advantages of sparsity.

Methods

The authors propose a mathematically grounded framework called CompeteSMoE:

  • Routing via Competition: Tokens are passed through all candidate experts to compute neural activation responses $s_i = \mathbb{E}[\text{Softplus}(g(x, W_{e_i}))]$. The top-$K$ experts with the strongest activations win the right to process the token; their outputs are normalized and aggregated, while losers are suppressed.
  • Theoretical Sample Efficiency: The authors prove that Gaussian MoEs with competition achieve parameter and expert estimation rates of order $\widetilde{\mathcal{O}}_P(n^{-1/2})$ for exact-specified experts and $\widetilde{\mathcal{O}}_P(n^{-1/4})$ for over-specified experts, dramatically reducing the sample complexity to $\mathcal{O}(\epsilon^{-4})$.
  • Scheduled Distillation into a Fast Router: Running full competition at every layer on every step is intractable. CompeteSMoE keeps a standard lightweight router and trains it with an interleaved schedule:
  • Layer-wise Stochastics & Concurrency Control: Competition activates with a small probability per layer ($\omega \approx 5\text{--}7\%$) subject to a strict global maximum of concurrently active layers ($A_{\text{max}}$) to prevent out-of-memory errors.
  • Weighted Distillation Loss ($\mathcal{L}_D$): When competition activates, the standard router updates its parameters via MSE distillation against the competition scores, with extra penalty weight $\alpha$ placed on the winning experts.
  • Diversity Loss ($\mathcal{L}_{\text{div}}$): To assist sparsely upcycled models (where cloned experts initially output identical features), a pairwise correlation penalty forces winning experts to differentiate representations.

At inference time, the model runs standard, fast top-$K$ routing with zero added overhead.

Datasets

The authors evaluate on both multimodal fine-tuning and language pretraining from scratch:

  • Visual Instruction Tuning (VIT):
  • Training: Three-stage pipeline using LLaVA-558K (pretraining), ALLaVA-Caption 708K (pre-finetuning), and LLaVA-665K (instruction tuning).
  • Zero-shot Evaluation (11 benchmarks): AI2D, TextVQA, GQA, MMBench, HallusionBench, MathVista, MMMU, MMStar, POPE, OCRBench, and MME RealWorld Lite.
  • Language Model Pretraining:
  • Training: 13B tokens sampled from the SlimPajama corpus.
  • Zero-shot Evaluation (8 benchmarks): LAMBADA, BLiMP, Children’s Book Test (CBT), HellaSwag, PIQA, ARC-Easy, RACE, and SIQA.

No new benchmarks were created; all evaluations rely on standard community benchmarks.

Key Insights and Ideas

  • Vanilla SMoE Routing is Actively Flawed: In a diagnostic test, swapping the top-1 selected expert with the $(K+1)$-ranked expert improved vanilla SMoE performance by +0.51 points on average across vision tasks. In CompeteSMoE, the same degradation caused accuracy to drop by -1.03 points—proving that CompeteSMoE learns a truly optimal ordering whereas standard SMoE gating often prefers worse experts.
  • Faster Convergence in Wall-Clock Time: CompeteSMoE reached higher zero-shot accuracy at 8 hours of training than vanilla SMoE, XMoE, $\sigma$-MoE, and Shared-Expert baselines reached at their final 14-hour checkpoints.
  • Router Stability: By monitoring the Expert Change Rate (ECR) across checkpoints, the authors showed that CompeteSMoE's routers stabilize and converge substantially faster across training than baselines.

Key Results

  • Multimodal (5.1B parameter VLM, SigLIP + Phi-3.5 Mini): CompeteSMoE achieved an average accuracy of 50.16% (average rank 1.77) across 11 benchmarks, outperforming standard SMoE (49.39%, rank 4.55), XMoE (49.67%), and SharedE-V2 (49.63%).
  • Language Pretraining (1B parameter decoder-only model): CompeteSMoE reached 54.04% average accuracy (rank 1.75) across 8 reasoning/language tasks, beating standard SMoE (53.67%) and DeepSeek-V3-style SharedE-V3 (53.77%).
  • Computational Footprint: Training wall-clock time increased only marginally (13h 01m vs. 12h 39m for standard SMoE on the 5.1B model), while inference throughput was completely unaffected (9.88 samples/sec vs. 9.87 samples/sec). Peak GPU memory increased by less than 5%.

Limitations and Future Work

  • Hyperparameter Tuning Overhead: The method introduces several hyperparameters ($\omega, \alpha, \beta, \gamma, A_{\text{max}}$). Although ablation experiments indicate that performance is relatively robust to moderate settings, tuning them for different hardware memory budgets remains necessary.
  • Layer-Independent Competition: Competition is scheduled and evaluated per layer independently. It neglects inter-layer dependencies, where decisions in early layers shape optimal routing downstream. The authors note that exploring graph-traversal routing across the entire network depth is a promising future direction.

Summary updated

Abstract

Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, we argue that effective SMoE training remains challenging because of the suboptimal routing process where experts that perform computation do not directly contribute to the routing process. In this work, we propose competition, a novel mechanism to route tokens to experts with the highest neural response. Theoretically, we show that the competition mechanism enjoys a better sample efficiency than the traditional softmax routing. Furthermore, we develop CompeteSMoE, a simple yet effective algorithm to train large language models by deploying a router to learn the competition policy, thus enjoying strong performances at a low training overhead. Our extensive empirical evaluations on both the visual instruction tuning and language pre-training tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE compared to state-of-the-art SMoE strategies. We will publish the implementation upon acceptance.
BibTeX
@misc{nguyen2025competesmoe,
  author = {Nam V. Nguyen and Huy Nguyen and Quang Pham and Van Nguyen and Savitha Ramasamy and Nhat Ho},
  title = {{CompeteSMoE - Statistically Guaranteed Mixture of Experts Training via Competition}},
  year = {2025},
  eprint = {2505.13380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.13380},
}
Figure