Exploration-Driven Reinforcement Learning for Expert Routing Improvement in Mixture-of-Experts Language Models

Gyunyeop Kim, Sangwoo KangEMNLP 2025

TL;DR

This paper introduces Exploration-Driven Reinforcement Learning (ERL) to explicitly train the routing mechanism in Mixture-of-Experts (MoE) language models. By comparing the performance of original routing paths with randomly perturbed ones, ERL generates an advantage signal to guide the router towards more optimal expert selection, significantly improving performance across various NLP tasks without adding parameters.

What Problem Does This Paper Address?

Large language models (LLMs) have seen remarkable progress, but scaling them to trillions of parameters incurs substantial computational costs. Mixture-of-Experts (MoE) architectures offer a solution by dynamically activating only a subset of "experts" (sub-networks) per input, enabling conditional sparse computation. However, the performance of MoE models heavily relies on the "router," which decides which experts to activate. Crucially, current MoE training methods do not explicitly supervise this routing ability, relying on implicit learning through standard language modeling objectives. This paper highlights that this implicit learning is often insufficient, as demonstrated by significant performance degradation when routing decisions are randomly perturbed. The core problem is the lack of a direct mechanism to train and optimize the MoE router for accurate expert selection.

Methods

The paper proposes Exploration-Driven Reinforcement Learning (ERL). The core idea is to treat the MoE router as a policy in a reinforcement learning framework. For each input token, ERL performs two types of forward passes:

  1. Original Forward: The standard MoE forward pass using the router's current probability distribution to select experts.
  2. Perturbed Forward: A fraction ($\alpha$) of the routing decisions in the original forward pass are randomly disrupted. This means experts are selected randomly for a subset of tokens.

The difference in prediction performance (e.g., likelihood of the correct next token) between the original and perturbed forward passes is used to compute an advantage signal. This signal directly informs the router's update: if a perturbed path leads to better performance, the router is encouraged to explore similar paths; if it leads to worse performance, the original routing is reinforced. This process directly optimizes the router without requiring external reward models or additional parameters.

To mitigate the risk of "expert misclassification" (where over-specialization leads to poor performance if an inappropriate expert is chosen), the authors also introduce MoE-ERLwPL. This variant incorporates an auxiliary language modeling loss on the perturbed routing paths to encourage knowledge overlap among experts, making the model more robust to suboptimal expert selection.

Datasets

The proposed method was evaluated on the following standard NLP datasets:

  • Summarization:
    • SAMSum: 15k dialogue-summary pairs.
  • XSUM: 227k BBC news articles with single-sentence summaries.
  • Question Answering:
    • SQuAD: ~98k question-answer pairs from Wikipedia.
  • Language Modeling:
    • WikiText-2: A corpus of ~2.5M tokens, commonly used for small- to medium-scale models.

Key Insights and Ideas

  • Explicit Router Optimization is Crucial: The paper strongly demonstrates that MoE models are highly sensitive to the router's ability to select appropriate experts. Simply relying on standard training objectives is insufficient, and explicit optimization of the router is necessary for optimal performance.
  • Perturbation as a Learning Signal: Randomly perturbing routing decisions, a seemingly destructive action, is ingeniously repurposed as a powerful signal for reinforcement learning. The performance gap between the original and perturbed paths provides a direct, data-driven reward for the router.
  • No Extra Parameters Needed: A significant advantage of ERL is that it achieves improved performance by optimizing the existing router weights, without introducing new parameters or complex external modules, making it easily integrable into existing MoE models.
  • Balancing Specialization and Robustness: The MoE-ERLwPL variant highlights a trade-off. While ERL enhances routing quality, it can lead to over-specialization. The auxiliary loss in MoE-ERLwPL effectively encourages knowledge overlap, improving robustness at the cost of slightly lower routing certainty compared to pure ERL.

Key Results

  • Significant Routing Quality Improvement: MoE-ERL and MoE-ERLwPL consistently outperformed baseline MoE models across summarization, QA, and language modeling tasks. They achieved up to 8.9x higher Mean Reciprocal Rank (MRR) in routing quality compared to the baseline MoE.
  • Task-Specific Performance Gains:
    • Summarization (SAMSum, XSUM): MoE-ERL and MoE-ERLwPL showed notable improvements in ROUGE scores.
  • Question Answering (SQuAD): Modest gains were observed, likely due to the limited impact of decoder-side routing in extractive QA.
  • Language Modeling (WikiText-2): MoE-ERL reduced perplexity by 0.67 compared to the MoE baseline, and MoE-ERLwPL further improved it by 0.09.
  • Reduced Expert Misclassification Risk: MoE-ERLwPL demonstrated greater robustness to perturbations, with only minor performance degradation, thanks to the induced knowledge overlap.
  • Increased Routing Confidence: ERL models showed significantly higher top-1 expert routing probabilities, indicating increased confidence in expert selection.

Limitations and Future Work

  • Applicability to Decoder Models: The current ERL formulation is primarily applicable to decoder-only models. Adapting it to encoder models, which typically have single-document classification probabilities, is challenging due to the difficulty of assigning a single advantage value across multiple tokens and layers. Future work could explore token-level reward schemes for encoder-based MoE.
  • Increased Training Cost: ERL requires additional forward passes during training to compute perturbed paths, leading to higher training costs (e.g., 1.2x-2.8x longer training time depending on the model architecture). However, inference time remains unaffected.
  • Expert Misclassification Risk: While MoE-ERL improves routing, it can lead to over-specialization. MoE-ERLwPL addresses this, but future work could explore additional strategies to further reduce this risk while maintaining routing precision.

Summary updated

BibTeX
@inproceedings{kim2025exploration,
  author = {Gyunyeop Kim and Sangwoo Kang},
  title = {{Exploration-Driven Reinforcement Learning for Expert Routing Improvement in Mixture-of-Experts Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure