PMR: MoE Routing with Shared Prototypes and Congestion Pricing

Wending Wang, Binqi Huang, Ju'an Li, Yu Guo, Lei Duan, Yuqi Ouyang, Dengchun LiarXiv 2026 · arXiv

TL;DR

Prototype-Mediated Routing (PMR) is a novel two-stage router for Mixture-of-Experts (MoE) models that decouples input feature matching from expert allocation. By using shared prototypes that adapt to context and a "congestion pricing" mechanism for load balancing, PMR significantly improves zero-shot accuracy and fine-tuning performance while maintaining high training throughput.

What Problem Does This Paper Address?

Sparse Mixture-of-Experts (MoE) models are the current darlings of LLM scaling, allowing models to have massive parameter counts while only activating a fraction of them per token. However, the "router"—the component that decides which expert handles which token—remains a bottleneck.

Conventional linear routers calculate a score for each expert based on a single learnable vector. This approach has two major flaws:

  1. Coupling: It forces the model to learn input feature matching and expert allocation simultaneously in one step.
  2. Implicit Mapping: It struggles to represent "many-to-many" relationships (e.g., one semantic feature being relevant to multiple experts, or one expert specializing in multiple distinct features).
  3. Load Balancing Woes: To prevent all tokens from going to a single "super-expert," researchers usually add "auxiliary losses." These can be finicky and sometimes conflict with the model's primary learning objective.

Methods

PMR introduces a two-stage routing framework that treats routing as a probabilistic matchmaking process:

  1. Stage 1: Context-Adaptive Prototype Matching: Instead of looking at experts immediately, the token is compared against a set of shared prototypes (representative feature vectors). These prototypes are not static; they are updated by "short-term states" that summarize the preceding tokens in the sequence, allowing the routing to be context-aware.
  2. Stage 2: Price-Regulated Associations: A learnable association table maps these prototypes to specific experts. To ensure experts are used evenly, PMR uses Congestion Pricing. If an expert was over-utilized in previous steps, its "price" increases, making it temporarily less attractive for the next batch of tokens. This achieves load balancing without needing a separate loss function.

To make this computationally feasible, the authors developed a causal two-pass blockwise approximation. This allows the model to process tokens in parallel blocks rather than one-by-one, which is critical for modern GPU training.

Datasets

The authors evaluated PMR across two main settings:

  • Pretraining: Models were trained on FineWeb-Edu (high-quality web text) and Cosmopedia v2 (synthetic textbooks and stories).
  • Zero-Shot Evaluation: Performance was measured on seven standard benchmarks: HellaSwag, PIQA, WinoGrande-XL, ARC-Easy, ARC-Challenge, OpenBookQA, and LAMBADA.
  • Fine-Tuning: They used the official training splits of PIQA, ARC-Easy, ARC-Challenge, and OpenBookQA to test PMR in Parameter-Efficient Fine-Tuning (PEFT) scenarios on backbones like Llama-2, Llama-3, and Qwen.

Key Insights and Ideas

  • Decoupling is Power: By separating "What is this token?" (prototypes) from "Who should handle it?" (associations), the model can more flexibly assign experts to complex, multi-faceted inputs.
  • Pricing > Auxiliary Losses: Congestion pricing acts as a "loss-free" balancing mechanism. It regulates expert load by adjusting the routing distribution directly based on feedback, rather than adding a penalty term to the gradient.
  • Contextual Routing: Most routers treat tokens in isolation. PMR’s use of short-term states means the router "remembers" what it just saw, leading to more consistent and intelligent expert selection across a sentence.
  • Non-Linearity: Unlike linear routers where the log-probability ratio between experts is a simple linear function of the input, PMR allows for non-linear relationships, providing a more expressive routing manifold.

Key Results

  • Accuracy Gains: Across three pretraining scales (0.43B to 1.18B parameters), PMR improved mean zero-shot accuracy by 1.45 to 2.65 percentage points (pp) over standard linear routing.
  • Specific Wins: On the ARC-Easy benchmark, PMR showed massive improvements of up to 6.71 pp.
  • Fine-Tuning Efficiency: When used with MixLoRA on 7B-8B backbones, PMR outperformed the baseline by up to 1.76 pp in mean accuracy across four tasks.
  • Throughput: The blockwise implementation is 21.8x faster than a naive tokenwise implementation, making it practical for large-scale training (achieving ~40,000 tokens/sec).

Limitations and Future Work

  • Hyperparameter Sensitivity: The system introduces new hyperparameters, such as the adaptation step size and congestion price update rates, which may require careful tuning for different architectures.
  • Prototype Scaling: While the paper tests up to 16 prototypes, it’s unclear how the system behaves with a much larger number of prototypes or experts.
  • Inference Overhead: Although the blockwise approximation helps training, the two-stage process is inherently more complex than a single linear layer, which might introduce slight latency during real-time, single-token inference.
  • Future Directions: The authors suggest exploring more sophisticated "short-term state" architectures and applying PMR to multi-modal MoE models.

Summary updated

Abstract

Conventional linear routers in sparse mixture-of-experts models select experts directly from token-expert matching scores, coupling input matching with expert allocation. We propose Prototype-Mediated Routing (PMR), a two-stage router with shared prototypes. PMR first computes each token's distribution over prototypes, then uses it to mix prototype-conditioned expert distributions according to the law of total probability. This formulation models input matching and expert allocation separately, explicitly represents many-to-many associations between prototypes and experts, and provides separate interfaces for context adaptation and load regulation. Within this framework, short-term states summarize preceding tokens to adapt the prototypes, while congestion prices updated from dispatch feedback achieve loss-free load balancing by adjusting the conditional expert distributions. We use a two-pass blockwise approximation to accelerate short-term state processing, achieving over 20 times the training throughput of the eager tokenwise PMR implementation while maintaining high agreement in routing decisions. Across three pretraining configurations with 0.43B, 1.18B, and 0.48B total parameters, PMR improves mean zero-shot accuracy on seven benchmarks over linear routing baselines with matched expert architectures by 1.54, 2.65, and 1.45 percentage points (pp), respectively. The corresponding gains on ARC-Easy are 6.71, 5.80, and 3.92 pp. In parameter-efficient fine-tuning of 7B-8B backbones, PMR's improvements over MixLoRA vary across backbones, with gains of up to 1.76 pp in mean accuracy across four tasks.
BibTeX
@misc{wang2026pmr,
  author = {Wending Wang and Binqi Huang and Ju'an Li and Yu Guo and Lei Duan and Yuqi Ouyang and Dengchun Li},
  title = {{PMR: MoE Routing with Shared Prototypes and Congestion Pricing}},
  year = {2026},
  eprint = {2611.19413},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19413},
}
Figure