LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes

Jiaming Hu, Zhida Jiang, Zhaolong Xing, Yu Li, Zhen Chen, Ke ZhangarXiv 2026 · arXiv

TL;DR

LRMoE is a memory-aware training system for Sparse Mixture-of-Experts (MoE) models that tackles load imbalance by selectively dropping less important token-expert pairs during the backward pass. By combining a novel "contribution-aware" importance metric with physical workload compaction, it speeds up training by up to 44.8% while staying within strict GPU memory and bandwidth limits.

What Problem Does This Paper Address?

Training massive Sparse MoE models is a balancing act. While they offer huge parameter capacity, the "routing" process—where tokens are sent to specific "expert" sub-networks—is often highly uneven. Some experts become "hot," receiving far more work than others, which creates massive computation and communication bottlenecks.

Existing solutions usually try to redistribute this work through expert replication or adaptive placement. However, these methods hit a "memory wall": adding more expert copies consumes precious GPU memory that is also needed for token buffers and optimizer states. When a cluster is memory-constrained, you simply can't replicate your way out of a massive load imbalance. The authors argue that instead of just moving a fixed workload around, we need to recalibrate the demand itself to fit the available hardware capacity.

Figure 1: Uncalibrated load. All layers exceeds the maximum capacity for load balancing.

Methods

The core of LRMoE is a two-phase design that separates the forward and backward passes to optimize resource usage:

  1. Forward Planning & Execution: The system keeps the forward pass completely intact. This ensures that every token gets the full benefit of the model's capacity for prediction. During this phase, the controller observes the routed load and sets "retention quotas" for the backward pass based on available GPU memory and bandwidth.
  2. Backward Load Recalibration: During backpropagation, the system uses a new Load Recalibration Metric (LRM) to rank the importance of each token-expert pair. LRM estimates how much a specific pair contributes to the expert's weight update by combining three factors: routing strength, the magnitude of the incoming gradient, and the magnitude of the saved activation.
  3. Selective Dropping & Physical Compaction: Only the highest-ranked pairs (within the pre-set quotas) are kept for the backward update. Crucially, LRMoE doesn't just "mask" the dropped work; it physically compacts the retained data into smaller tensors. This reduces communication traffic (All-to-All) and allows for variable-size "grouped GEMMs" (matrix multiplications) that skip the dropped rows entirely, freeing up both compute cycles and memory.
Figure 2: Selective backward load calibration for uncalibrated load (a) and calibrated load (b). In backward, LRMoE drops low-value token-expert pairs and reassign, ensuring only retained pairs with load fitting within per-GPU capacity enter All-to-All, eliminating congestion.

Datasets

The authors evaluated LRMoE using two large-scale language models:

  • Qwen3-30B-A3B (30B total parameters, 3B active)
  • GLM4.5-106B-A12B (106B total parameters, 12B active)

Both were trained using the Alpaca SFT dataset, a common corpus for fine-tuning large language models.

Key Insights and Ideas

  • Demand as a Variable: The most striking idea is treating backward workload as a "scheduling variable" rather than an immutable consequence of the forward pass. By accepting a slightly approximate gradient update, the system gains a powerful lever to manage hardware bottlenecks.
  • The LRM Proxy: Computing the exact contribution of every token to every weight update is too expensive. The LRM is a clever "rank-one" approximation that provides a high-fidelity importance score using data already available in the training loop, without requiring extra heavy computation.
  • Physical vs. Logical Reduction: Simply zeroing out gradients (masking) doesn't save memory or time because the hardware still processes the full-sized matrices. LRMoE’s "physical compaction" (shown in Figure 5) is what actually delivers the performance gains.
Figure 4: Expert replication reduction process. Dropped token-expert backward contributions are removed before expert-backward computation, while retained gradients and saved expert states are packed into contiguous expert seg- ments. Their counts and offsets define variable-𝑀grouped GEMMs without padding. The reduced per-expert row counts lower both expert-backward computation and the number of physical expert replicas required to satisfy the capacity bound.

Key Results

  • Faster Training: LRMoE reduced the cumulative training time to reach target loss levels by 21.2% to 44.8% compared to SYMI, a state-of-the-art baseline.
  • Memory Efficiency: It achieved a 57.8% reduction in the memory subtotal required for expert placement, replicas, and token buffers compared to fixed-ownership strategies.
  • Throughput Stability: While standard methods lose throughput rapidly as batch sizes increase (due to memory pressure), LRMoE maintained 100% effective throughput even at high batch-size multipliers where other systems failed.
  • Convergence: Training curves showed that dropping up to 20% of the backward workload had a negligible impact on model convergence, proving that the LRM metric successfully identifies the most "disposable" computations.

Limitations and Future Work

  • Approximate Updates: The primary limitation is that the backward pass is no longer exact. While empirical results are strong, the long-term impact on extremely large-scale pre-training (trillions of tokens) remains to be seen.
  • Metric Scope: The LRM is currently a "projection-level" proxy. The authors note that for more complex, multi-layer expert architectures, the formula might need further refinement to remain an accurate proxy for the total parameter gradient.
  • Future Directions: The authors suggest exploring how this demand-side recalibration could be integrated with other techniques like quantization or more advanced topology-aware scheduling to further push the limits of memory-constrained training.

Summary updated

Abstract

Sparse Mixture-of-Experts (MoE) models offer scalable capacity for Web applications, but uneven routing creates computation and communication bottlenecks. Existing placement and replication methods redistribute this workload, yet additional expert copies compete with token-related buffers for limited GPU memory. We present LRMoE, a memory-aware MoE training system that jointly recalibrates backward demand and balances the retained workload. During forward execution, it plans retention quotas and expert placement under resource constraints while preserving all routed computations. During backpropagation, a load recalibration metric estimates each token-expert pair's contribution to the expert update and selects pairs within these quotas. A capacity-aware routing algorithm distributes retained pairs among eligible instances of their original experts. Physical compaction then forms smaller communication payloads and variable-size grouped GEMMs, reducing backward traffic, computation, and execution-buffer demand. Experiments on Qwen3-30B-A3B and GLM4.5-106B-A12B show that LRMoE reduces cumulative MoE-layer training time to fixed-loss targets by 21.2-44.8% relative to SYMI.
BibTeX
@misc{hu2026lrmoe,
  author = {Jiaming Hu and Zhida Jiang and Zhaolong Xing and Yu Li and Zhen Chen and Ke Zhang},
  title = {{LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes}},
  year = {2026},
  eprint = {2610.37720},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37720},
}
Figure