Mixture-of-Experts (MoE) architectures allow models to scale to trillions of parameters by activating only a small subset of "experts" for any given input. However, this efficiency is often theoretical. In practice, MoE models suffer from a chronic "hotspot" problem: certain experts are naturally more popular than others, leading to massive computational imbalances where a few GPUs are overwhelmed while the rest of the cluster sits idle.
Recent research has shifted from static model partitioning to dynamic expert management, where the system actively moves or replicates expert weights across the cluster in response to real-time demand.
The Bottleneck: The "Hot Expert" Problem
In a standard MoE implementation, experts are statically assigned to specific GPUs. When a gating network routes a disproportionate number of tokens to a single expert, the GPU hosting that expert becomes a bottleneck. As Shengwei Li and Zhiquan Lai observe in Pro-Prophet, this "hotspot" problem leads to wasted GPU cycles and stalls the entire training or inference pipeline.
Traditional solutions relied on "auxiliary losses" to force the model to use experts more uniformly. However, these constraints can hurt model convergence and accuracy. The emerging consensus is to solve load balancing at the system level rather than the algorithmic level, preserving the model's natural learning dynamics while optimizing hardware utilization.
Dynamic Placement: Moving Weights to the Work
One approach to fixing imbalance is to move the expert weights to where the idle compute is. Xuan-Phi Nguyen and colleagues introduced Least-Loaded Expert Parallelism (LLEP) in Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts. LLEP monitors the computational load across a cluster and migrates expert parameters and their assigned tokens from overloaded GPUs to idle ones at runtime. This migration happens without altering the model's mathematical output, achieving significant speedups by ensuring no GPU remains a straggler.
Similarly, Yechan Kim and the team behind ES-MoE use a greedy approximation algorithm to place experts on available hardware based on the actual token distribution from the gating network. By dynamically scheduling these placements, they eliminate the need for "zero-padding"—a common but wasteful technique used to make uneven workloads appear uniform to the hardware.
Selective Replication: Trading Memory for Throughput
If an expert is so popular that a single GPU cannot handle its load even with dynamic placement, the system must replicate it. By placing copies of "hot" experts on multiple nodes, the system can distribute the token load across several processing units simultaneously.
The challenge is doing this without exhausting GPU memory. Lehan Pan and Ziyang Tao propose GRACE-MoE (/work/grace-moe-grouping-and-replication-with-locality-aware-dwbr), which selectively replicates only the most frequently accessed experts. This strategy resolves the conflict between minimizing communication and maintaining uniform resource utilization.
To make this replication effective, the system must predict which experts will be "hot." Haiyue Ma and Yiran Chen developed MoE-GPS, which uses a lightweight "Distribution-Only" prediction strategy to anticipate token distributions and trigger duplication before bottlenecks occur.
Disaggregation: Decoupling Attention and Experts
A more radical shift involves breaking the monolithic Transformer block entirely. In standard models, every GPU handles both Attention and MoE layers. However, these layers have different resource requirements: Attention is memory-bandwidth intensive, while MoE is compute and parameter intensive.
Systems like JANUS, developed by Zhiwen Zhang and Ye Wang, disaggregate these components into independent sub-clusters (Janus: Disaggregating Attention and Experts for Scalable MoE Inference). By decoupling them, JANUS can scale the expert pool independently and use a specialized scheduler to manage expert replicas based on real-time activation frequency. Ao Xiao and the Huawei team take a similar approach with xDeepServe (/work/xdeepserve-model-as-a-service-on-huawei-cloudmatrix384), using a "Transformerless" disaggregated architecture to achieve massive throughput on large-scale SuperPods.
Resilience through Elasticity
Beyond performance, dynamic expert management provides a path to hardware resilience. In traditional distributed training, a single GPU failure requires restarting from a checkpoint.
Wenjie Qu and Xueshen Liu introduced Lazarus (/work/lazarus-resilient-and-elastic-training-of-mixture-6en7), which uses adaptive expert replication to provide redundancy. If a node fails, the system already has replicas of critical experts elsewhere, allowing training to continue without interruption. Similarly, Chujie Chen and Xiao He developed AnchorTP (/work/anchortp-resilient-llm-inference-with-state-preserving), which uses "elastic" tensor parallelism to reconfigure surviving GPUs with minimal data movement, slashing recovery times from minutes to seconds.
By treating expert weights as fluid resources rather than static assignments, these systems transform the inherent imbalance of MoE models into an opportunity for both extreme scale and robust fault tolerance.