Sparsely activated Mixture-of-Experts (MoE) models scale model capacity without a linear increase in computational cost. By routing each token to only a small subset of specialized sub-networks ("experts"), MoEs achieve remarkable efficiency on paper. However, in distributed environments, this architectural flexibility introduces a massive physical bottleneck: the All-to-All communication phase.
When tokens are distributed across a cluster of accelerators, a token on GPU $A$ may need an expert residing on GPU $B$. Shuffling millions of tokens across slow network links to their designated experts creates severe latency penalties and hardware underutilization.
To overcome this bottleneck, a prominent research direction focuses on locality-aware expert placement and routing. Instead of treating token routing as a purely mathematical optimization, this paradigm co-designs the model's routing logic with the physical topology of the underlying hardware. By exploiting statistical patterns in how tokens select experts, these techniques maximize local computation and minimize expensive inter-device data transfers.
Affinity-Aware Expert Placement
The most direct way to reduce communication overhead is to ensure that experts frequently needed by the same tokens are physically co-located on the same device or high-bandwidth interconnect. Tokens often exhibit predictable affinity patterns, meaning certain groups of experts are consistently co-activated.
Olivier Beaumont et al. demonstrate in Leveraging Expert Usage to Speed up LLM Inference with Expert Parallelism that mapping experts to hardware units based on their joint activation probabilities maximizes parallel execution and avoids resource collisions. Rather than distributing experts uniformly or randomly, profiling these activation patterns allows systems to group highly correlated experts together.
This spatial alignment can be optimized across various hardware granularities:
- Wafer-Scale and Chiplet Architectures: In Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures, Shuqing Luo et al. cluster frequently co-activated experts on adjacent physical chiplets. This increases the probability that a token finds its required expert within a local high-bandwidth interconnect rather than requiring a global network traversal.
- Communication Graph Pruning: In Occult: Optimizing Collaborative Communications across Experts for Accelerated Parallel MoE Training and Inference, Shuqing Luo et al. restrict token routing to a limited subset of available devices. By selecting high-scoring experts that reside within a predefined number of GPUs, the system prunes the communication graph.
- Co-locating Squads: Mohan Zhang et al. propose a Collaboration-Constrained Routing (C2R) strategy in Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design. By forcing frequently collaborating experts into specialized "squads" on the same GPU, the system enables a send-once, use-many communication pattern, eliminating redundant token transmissions across the network.
To formalize these placement strategies, researchers leverage mathematical optimization frameworks. Yan Li et al. (Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling) and Alexandr Katrutsa et al. (Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inference) utilize Integer Linear Programming (ILP) to map experts onto physical cluster topologies (such as FatTree or Dragonfly). This minimizes the total number of network hops between dispatching layers and expert locations based on offline profiling data.
Localized Expert Replication and Dynamic Load Balancing
While clustering unique experts reduces communication, highly popular experts can still become severe bottlenecks. A complementary strategy is localized expert replication, where copies of high-demand or low-overhead experts are duplicated across multiple hardware nodes to serve tokens locally.
In MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts, Peng Jin et al. introduce the concept of placing parameter-free or low-overhead experts on every hardware node. When tokens are routed to these local experts, they completely bypass the expensive data transfers typically required by standard MoE architectures.
For standard parameter-heavy experts, replication must be handled dynamically to balance memory constraints and traffic fluctuations:
- Dynamic Migration: Xiaonan Nie et al. develop a framework called FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement, which adaptively duplicates and migrates "hot" experts across GPUs in real time based on actual data traffic. A greedy routing strategy then prioritizes processing tokens on the local GPU to minimize inter-device communication overhead.
- Temporal Masking of Overhead: In Libra: Effective yet Efficient Load Balancing for Large-scale MoE Inference, Jaehoon Yang et al. prioritize replicating experts that serve locally residing tokens. This maximizes the duration of the local computation phase, providing a larger temporal window to perform background system tasks (such as calculating optimal sharding plans for remote tokens) without stalling the GPU.
Replication and locality also redefine how systems handle capacity overflows. Standard MoE routers drop tokens or send them to remote nodes when an expert reaches its capacity limit. To mitigate this, Shwai He et al. (Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts) and Qipeng Guo et al. (Turn Waste into Worth: Rectifying Top-k Router of MoE) propose rerouting overflow tokens to underutilized experts residing on the same physical accelerator. This localized load balancing maintains high hardware utilization and reduces token loss without incurring the latency penalties of global All-to-All communication primitives.
Exploiting Data and Routing Locality
Instead of only moving experts to match token demands, a third facet of this research direction alters data scheduling or the routing function itself to enforce locality.
Tokens within a single sequence or training sample are not independent; they often share a strong semantic context. Xinyi Liu et al. exploit this in NetMoE: Accelerating MoE Training through Dynamic Sample Placement by implementing dynamic sample placement. Because tokens within a single training sample exhibit a strong preference for specific experts, the authors dynamically reassign entire lightweight training samples to the hardware nodes where their preferred experts reside. Moving lightweight data samples is significantly more efficient than moving heavy model parameters or routing individual tokens across high-latency network links.
Alternatively, the routing algorithm can be regularized to favor local hardware structures:
- Gating Dropout: Rui Liu et al. introduce a technique in Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers that randomly restricts token routing to experts located on the local machine rather than globally optimal experts across a distributed cluster. This bypasses the standard All-to-All communication phase entirely for those tokens, reducing network overhead while acting as a powerful regularizer.
- Locality-Based Regularization: In LocMoE: A Low-overhead MoE for Large Language Model Training, Jing Li et al. utilize a specialized loss function to bias the router toward selecting experts on the same physical server node. This prioritizes high-bandwidth intra-node communication over slower inter-node transfers.
- Local Routing Consistency: This natural tendency of routers is evaluated by Jingcong Liang et al. in Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models. The authors define "local routing consistency" as the property where a model repeatedly selects the same experts for consecutive tokens within a text segment. This locality is a critical mechanism for reducing overhead, allowing a small cache of local experts to serve a large number of sequential computations.
By aligning the mathematical routing of tokens with the physical layout of the hardware cluster, locality-aware expert placement transforms the communication-heavy MoE architecture into a highly localized, hardware-efficient paradigm.
Key Foundational Work
FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement (2023) — contributed 1 concept elements to this direction.
Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers (2022) — contributed 1 concept elements to this direction.