Locality Aware Expert Placement and Routing

33 concepts18 papers1 proposal

Sparsely activated Mixture-of-Experts (MoE) models scale model capacity without a linear increase in computational cost. By routing each token to only a small subset of specialized sub-networks ("experts"), MoEs achieve remarkable efficiency on paper. However, in distributed environments, this architectural flexibility introduces a massive physical bottleneck: the All-to-All communication phase.

When tokens are distributed across a cluster of accelerators, a token on GPU $A$ may need an expert residing on GPU $B$. Shuffling millions of tokens across slow network links to their designated experts creates severe latency penalties and hardware underutilization.

To overcome this bottleneck, a prominent research direction focuses on locality-aware expert placement and routing. Instead of treating token routing as a purely mathematical optimization, this paradigm co-designs the model's routing logic with the physical topology of the underlying hardware. By exploiting statistical patterns in how tokens select experts, these techniques maximize local computation and minimize expensive inter-device data transfers.

Affinity-Aware Expert Placement

The most direct way to reduce communication overhead is to ensure that experts frequently needed by the same tokens are physically co-located on the same device or high-bandwidth interconnect. Tokens often exhibit predictable affinity patterns, meaning certain groups of experts are consistently co-activated.

Olivier Beaumont et al. demonstrate in Leveraging Expert Usage to Speed up LLM Inference with Expert Parallelism that mapping experts to hardware units based on their joint activation probabilities maximizes parallel execution and avoids resource collisions. Rather than distributing experts uniformly or randomly, profiling these activation patterns allows systems to group highly correlated experts together.

This spatial alignment can be optimized across various hardware granularities:

To formalize these placement strategies, researchers leverage mathematical optimization frameworks. Yan Li et al. (Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling) and Alexandr Katrutsa et al. (Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inference) utilize Integer Linear Programming (ILP) to map experts onto physical cluster topologies (such as FatTree or Dragonfly). This minimizes the total number of network hops between dispatching layers and expert locations based on offline profiling data.

Localized Expert Replication and Dynamic Load Balancing

While clustering unique experts reduces communication, highly popular experts can still become severe bottlenecks. A complementary strategy is localized expert replication, where copies of high-demand or low-overhead experts are duplicated across multiple hardware nodes to serve tokens locally.

In MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts, Peng Jin et al. introduce the concept of placing parameter-free or low-overhead experts on every hardware node. When tokens are routed to these local experts, they completely bypass the expensive data transfers typically required by standard MoE architectures.

For standard parameter-heavy experts, replication must be handled dynamically to balance memory constraints and traffic fluctuations:

Replication and locality also redefine how systems handle capacity overflows. Standard MoE routers drop tokens or send them to remote nodes when an expert reaches its capacity limit. To mitigate this, Shwai He et al. (Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts) and Qipeng Guo et al. (Turn Waste into Worth: Rectifying Top-k Router of MoE) propose rerouting overflow tokens to underutilized experts residing on the same physical accelerator. This localized load balancing maintains high hardware utilization and reduces token loss without incurring the latency penalties of global All-to-All communication primitives.

Exploiting Data and Routing Locality

Instead of only moving experts to match token demands, a third facet of this research direction alters data scheduling or the routing function itself to enforce locality.

Tokens within a single sequence or training sample are not independent; they often share a strong semantic context. Xinyi Liu et al. exploit this in NetMoE: Accelerating MoE Training through Dynamic Sample Placement by implementing dynamic sample placement. Because tokens within a single training sample exhibit a strong preference for specific experts, the authors dynamically reassign entire lightweight training samples to the hardware nodes where their preferred experts reside. Moving lightweight data samples is significantly more efficient than moving heavy model parameters or routing individual tokens across high-latency network links.

Alternatively, the routing algorithm can be regularized to favor local hardware structures:

By aligning the mathematical routing of tokens with the physical layout of the hardware cluster, locality-aware expert placement transforms the communication-heavy MoE architecture into a highly localized, hardware-efficient paradigm.

Key Foundational Work

FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement (2023) — contributed 1 concept elements to this direction.
Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_locality_aware_expert_placement_658e7bd4,
  title = {locality-aware expert placement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/locality-aware-expert-placement/rdr_658e7bd45fea45da9f4d8886fdd7986b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_locality_aware_expert_placement_658e7bd4_v1,
  title = {locality-aware expert placement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/locality-aware-expert-placement/rdr_658e7bd45fea45da9f4d8886fdd7986b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake