Mitigating the All-to-All Communication Bottleneck in Distributed MoEs

236 concepts156 papers4 proposals

Sparse Mixture-of-Experts (MoE) models offer a powerful architectural paradigm: they allow neural networks to scale to trillions of parameters while keeping the computational cost (FLOPs) per token relatively constant. As demonstrated by Dmitry Lepikhin et al. in GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, this is achieved through conditional computation, where a gating network dynamically routes each token to only a small subset of specialized "experts."

However, a severe physical constraint emerges when scaling these models across distributed hardware clusters. Because an MoE model contains far more parameters than can fit on a single GPU, the experts must be sharded across multiple physical devices or nodes—a strategy known as expert parallelism.

When tokens are distributed across devices via data parallelism, but the experts themselves are distributed via expert parallelism, every device must send its tokens to the specific devices hosting their assigned experts. This creates a massive, synchronized All-to-All communication pattern during the dispatch and combine phases. As models scale, this interconnect bottleneck frequently chokes cluster performance, often consuming over half of the total execution time and leaving high-performance accelerators idle.


The Core Serialization Problem

In standard dense neural networks, communication tasks (such as an All-Reduce to synchronize gradients) can be easily overlapped with the computation of subsequent layers. In distributed MoE architectures, however, a strict data dependency prevents this simple parallelization.

[Token Input] ──> [Gating Network] ──> [All-to-All Dispatch] ──> [Expert Computation] ──> [All-to-All Combine]

As analyzed by Weilin Cai et al. in Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts, the destination device for any given token is completely unknown until the gating network processes that token's current representation. This creates a hard serialization barrier. Standard optimization frameworks try to hide this latency by overlapping data transfers with computation strictly within the MoE layer itself. Yet, as Guichao Zhu et al. observe in FoldMoE: Efficient Long Sequence MoE Training via Attention-MoE Pipelining, the computational workload of the expert layers is often too small to fully mask the extensive network latency required for large-scale distributed routing, especially during long-sequence training.

To break this bottleneck, modern research focuses on three main strategies: predictive pipelining, locality-aware routing, and low-dimensional data shuffling.


Strategy 1: Predictive and Holistic Pipelining

If the primary bottleneck is the data dependency of the gating network, one intuitive solution is to compute or predict routing decisions early, widening the window available for data transfer.

In Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts, the authors decouple the gating network from the immediate layer's computation. By utilizing token representations from the preceding layer to trigger the All-to-All communication prematurely, they achieve up to a 100% overlap between network transfer and layer computation. A similar predictive approach is taken by Yan Li et al. in Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling, where a token's semantic identity is used to anticipate its expert path, allowing data and model experts to be co-scheduled proactively rather than reactively.

Alternatively, systems can pipeline communication across the entire Transformer block rather than isolating it to the MoE layer. The authors of FoldMoE introduce a "1A1M" pipeline that interleaves Multi-Head Attention (MHA) and MoE computations. By buffering tokens, they hide the All-to-All routing latency behind the heavy computational load of the attention mechanism. Yunqi Gao et al. extend this holistic scheduling in FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training by concurrently pipelining attention layers, gating functions, expert executions, and weight synchronizations to eliminate serial bottlenecks.


Strategy 2: Locality-Aware and Constrained Routing

Another way to bypass the All-to-All bottleneck is to minimize the physical distance tokens must travel. In a typical GPU cluster, intra-node bandwidth (e.g., NVLink) is orders of magnitude faster than inter-node bandwidth (e.g., InfiniBand). Standard load-balancing algorithms distribute tokens evenly across all experts globally, causing severe network contention on the slowest inter-node links.

Unconstrained Routing:  [Token on Node 1] ───(Slow Inter-Node Link)───> [Expert on Node 2]
Locality-Aware:         [Token on Node 1] ───(Fast Intra-Node Link)───> [Expert on Node 1]

To exploit this hardware hierarchy, Chang Chen et al. developed TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training, introducing an auxiliary loss that penalizes routing paths traversing slow network links. Similarly, Rui Liu et al. propose Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers, a regularization technique that occasionally forces tokens to remain on their local machine rather than traveling to a globally "perfect" expert. This not only accelerates training throughput but also acts as a powerful regularizer.

Other approaches restructure how data or experts are grouped:


Strategy 3: Low-Dimensional Data Shuffling

A third highly effective strategy targets the volume of the data payload being transmitted over the network. In standard MoE implementations, tokens are transmitted across the All-to-All interconnect at their full hidden dimensionality ($d_{\text{model}}$).

Standard MoE:  [Gate] ──> All-to-All (High Dim: d_model) ──> [Expert Projection] ──> [Expert Activation]
BigMac MoE:    [Gate] ──> [Down Projection] ──> All-to-All (Low Dim) ──> [Up Projection] ──> [Expert Activation]

In BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference, the authors reorder the internal projection layers of the MoE block to alleviate network congestion. Instead of performing the standard token routing first and projecting inside the expert, the model applies a descending projection layer before the All-to-All transfer.

By compressing tokens into a low-dimensional space prior to network dispatch, this structural shift reduces network traffic by up to 75%. Once the compressed tokens arrive at their destination device, they undergo an internal expert expansion to restore their original capacity. This simple mathematical reordering drastically lowers interconnect pressure, making it feasible to activate more experts per token without incurring prohibitive latency penalties.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 135 papers
  2. Version 1 · 2026-10-04 · 300 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_all_to_all_communication_bottleneck_all__3430ea40,
  title = {all-to-all communication bottleneck (all-to-all communication overhead)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/all-to-all-communication-bottleneck-all-to-all-communication-overhead/rdr_3430ea4027d147168b077f5e486c959e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_all_to_all_communication_bottleneck_all__3430ea40_v2,
  title = {all-to-all communication bottleneck (all-to-all communication overhead)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/all-to-all-communication-bottleneck-all-to-all-communication-overhead/rdr_3430ea4027d147168b077f5e486c959e/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake