SCoMoE: Efficient Mixtures of Experts with Structured Communication

zhiyuan zeng, Deyi XiongICLR 2023 · OpenReview

TL;DR

SCoMoE (Structured Communication Mixture-of-Experts) significantly speeds up the training of large-scale sparse models by optimizing the "all-to-all" communication bottleneck. By tailoring data transfer to match the physical hierarchy of compute clusters (fastest within a chip, slowest between nodes), it achieves up to a 1.44x speedup over the standard GShard implementation while maintaining or even improving translation quality.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) models are a popular way to scale neural networks to billions or trillions of parameters. Instead of every input passing through every part of the model, a "router" sends tokens to a small subset of "experts." This allows for sublinear scaling—massive capacity without a massive increase in compute cost.

However, there is a hidden "communication tax." Because experts are distributed across many different GPUs and servers (nodes), the model must constantly perform "all-to-all" communication to move tokens to their assigned experts. As models grow, this data movement becomes the primary bottleneck, often leaving expensive GPUs idling while they wait for data to arrive over slow inter-node network links.

Methods

The authors introduce SCoMoE, which replaces uniform all-to-all communication with a structured approach that respects the hardware's topology:

Hardware topology and SCoMoE data transformation strategies (Slicing vs. Projection).
  1. Structured All-to-All: The system recognizes three levels of bandwidth: intra-accelerator (fastest), intra-node (medium), and inter-node (slowest). It encourages more data to stay within the faster channels.
  2. Data Transformation:
    • SCoMoE-Seq (Slicing): Tokens are sliced into groups. Tokens with high affinity for local experts are kept on-device or within the node, reducing the volume of data sent over the slow global network.
  3. SCoMoE-Feat (Projection): Instead of slicing tokens, this method projects token embeddings into lower-dimensional representations for different communication levels, reducing the total number of bits moved.
  4. Token Clustering: To prevent performance loss from "routing locality" (where tokens can only see a subset of experts), the authors propose clustering related tokens on the same device before the MoE layer.
  5. Differentiable Sorting: The authors improve the "balanced router" (used for clustering) by replacing standard sigmoid gating with a softmax-based differentiable sorting mechanism. This allows for better gradient flow during training.

Datasets

The researchers evaluated SCoMoE on two major Neural Machine Translation (NMT) benchmarks:

  • WMT17-En-Fr: A bilingual dataset consisting of over 35 million sentence pairs.
  • OPUS-100: A massively multilingual dataset covering 100 languages and over 107 million training examples.

Key Insights and Ideas

  • Hardware-Awareness is Key: Standard MoE treats the connection between two GPUs on the same chip the same as two GPUs in different buildings. SCoMoE’s "aha moment" is that we can trade off a bit of routing flexibility for massive gains in communication efficiency by favoring high-bandwidth paths.
  • Clustering as Regularization: Surprisingly, the token clustering intended to fix performance drops actually improved results on multilingual tasks. By forcing related tokens together, the model seems to learn better representations, particularly for low-resource languages.
  • The Bottleneck Shift: The paper demonstrates that as you optimize communication, the overhead of the routing logic itself (like token clustering) becomes a non-negligible factor.

Key Results

SCoMoE demonstrates that structured communication doesn't have to come at the cost of accuracy. In fact, it often improves both speed and translation quality.

Performance comparison of SCoMoE variants against baselines on multilingual and bilingual tasks.
  • Speed: SCoMoE-Seq achieved a 1.44x speedup over GShard on bilingual tasks. On large-scale multilingual tasks (4.1B parameter model), variants like SCoMoE-Feat-α maintained up to a 1.32x speedup.
  • Quality: On the OPUS-100 multilingual benchmark, SCoMoE-Seq-β significantly outperformed GShard by 2.8 BLEU points (29.60 vs 26.78), proving the efficacy of the method.
  • Efficiency: SCoMoE consistently reduced communication time compared to both GShard and Tutel (another hierarchical MoE implementation), especially as the number of experts increased.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance of SCoMoE depends on tuning the ratios (α and β) of data sent through different communication levels. The "best" settings may vary depending on the specific hardware interconnect (e.g., NVLink vs. PCIe).
  • Clustering Overhead: While clustering improves quality and reduces communication, it adds computational overhead. In some configurations, this overhead can partially offset the communication savings.
  • Inference vs. Training: While the paper focuses heavily on training speed, the authors note that the speedup during autoregressive decoding (inference) is less pronounced because communication is bottlenecked by frequency rather than data size. Future work could focus on optimizing the inference path specifically.

Summary updated

Abstract

Mixture-of-Experts (MoE) models are promising architectures for massively multilingual neural machine translation and large language models due to the advantage of sublinear scaling. However, the training of large MoE models is usually bottlenecked by the all-to-all communication (Lepikhin et al., 2020). To reduce the communication cost, we propose SCoMoE, an MoE architecture with structured all-to-all communication, inspired by the hierarchical architecture of the communication topology. SCoMoE encourages data to be communicated across devices through fast intra-accelerator/node communication channels, reducing communication throughput in the slow inter-node communication channel. We slice the data on the sequence dimension (SCoMoE-Seq) into three communication groups and project the data on the feature dimension (SCoMoE-Feat) into low-dimensional representations. To compensate the potential performance drop caused by the routing locality in SCoMoE, we further propose a token clustering approach to aggregating related tokens from different devices before the MoE layers. The sigmoid gating in the balanced router used in the token clustering is substituted with the softmax gating with differential sorting. Experiments on bilingual and massively multilingual machine translation demonstrate that SCoMoE achieves a speedup of 1.44x over GShard with comparable performance, and substantially outperforms Gshard (2.8 BLEU) on OPUS-100 with a speedup of 1.25x.
BibTeX
@inproceedings{zeng2023scomoe,
  author = {zhiyuan zeng and Deyi Xiong},
  title = {{SCoMoE: Efficient Mixtures of Experts with Structured Communication}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=s-c96mSU0u5},
}
Figure