AMPipe: Accelerating MoE Model Training with Intra-Block Pipelining

Yichao Fu, Yuhao Qing, Shixiong Zhao, Fanxin Li, Bocheng Xiao, Dong Huang, Heming CuiOpenReview 2024 · Submission to ICLR 2024 · Rejected · OpenReview

Abstract

The Mixture-of-Experts (MoE) architecture presents a compelling adaptation for expanding the model size of pre-trained models, such as large language models (LLMs), to enhance overall model performance (e.g., lower perplexity). However, as the sequence length (represented as $N$) increases, both the execution time of the attention layer ($O(N^2)$) and the all-to-all communication time of the MoE layer ($O(N)$, with a significant coefficient) become training bottlenecks. Current training systems have primarily focused on either optimizing the MoE layer (e.g., Tutel) or enhancing the attention layer (e.g., FlashAttention), yet they have demonstrated bounded performance improvements when confronted with long sequences---an essential consideration for modeling a potent language model with a long context window. In this paper, we introduce AMPipe, a novel pipeline system and paragdim for accelerating the training of large MoE models using Intra-Block Pipelining, particularly when dealing with lengthy sequences. AMPipe smartly optimizes two bottlenecks together by dividing and pipelining both the attention layer and MoE layer to strategically mitigate the time costs associated with these operations. Experimental results illustrate that AMPipe can consistently outperform current training systems, which solely focus on optimizing either the MoE or attention layer. Notably, AMPipe enhances the training throughput of a highly optimized transformer block by an average of 23\% across 56 benchmark cases and by up to 41\% in long sequence training, all without introducing statistical impact on model convergence. Our code is available at https://github.com/iclr24-3434/AMPipe.git
BibTeX
@misc{fu2024ampipe,
  author = {Yichao Fu and Yuhao QING and Shixiong Zhao and Fanxin Li and Bocheng Xiao and Dong HUANG and Heming Cui},
  title = {{AMPipe: Accelerating MoE Model Training with Intra-Block Pipelining}},
  year = {2024},
  url = {https://openreview.net/forum?id=yLgr02IsXY},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}

Loading related content...

Figure