LightSeq2: Accelerated Training for Transformer-Based Models on GPUs

Xiaohui Wang, Yang Wei, Ying Xiong, Guyue Huang, Xian Qian, Yufei Ding, Mingxuan Wang, Lei Li

TL;DR

Training massive Transformer models is a significant bottleneck in AI development due to high computational costs and memory demands. LightSeq2 introduces a novel system that accelerates Transformer training across various architectures (encoder-only, decoder-only, encoder-decoder, and vision Transformers) by optimizing GPU utilization and memory management, achieving up to 3.5x speedup.

What Problem Does This Paper Address?

The paper tackles the escalating cost and time required to train Transformer-based models, which are now ubiquitous in NLP, computer vision, and speech recognition. Existing acceleration techniques often focus solely on inference or are limited to specific model types like BERT (encoder-only). This leaves a gap for a general-purpose training acceleration system that can handle the full spectrum of Transformer architectures, including encoder-decoder models crucial for tasks like machine translation, and that addresses the unique challenges of training, such as backward propagation and parameter updates. The variable input lengths inherent in natural language also pose a challenge for many optimization methods.

Methods

LightSeq2 employs a multi-pronged approach to accelerate Transformer training:

  • Computational Graph Optimizations: This involves fusing small, fine-grained GPU kernels into larger, coarse-grained ones. This reduces kernel launch overhead and GPU memory I/O for intermediate results, particularly for operations like embedding, encoder, and decoder layers.
  • Dependent Reduction Rewriting: LightSeq2 optimizes reduction operations like LayerNorm and Softmax, which are critical for Transformer performance. They achieve this by rewriting the gradient calculation formulas to enable parallel computation and efficient memory access.
  • Accelerated Mixed-Precision Update for Trainer: To speed up the parameter update process (which can be a significant portion of training time), LightSeq2 uses a mixed-precision approach. It leverages symbolic tensor linking and on-the-fly conversion between FP16 and FP32 to reduce memory footprints and data movement without sacrificing accuracy.
  • Dangling-Tensor Aware Memory Manager: This technique addresses the memory-intensive nature of training by compacting memory usage. It pre-allocates a fixed amount of memory for parameters and gradients and reuses temporary memory blocks for intermediate tensors, avoiding frequent allocation/deallocation cycles and enabling larger batch sizes.

Datasets

The paper evaluates LightSeq2 on several standard benchmarks:

  • Machine Translation: WMT14 English-German dataset.
  • Image Classification: CIFAR-10 dataset.
  • Text Classification: GLUE benchmark (specifically the MRPC task for BERT).
  • Language Modeling: WikiText dataset.

No new benchmarks were introduced.

Key Insights and Ideas

  • Training is Fundamentally Different from Inference: The paper highlights that accelerating training involves unique challenges beyond inference, particularly the backward pass and parameter updates, which require different optimization strategies.
  • Generalization is Key: Unlike many prior systems that focus on specific architectures (e.g., BERT), LightSeq2's strength lies in its ability to accelerate a broad family of Transformer models, including encoder-decoder and vision Transformers.
  • Memory Management is Crucial for Large Models: The memory manager's ability to pre-allocate and reuse memory is vital for training larger models that would otherwise run into out-of-memory errors.
  • Kernel Fusion and Optimized Reductions Pay Off: The detailed breakdown shows that optimizing core operations like LayerNorm and Softmax, along with fusing kernels, significantly contributes to the overall speedup.
  • Mixed-Precision Training for Trainers: Applying mixed-precision not just to model computations but also to the trainer's parameter update step offers substantial benefits.

Key Results

  • Overall Speedup: LightSeq2 achieves consistent speedups across various models and GPUs, ranging from 1.4x to 3.5x compared to existing systems.
  • Machine Translation: A 308% training speedup was observed on the WMT14 English-German benchmark using eight NVIDIA Tesla A100 GPUs.
  • Vision Transformer (ViT): Speedups of 1.2x-1.7x were achieved on ViT-B-32 and ViT-L-32 models compared to Hugging Face.
  • BERT: Speedups of 1.44x for BERT-Base and 1.28x for BERT-Large were reported compared to DeepSpeed.
  • GPT-2: Speedups of 1.7-1.8x for GPT-2 Base and 1.6-1.9x for GPT-2 Large were achieved compared to Hugging Face.
  • Memory Efficiency: LightSeq2 uses significantly less GPU memory, for example, requiring 65% of the GPU memory for a WMT14 English-German task compared to PyTorch. It also avoids the gradual memory increase seen in Fairseq due to dynamic allocation.

Limitations and Future Work

  • GEMM Optimization: The paper notes that for models with simpler computational graphs like ViT, where matrix multiplication (GEMM) dominates, the speedup decreases as batch size increases because LightSeq2 does not currently optimize GEMM kernels directly.
  • Synchronization Overhead: As the number of GPUs or model sizes increases, the proportion of time spent on synchronization grows, leading to diminishing returns in speedup.
  • Future Work: The authors suggest further optimization in areas like padding removal and exploring better memory management strategies. They also mention integrating synchronization optimization techniques like gradient compression, which are orthogonal to their current work.

Summary updated

Abstract

Transformer-based neural models are used in many AI applications. Training these models is expensive, as it takes huge GPU resources and long duration. It is challenging because typical data like sentences have variable lengths, and Transformer's computation patterns are more complex than convolutional neural networks. Existing systems either only focus on model inference or optimization for only BERT-like encoder models. In this paper, we present LightSeq2, a system to accelerate training for a general family of Transformer models on GPUs. We propose a series of GPU optimization techniques tailored to the specific computation flow and memory access patterns of Transformer models. LightSeq2 supports many model architectures, including BERT (encoder-only), GPT (decoder-only), Transformer (encoder-decoder), and vision Transformer. Our experiments for a variety of models and benchmarks show that LightSeq2 is consistently faster (1.4-3.5 x) than previous systems on different GPUs. In particular, it gains 308 % training speedup compared with existing systems on a large public machine translation benchmark (WMTI4 English-German).
BibTeX
@inproceedings{wang2022lightseq,
  author = {Xiaohui Wang and Yang Wei and Ying Xiong and Guyue Huang and Xian Qian and Yufei Ding and Mingxuan Wang and Lei Li},
  title = {{LightSeq2: Accelerated Training for Transformer-Based Models on GPUs}},
  booktitle = {SC},
  year = {2022},
  doi = {10.1109/sc41404.2022.00043},
}
Figure