LightSeq: Accelerated Training for Transformer-based Models on GPUs

Xiaohui Wang, Ying Xiong, Xian Qian, Yang Wei, Lei Li, Mingxuan Wang

TL;DR

This paper introduces LightSeq2, a system designed to significantly accelerate the training of Transformer-based models on GPUs. By developing a suite of GPU optimization techniques tailored to Transformer architectures, LightSeq2 achieves substantial speedups and memory savings across various models and tasks, making large-scale AI model training more accessible.

What Problem Does This Paper Address?

Training large Transformer models, which are ubiquitous in AI applications like NLP and computer vision, is notoriously expensive and time-consuming. This is due to their complex computational patterns, variable input lengths, and the sheer scale of parameters. Existing acceleration systems often focus solely on model inference or are limited to specific Transformer variants like BERT (encoder-only). This leaves a significant gap for accelerating the training of more general Transformer architectures, such as encoder-decoder models used in machine translation or decoder-only models like GPT, which are crucial for many generative AI tasks. The challenge lies in optimizing both the forward and backward passes, managing memory efficiently for large intermediate activations, and ensuring training quality and reproducibility.

Methods

LightSeq2 tackles these challenges with a three-pronged approach:

  • Layer-Specific Kernels for GPU Utilization: The system fuses small, fine-grained GPU kernels into larger, coarse-grained ones. This reduces kernel launch overhead and GPU memory I/O for intermediate results during both forward and backward passes. They analyze Transformer-specific layers to identify opportunities for parallelism and kernel fusion.
  • Fine-Grain Mixed-Precision Trainer: To accelerate the parameter update process (optimizer step), LightSeq2 employs a mixed-precision strategy. Instead of updating full-precision (FP32) parameters individually, it performs batched updates on reduced-precision (FP16) parameters. This is achieved through a "symbolic tensor linking" and "on-the-fly conversion" mechanism, which avoids redundant FP32 copies and reduces memory footprints.
  • Efficient GPU Memory Management: Training requires storing intermediate activations for the backward pass, which can consume vast amounts of GPU memory. LightSeq2 implements a "dangling-tensor aware memory manager" that estimates the maximum memory needed for intermediate tensors and allocates it once upfront. This memory is then reused across different batches, significantly reducing memory fragmentation and the overhead of frequent allocation/deallocation.

Datasets

The paper evaluates LightSeq2 on several standard benchmarks:

  • Machine Translation: WMT14 English-German dataset.
  • Image Classification: CIFAR-10 dataset.
  • Text Classification: GLUE benchmark (specifically MRPC task for BERT).
  • Language Modeling: WikiText dataset.

The paper does not introduce new benchmarks but rather uses existing ones to demonstrate the effectiveness of LightSeq2 across diverse tasks and model types.

Key Insights and Ideas

  • Unified Optimization for General Transformers: Unlike prior work focusing on specific architectures (e.g., BERT), LightSeq2 provides a general solution applicable to encoder-only, decoder-only, and encoder-decoder Transformer models, as well as Vision Transformers.
  • Backward Pass is a Bottleneck: The paper highlights that while forward passes are often optimized, the backward pass presents unique challenges (e.g., sparse aggregation, complex dependencies) that also require specialized optimization.
  • Memory Management is Crucial for Large Models: For very large models, memory bandwidth and efficient memory utilization become more critical than raw computational throughput, and LightSeq2's memory manager directly addresses this.
  • Symbolic Tensor Linking for Trainer Efficiency: The novel approach to mixed-precision training, by symbolically linking tensors and performing on-the-fly conversions, significantly reduces memory overhead and improves trainer speed.

Key Results

LightSeq2 demonstrates significant improvements across various models and hardware:

  • Machine Translation (WMT14 English-German): Achieves up to 308% training speedup and uses 65% GPU memory compared to PyTorch on eight NVIDIA Tesla A100 GPUs.
  • General Transformer Training: Consistently faster than previous systems by 1.4-3.5x on different GPUs.
  • Vision Transformer (ViT): Achieves 1.2-1.7x speedup on ViT-B-32 and 1.2-1.5x speedup on ViT-L-32 compared to Hugging Face.
  • BERT: Achieves 1.44x speedup for BERT-Base and 1.28x speedup for BERT-Large compared to DeepSpeed.
  • GPT-2: Achieves 1.7-1.8x speedup for GPT-2 Base on V100 and 1.6-1.9x speedup for GPT-2 Large on A100 compared to Hugging Face.
  • Memory Savings: Reduces GPU memory usage by up to 2 GB and runtime by 54.9% compared to Fairseq on a Transformer-Base model.

Limitations and Future Work

  • GEMM Kernel Optimization: The paper notes that for Vision Transformer models, the speedup decreases as batch size increases because the proportion of time spent on General Matrix Multiply (GEMM) kernels grows. LightSeq2 currently does not deeply optimize GEMM kernels, which presents an area for future work.
  • Synchronization Overhead: For distributed training scenarios with an increasing number of GPUs or model sizes, the proportion of time spent on synchronization increases, leading to diminishing speedup. Future work could explore gradient compression techniques to mitigate this.
  • Padding Removal: The authors suggest that further optimization could be achieved by removing padding more effectively.
  • Better Memory Management: While LightSeq2 significantly improves memory management, there's still room for additional optimization in this area.

Summary updated

Abstract

Transformer-based neural models are used in many AI applications. Training these models is expensive, as it takes huge GPU resources and long duration. It is challenging because typical data like sentences have variable lengths, and Transformer's computation patterns are more complex than convolutional neural networks. Existing systems either only focus on model inference or optimization for only BERT-like encoder models. In this paper, we present LightSeq2, a system to accelerate training for a general family of Transformer models on GPUs. We propose a series of GPU optimization techniques tailored to the specific computation flow and memory access patterns of Transformer models. LightSeq2 supports many model architectures, including BERT (encoder-only), GPT (decoder-only), Transformer (encoder-decoder), and vision Transformer. Our experiments for a variety of models and benchmarks show that LightSeq2 is consistently faster (1.4-3.5x) than previous systems on different GPUs. In particular, it gains 308% training speedup compared with existing systems on a large public machine translation benchmark (WMT14 English-German).
BibTeX
@inproceedings{wang2022lightseq,
  author = {Xiaohui Wang and Ying Xiong and Xian Qian and Yang Wei and Lei Li and Mingxuan Wang},
  title = {{LightSeq: Accelerated Training for Transformer-based Models on GPUs}},
  booktitle = {SC},
  year = {2022},
  doi = {10.1109/sc41404.2022.00043},
}
Figure