Modern Distributed Data-Parallel Large-Scale Pre-training Strategies For NLP models

Hao Bai

TL;DR

As deep learning models grow larger, distributed training becomes essential. This paper systematically compares six data-parallel strategies for training large NLP models like GPT-2 using PyTorch. The authors find that while Distributed Parameter Server with Apex mixed-precision excels on single nodes, Horovod with Apex offers the most robust and scalable solution across single and multiple nodes.

What Problem Does This Paper Address?

The ever-increasing size of deep learning models, particularly in Natural Language Processing (NLP), demands massive computational resources. Training these "Very Large Scale" (VLS) models efficiently requires distributed computing. Data parallelism, where data is distributed across multiple compute nodes, is a key technique to speed up training. However, choosing the right data-parallel strategy can be challenging due to the variety of frameworks and their varying performance characteristics. This paper aims to demystify this landscape by providing a comprehensive, empirical comparison of popular data-parallel strategies using a well-known NLP model (GPT-2) and the PyTorch framework, offering practical guidance for researchers and engineers.

Methods

The authors implemented and evaluated six data-parallel strategies for training the GPT-2 language model (100M parameters) on the HAL mini-supercomputer. The strategies explored are:

  • Single GPU: The baseline, training on a single GPU.
  • Single Parameter Server (SPS): A naive approach where one GPU acts as both a worker and a central parameter server.
  • Distributed Parameter Server (DPS): A more advanced strategy where multiple parameter servers are used, or each worker acts as a server.
  • Horovod: A popular framework that utilizes MPI for inter-node communication and the ring-allreduce algorithm for efficient synchronization.
  • Distributed Parameter Server with Apex mixed-precision: Combines the DPS strategy with NVIDIA's Apex library for mixed-precision (FP16/FP32) training to reduce memory usage and speed up computation.
  • Horovod with Apex mixed-precision: Integrates Horovod with Apex for similar benefits.

The evaluation involved both qualitative analysis of the strategies' behavior and quantitative analysis of GPU utilization and training time.

Datasets

The paper uses a dataset consisting of sentences extracted from a novel for training and evaluation. No new benchmarks were introduced.

Key Insights and Ideas

  • SPS is Inefficient: The Single Parameter Server strategy, while simple to implement, suffers from significant performance bottlenecks due to the centralized parameter server becoming a bottleneck, leading to very low GPU utilization.
  • DPS Shines on Single Node: For single-node training, the Distributed Parameter Server (DPS) strategy, especially when combined with Apex mixed-precision, demonstrates excellent performance and efficiency.
  • Horovod's Robustness: Horovod, due to its ring-allreduce algorithm and built-in memory management features, proves to be a more robust and scalable solution, particularly when dealing with multiple nodes or potential memory constraints.
  • Apex is a Game-Changer: Integrating Apex mixed-precision training significantly boosts performance for both DPS and Horovod, reducing training time and memory footprint.
  • Inter-node Communication is a Bottleneck: When scaling to multiple nodes, inter-node communication becomes a significant performance bottleneck, impacting overall training speed.

Key Results

  • Single Node Performance:
    • DPS with Apex mixed-precision achieved the best performance on a single node.
  • Horovod with Apex was only slightly behind DPS on a single node but offered better robustness.
  • SPS performed significantly worse than both DPS and Horovod, even underperforming the single-GPU baseline.
  • Training Time:
    • For the 100M parameter GPT-2 model, DPS-Apex took around 133 hours, while Horovod-Apex took around 198 hours on a single node.
  • When using Apex, Horovod showed a performance improvement of approximately a factor of 2 compared to DPS.
  • Multi-node Horovod configurations showed that increasing the number of nodes did not always translate to proportional speedups due to inter-node communication overhead.
  • GPU Utilization: Horovod with multiple nodes showed significantly higher GPU utilization (summed across nodes) compared to other strategies.

Limitations and Future Work

  • Single Node Focus: While the paper compares single-node performance extensively, the multi-node analysis is more limited, particularly for DPS.
  • PyTorch Centric: The study is primarily focused on PyTorch. A comparison with TensorFlow would provide a more complete picture.
  • Model Parallelism: The paper focuses solely on data parallelism. Future work will explore model parallelism to offer a more comprehensive view of distributed training strategies.
  • Dataset Size: The experiments were conducted on a relatively small dataset, and the performance might vary with larger datasets.

The authors plan to extend their research to include quantitative results for data parallelism on TensorFlow and to investigate the power of model parallelism on PyTorch.

Summary updated

Abstract

Distributed deep learning is becoming increasingly popular due to the expanding demand for computing resources for deep learning models with a larger amount of parameters. Different from traditional training approaches, data-parallel training allows multiple compute nodes to train large deep learning models simultaneously in order to boost the training efficiency. In this paper, we present and compare six strategies for data-parallel training using PyTorch on the language model GPT-2 with 100M parameters using a qualitative approach. These strategies are Single GPU, Single Parameter Server, Distributed Parameter Server, Horovod, Distributed Parameter Server with Apex mixed-precision strategy, and Horovod with Apex mixed-precision strategy. We also analyze the quantitative experiment results from each strategy. In the end, we draw the conclusion that the Distributed Parameter Server with Apex mixed-precision strategy has the best performance on single node training, while Horovod with Apex is the most robust approach to use when we have single or multiple nodes.
BibTeX
@inproceedings{bai2021modern,
  author = {Hao Bai},
  title = {{Modern Distributed Data-Parallel Large-Scale Pre-training Strategies For NLP models}},
  booktitle = {HP3C},
  year = {2021},
  url = {https://openreview.net/forum?id=j2T-0oEbrz},
}
Figure