Characterizing Deep Learning Training Workloads on Alibaba-PAI

Mengdi Wang, Chen Meng, Guoping Long, Chuan Wu, Jun Yang, Wei Lin, Yangqing Jia

Introduction

Researchers from Alibaba Group and the University of Hong Kong have provided a comprehensive analysis of production-scale deep learning (DL) training workloads. By examining tens of thousands of jobs on Alibaba’s Platform of Artificial Intelligence (PAI), the study moves beyond synthetic benchmarks to identify real-world bottlenecks in industrial AI clouds.

The Production Reality

While academic research often focuses on maximizing FLOPs or model accuracy, the operational reality of a multi-tenant production environment is governed by different constraints. Most existing characterization studies rely on small-scale benchmarks like MLPerf, which may not capture the diversity or the "noisy" constraints of a massive AI cloud. This study addresses three critical questions: the actual resource demands of real-world workloads, the primary performance bottlenecks in current hardware/software stacks, and the potential speedups from architectural shifts or hardware upgrades.

Characterization Methodology

The researchers developed a lightweight analytical framework designed to minimize profiling overhead while providing accurate performance projections.

The characterization workflow involves profiling, feature extraction, and performance projection to identify bottlenecks and optimization hints.

The framework follows a three-step process:

  1. Profiling: Utilizing tf.RunMetadata from TensorFlow to trace kernel execution times, memory access patterns, and tensor attributes.
  2. Analytical Modeling: Execution time is decomposed into three primary components: Input Data I/O ($T_d$), Computation ($T_c$), and Weight/Gradient Communication ($T_w$).
  3. Performance Projection: This model estimates how a job's performance would change under different architectures (e.g., moving from Parameter Server to NVLink-based AllReduce) or varied hardware specifications (e.g., increasing network bandwidth).

Key Insights from Production Traces

Analysis of workload traces from December 2018 to January 2019 revealed several counter-intuitive findings:

  • Resource Distribution: Although single-GPU jobs are the most frequent, distributed training jobs (utilizing the Parameter Server/Worker architecture) consume 81% of total computation nodes. Consequently, optimizing distributed training offers the highest return on investment.
  • The Communication Bottleneck: On average, weight and gradient communication accounts for 62% of total execution time. In many distributed scenarios, this figure exceeds 80%, identifying communication as the primary inhibitor of scaling.
  • Memory vs. Compute: Within the computation phase, memory-bound operations (such as element-wise additions) often consume more time than compute-bound operations (like Matrix Multiplications). This suggests that memory access optimization (e.g., via XLA) is frequently more impactful than increasing raw TFLOPS.
  • Bottleneck Shifting: Addressing one bottleneck often reveals another. For instance, resolving communication issues with high-speed NVLink can shift the bottleneck to PCIe data I/O, as the system struggles to feed data from the CPU to the accelerated GPU cluster.

Architecture and Performance Gains

The study evaluated the impact of transitioning between different distributed training architectures.

Comparison of (a) Parameter Server (PS)/Worker and (b) AllReduce architectures used in distributed deep learning training.

Key results include:

  • Architecture Shifts: Approximately 60% of PS/Worker workloads could achieve significant speedups by migrating to an AllReduce architecture leveraging NVLink.
  • Network Upgrades: Increasing Ethernet bandwidth from 25 Gbps to 100 Gbps resulted in an average 1.7x speedup across distributed workloads.
  • PEARL for Sparse Models: For models with massive embedding tables, such as Graph Convolutional Networks (GCNs), the researchers proposed the Partitioned Embedding And RepLicated (PEARL) hybrid architecture. PEARL reduced communication time from 95% to just 25% of total execution time.
  • Model Accuracy: The analytical model proved highly effective, with predicted execution times falling within 10% of actual measurements for most production models.

Conclusion and Future Directions

This large-scale characterization highlights that communication, rather than raw computation, is the dominant factor in modern DL training performance. While the study focused on TensorFlow-based training, the methodology provides a blueprint for analyzing other frameworks and inference workloads. Future work will explore the non-negligible CPU overheads associated with framework scheduling and kernel launches, particularly in models with many fine-grained operations.

Summary updated

Abstract

Modern deep learning models have been exploited in various domains, including computer vision (CV), natural language processing (NLP), search and recommendation. In practical AI clusters, workloads training these models are run using software frameworks such as TensorFlow, Caffe, PyTorch and CNTK. One critical issue for efficiently operating practical AI clouds, is to characterize the computing and data transfer demands of these workloads, and more importantly, the training performance given the underlying software framework and hardware configurations. In this paper, we characterize deep learning training workloads from Platform of Artificial Intelligence (PAI) in Alibaba. We establish an analytical framework to investigate detailed execution time breakdown of various workloads using different training architectures, to identify performance bottleneck. Results show that weight/gradient communication during training takes almost 62% of the total execution time among all our workloads on average. The computation part, involving both GPU computing and memory access, are not the biggest bottleneck based on collective behavior of the workloads. We further evaluate attainable performance of the workloads on various potential software/hardware mappings, and explore implications on software architecture selection and hardware configurations. We identify that 60% of PS/Worker workloads can be potentially sped up when ported to the AllReduce architecture exploiting the high-speed NVLink for GPU interconnect, and on average 1.7X speedup can be achieved when Ethernet bandwidth is upgraded from 25 Gbps to 100 Gbps.
BibTeX
@inproceedings{wang2019characterizing,
  author = {Mengdi Wang and Chen Meng and Guoping Long and Chuan Wu and Jun Yang and Wei Lin and Yangqing Jia},
  title = {{Characterizing Deep Learning Training Workloads on Alibaba-PAI}},
  booktitle = {IISWC},
  year = {2019},
  url = {https://openreview.net/forum?id=WF9om3ublu},
}
Figure