Reliability of Large Scale GPU Clusters for Deep Learning Workloads

Junjie Qian, Taeyoon Kim, Myeongjae JeonWWW 2021 · OpenReview

Abstract

Recent advances on deep learning technologies have made GPU clusters popular as training platforms. In this paper, we study reliability issues while focusing on training job failures from analyzing logs collected from running deep learning workloads on a large-scale GPU cluster in production. These failures are largely grouped into two categories, infrastructure and user, based on their sources, and reveal diverse reasons causing the failures. With insights obtained from the failure analysis, we suggest several different ways to improve the stability of shared GPU clusters designed for DL training and optimize user experience by reducing failure occurrences.
BibTeX
@inproceedings{qian2021reliability,
  author = {Junjie Qian and Taeyoon Kim and Myeongjae Jeon},
  title = {{Reliability of Large Scale GPU Clusters for Deep Learning Workloads}},
  booktitle = {The Web Conference},
  year = {2021},
  url = {https://openreview.net/forum?id=fyP9beb82v},
}

Loading related content...

Figure