Deep Neural Network Training With Distributed K-FAC

J. Gregory Pauloski, Lei Huang, Weijia Xu, Kyle Chard, Ian T. Foster, Zhao Zhang

Abstract

Scaling deep neural network training to more processors and larger batch sizes is key to reducing end-to-end training time; yet, maintaining comparable convergence and hardware utilization at larger scales is challenging. Increases in training scales have enabled natural gradient optimization methods as a reasonable alternative to stochastic gradient descent and variants thereof. Kronecker-factored Approximate Curvature (K-FAC), a natural gradient method, preconditions gradients with an efficient approximation of the Fisher Information Matrix to improve per-iteration progress when optimizing an objective function. Here we propose a scalable K-FAC algorithm and investigate K-FAC’s applicability in large-scale deep neural network training. Specifically, we explore layer-wise distribution strategies, inverse-free second-order gradient evaluation, and dynamic K-FAC update decoupling, with the goal of preserving convergence while minimizing training time. We evaluate the convergence and scaling properties of our K-FAC gradient preconditioner, for image classification, object detection, and language modeling applications. In all applications, our implementation converges to baseline performance targets in 9–25% less time than the standard first-order optimizers on GPU clusters across a variety of scales.
BibTeX
@article{pauloski2022deep,
  author = {J. Gregory Pauloski and Lei Huang and Weijia Xu and Kyle Chard and Ian T. Foster and Zhao Zhang},
  title = {{Deep Neural Network Training With Distributed K-FAC}},
  journal = {IEEE Trans. Parallel Distributed Syst.},
  year = {2022},
  doi = {10.1109/tpds.2022.3161187},
}

Loading related content...

Figure