Deep Neural Network Training With Distributed K-FAC
Lists
Only you see this
Abstract
Scaling deep neural network training to more processors and larger batch sizes is key to reducing end-to-end training time; yet, maintaining comparable convergence and hardware utilization at larger scales is challenging. Increases in training scales have enabled natural gradient optimization methods as a reasonable alternative to stochastic gradient descent and variants thereof. Kronecker-factored Approximate Curvature (K-FAC), a natural gradient method, preconditions gradients with an efficient approximation of the Fisher Information Matrix to improve per-iteration progress when optimizing an objective function. Here we propose a scalable K-FAC algorithm and investigate K-FAC’s applicability in large-scale deep neural network training. Specifically, we explore layer-wise distribution strategies, inverse-free second-order gradient evaluation, and dynamic K-FAC update decoupling, with the goal of preserving convergence while minimizing training time. We evaluate the convergence and scaling properties of our K-FAC gradient preconditioner, for image classification, object detection, and language modeling applications. In all applications, our implementation converges to baseline performance targets in 9–25% less time than the standard first-order optimizers on GPU clusters across a variety of scales.
BibTeX
@article{pauloski2022deep,
author = {J. Gregory Pauloski and Lei Huang and Weijia Xu and Kyle Chard and Ian T. Foster and Zhao Zhang},
title = {{Deep Neural Network Training With Distributed K-FAC}},
journal = {IEEE Trans. Parallel Distributed Syst.},
year = {2022},
doi = {10.1109/tpds.2022.3161187},
}Loading related content...