Abstract
Nonlinear conjugate gradient (NLCG) based optimizers have shown superior loss convergence properties compared to gradient descent based optimizers for traditional optimization problems. However, in Deep Neural Network (DNN) training, the dominant optimization algorithm of choice is still Stochastic Gradient Descent (SGD) and its variants. In this work, we propose and evaluate the stochastic preconditioned nonlinear conjugate gradient algorithm for large scale DNN training tasks. We show that a nonlinear conjugate gradient algorithm improves the convergence speed of DNN training, especially in the large mini-batch scenario, which is essential for scaling synchronous distributed DNN training to large number of workers. We show how to efficiently use second order information in the NLCG pre-conditioner for improving DNN training convergence. For the ImageNet classification task, at extremely large mini-batch sizes of greater than 65k, NLCG optimizer is able to improve top-1 accuracy by more than 10 percentage points for standard training of the Resnet-50 model for 90 epochs. For the CIFAR-100 classification task, at extremely large mini-batch sizes of greater than 16k, NLCG optimizer is able to improve top-1 accuracy by more than 15 percentage points for standard training of the Resnet-32 model for 200 epochs.
Introduction
To accelerate Deep Neural Network (DNN) training, researchers increasingly rely on synchronous data parallelism across massive GPU clusters. This scaling strategy necessitates increasing the total mini-batch size to maintain high hardware utilization. However, standard first-order optimizers like SGD, Momentum, and RMSProp frequently encounter a "generalization gap" or convergence failure when mini-batch sizes exceed 32k. This paper explores Nonlinear Conjugate Gradients (NLCG) as a robust alternative for these extreme scales.
Methodology: Stochastic Preconditioned NLCG
The proposed approach adapts traditional NLCG for the noisy environment of stochastic mini-batch training through two primary innovations:
- Quasi-Newton Pre-conditioner: To handle millions of parameters without the prohibitive cost of a full Hessian, the authors employ a diagonal BFGS update. This approximates second-order curvature using gradient and weight history, effectively scaling individual variables to navigate ill-conditioned loss landscapes.
- Online Stochastic Line Search: A lightweight line search mechanism monitors loss values to dynamically adjust step sizes, ensuring robustness against the noise inherent in stochastic gradients.
Experimental Results
The authors evaluated the NLCG optimizer on ImageNet (ResNet-50) and CIFAR-100 (ResNet-32) benchmarks, pushing batch sizes to unprecedented levels.
Scaling and the Generalization Wall
A critical finding is NLCG's ability to maintain high accuracy where standard optimizers fail. As batch sizes increase, gradient variance decreases, allowing NLCG's "conjugate" directions—which avoid undoing previous optimization progress—to become highly effective.
Performance at Extreme Scale
At a batch size of 65,536, NLCG variants demonstrate superior convergence trajectories compared to baseline optimizers.
On ImageNet with batch sizes exceeding 65k, NLCG improved top-1 accuracy by over 10 percentage points. Similar gains were observed for CIFAR-100 at batch sizes over 16k, with improvements exceeding 15 percentage points. While NLCG steps are computationally more intensive (15-30% slower), the overall wall-clock efficiency remains competitive due to superior convergence at scale.
Conclusion
Nonlinear Conjugate Gradients offer a powerful solution for the scaling limits of synchronous distributed training. By leveraging second-order information and maintaining stability at massive batch sizes, NLCG enables efficient training on thousands of workers where traditional first-order methods falter. Future work remains to stabilize NLCG for small-batch scenarios and further optimize per-step computational overhead.
Summary updated