NAMSG: An Efficient Method For Training Neural Networks

Yushu Chen, Hao Jing, Wenlai Zhao, Zhiqiang Liu, Liang Qiao, Wei Xue, Haohuan Fu, Guangwen YangarXiv 2019 · OpenReview

Abstract

We present the remote stochastic gradient (RSG) method, which computes the gradients at configurable remote observation points, in order to improve the convergence rate and suppress gradient noise at the same time for different curvatures. RSG is further combined with adaptive methods to construct ARSG for acceleration. The method is efficient in computation and memory, and is straightforward to implement. We analyze the convergence properties by modeling the training process as a dynamic system, which provides a guideline to select the configurable observation factor without grid search. ARSG yields $O(1/\sqrt{T})$ convergence rate in non-convex settings, that can be further improved to $O(\log(T)/T)$ in strongly convex settings. Numerical experiments demonstrate that ARSG achieves both faster convergence and better generalization, compared with popular adaptive methods, such as ADAM, NADAM, AMSGRAD, and RANGER for the tested problems. In particular, for training ResNet-50 on ImageNet, ARSG outperforms ADAM in convergence speed and meanwhile it surpasses SGD in generalization.

Introduction: The Generalization Gap

In deep learning optimization, a persistent challenge is the "generalization gap." While adaptive methods like ADAM and RMSProp offer rapid convergence by scaling gradients element-wise, they frequently struggle to find the optimal solutions reached by Stochastic Gradient Descent (SGD), leading to inferior performance on unseen data. Conversely, momentum-based methods like Nesterov’s Accelerated Gradient (NAG) improve SGD's speed but remain highly sensitive to gradient noise, particularly in small-batch regimes. Furthermore, these advanced techniques often necessitate extensive hyper-parameter tuning, which is computationally prohibitive. This paper introduces the Remote Stochastic Gradient (RSG) method and its adaptive counterpart, ARSG, to bridge this gap.

Methodology: Remote Observation and Adaptation

The core innovation of this work is the Remote Stochastic Gradient (RSG) method. Unlike standard SGD, which evaluates gradients at the current parameter state, RSG calculates them at a "remote" observation point—a predicted future position derived from current momentum. This provides the optimizer with predictive information about the loss landscape.

RSG generalizes NAG by making the "look-ahead" distance (defined by the observation factor $\mu$) configurable. By combining RSG with an adaptive preconditioner (inspired by AMSGRAD), the authors developed ARSG (Adaptive RSG). This hybrid approach leverages both the predictive power of remote observation and the efficient element-wise scaling of adaptive methods.

Dynamic System Analysis

To move beyond "black box" tuning, the authors modeled the training process as a dynamic system. This theoretical framework allows for the selection of the observation factor $\mu$ based on the system's stability and noise suppression characteristics, rather than exhaustive grid searches.

Theoretical analysis of Gain Factor and Error Variance Limit across different observation factors (mu).

As shown in the theoretical analysis above, adjusting the remote observation point allows the algorithm to effectively suppress gradient noise in high-curvature directions while maintaining acceleration in low-curvature regions. The method also includes an Observation Boost (OBSB) policy, where $\mu$ is increased and the learning rate is scaled once the loss flattens, helping the model escape local noise and improve final generalization.

Empirical Results

The effectiveness of ARSG was evaluated across several standard benchmarks, including MNIST, WikiText-2, CIFAR-10, and ImageNet.

  • MNIST: ARSG and its boosted version (ARSGB) demonstrated a significant speed advantage, training logistic regression over 50% faster than ADAM.
  • ImageNet (ResNet-50): This benchmark highlighted the method's superior generalization. ARSG achieved a top-1 validation accuracy of 75.79%, outperforming SGD (75.59%) and significantly beating various ADAM variants, which ranged between 72.49% and 73.86%.
  • CIFAR-10: ARSG reached higher validation accuracies (approximately 91.96%) compared to ADAM (90.79%) and RANGER (91.02%).
Validation loss curves for various optimizers, demonstrating the superior convergence and final loss of ARSG and ARSGB.

As illustrated in the validation loss curves, ARSG and its boosted variant ARSGB consistently achieve lower loss values more quickly than traditional adaptive methods and standard SGD. This confirms that the method successfully combines the fast initial convergence of adaptive optimizers with the superior final performance of SGD.

Conclusion

ARSG represents a significant step forward in neural network optimization. By utilizing remote observation points and a robust theoretical framework based on dynamic systems, it achieves state-of-the-art performance on large-scale tasks like ImageNet while remaining computationally efficient. The "fast form" of the algorithm requires minimal additional overhead, making it a practical and powerful alternative to existing optimizers. Future work will explore its application to modern Transformer architectures and reinforcement learning environments.

Summary updated

Abstract

We present the remote stochastic gradient (RSG) method, which computes the gradients at configurable remote observation points, in order to improve the convergence rate and suppress gradient noise at the same time for different curvatures. RSG is further combined with adaptive methods to construct ARSG for acceleration. The method is efficient in computation and memory, and is straightforward to implement. We analyze the convergence properties by modeling the training process as a dynamic system, which provides a guideline to select the configurable observation factor without grid search. ARSG yields $O(1/\sqrt{T})$ convergence rate in non-convex settings, that can be further improved to $O(\log(T)/T)$ in strongly convex settings. Numerical experiments demonstrate that ARSG achieves both faster convergence and better generalization, compared with popular adaptive methods, such as ADAM, NADAM, AMSGRAD, and RANGER for the tested problems. In particular, for training ResNet-50 on ImageNet, ARSG outperforms ADAM in convergence speed and meanwhile it surpasses SGD in generalization.
BibTeX
@misc{chen2019namsg,
  author = {Yushu Chen and Hao Jing and Wenlai Zhao and Zhiqiang Liu and Liang Qiao and Wei Xue and Haohuan Fu and Guangwen Yang},
  title = {{NAMSG: An Efficient Method For Training Neural Networks}},
  year = {2019},
  eprint = {1905.01422},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.01422},
}
Figure