A Novel Structured Natural Gradient Descent for Deep Learning

Weihua Liu, Xiabi LiuarXiv 2021 · arXiv · OpenReview

TL;DR

This paper introduces Structured Natural Gradient Descent (SNGD), a novel optimization method for deep neural networks. SNGD effectively approximates the benefits of Natural Gradient Descent (NGD) by reconstructing the network with a "local Fisher layer," allowing for faster convergence and improved performance compared to traditional Gradient Descent (GD) while maintaining computational simplicity.

What Problem Does This Paper Address?

Training deep neural networks often relies on Gradient Descent (GD), which, while simple, can struggle with the "pathological curvature" of the loss landscape, leading to slow convergence. Natural Gradient Descent (NGD) offers a more efficient path by considering the curvature of the loss function, but its practical application to deep networks is hampered by the computationally prohibitive cost of calculating and inverting the Fisher Information Matrix (FIM). This paper aims to bridge this gap by developing an optimization method that captures the advantages of NGD without its computational burden.

Methods

The core innovation is the Structured Natural Gradient Descent (SNGD) method. SNGD achieves its goal by:

  • Network Reconstruction: The deep neural network is reconfigured by introducing a new "local Fisher layer." This layer is designed to capture the curvature information of the loss function at a local level, effectively decomposing the global FIM computation.
  • Equivalence to NGD: The reconstructed network, when optimized with traditional Gradient Descent (GD), achieves an effect equivalent to optimizing the original network with NGD. This is mathematically derived through a transformation of parameters ($w' = w \cdot G^{-1/2}$), where $G$ is the Fisher matrix.
  • Local Fisher Layer: This new layer calculates a local Fisher matrix for each subsystem (layer) of the network. This local computation significantly reduces the overall computational complexity compared to calculating a global FIM.
  • Efficient Matrix Square Root Calculation: To further reduce computational cost, the paper proposes two strategies:
  • Efficient Calculation of $xx^T$: Optimizing the calculation of the Gram matrix, which is a component of the local Fisher matrix.
  • Newton's Method for Matrix Square Root: Employing the Denman-Beavers iteration method and a further optimized version to efficiently compute the matrix square root ($G^{-1/2}$) required for the parameter transformation.

Datasets

  • MNIST: Used for handwritten digit classification.
  • LUNA16: Used for lung nodule classification and detection tasks.

Key Insights and Ideas

  • Decomposition is Key: The paper demonstrates that decomposing the complex global Fisher Information Matrix into local Fisher matrices for each layer is a viable strategy to make NGD computationally tractable for deep networks.
  • Reconstruction for Optimization: By restructuring the network with a "local Fisher layer," the authors cleverly enable the use of standard GD to achieve the benefits of NGD. This is a powerful idea that leverages existing optimization tools.
  • Curvature as Regularization: The "local Fisher layer" not only adjusts the gradient direction for faster convergence but also acts as a form of regularization, contributing to improved model stability and performance.
  • Universality of SNGD: The experimental results across classification and detection tasks suggest that SNGD is a broadly applicable optimization method for various deep learning architectures.

Key Results

  • Faster Convergence: SNGD significantly accelerates the convergence speed of deep network models compared to traditional SGD, particularly noticeable in the early stages of training (e.g., before 40 epochs on MNIST).
  • Improved Accuracy: SNGD consistently leads to better model accuracy. On MNIST, final classification accuracy increased from 97.62% to 97.79%. In lung nodule classification, SNGD outperformed SGD in the Competitive Performance Index (CPM).
  • Effective for Detection: In lung nodule detection, SNGD showed superior performance over SGD and SGD+BN, achieving higher sensitivity and a better overall CPM value (0.927 vs. 0.910 for SGD+BN).
  • Batch Size Independence: The effectiveness of SNGD is highlighted by its performance even when batch sizes are small, where methods like Batch Normalization (BN) show limited improvement.

Limitations and Future Work

  • Computational Cost of Local Fisher Layer: While SNGD significantly reduces the computational burden compared to global NGD, the calculation of the local Fisher matrix and its square root still adds overhead compared to standard GD. The paper addresses this with optimization strategies, but it remains a consideration.
  • Generalization to All Architectures: While experiments show broad applicability, further validation on a wider range of complex deep learning architectures would be beneficial.
  • Future Work: The paper implicitly suggests future work in exploring more efficient methods for calculating local Fisher matrices and their square roots, as well as applying SNGD to even more diverse and challenging deep learning tasks.

Summary updated

Abstract

Natural gradient descent (NGD) provided deep insights and powerful tools to deep neural networks. However the computation of Fisher information matrix becomes more and more difficult as the network structure turns large and complex. This paper proposes a new optimization method whose main idea is to accurately replace the natural gradient optimization by reconstructing the network. More specifically, we reconstruct the structure of the deep neural network, and optimize the new network using traditional gradient descent (GD). The reconstructed network achieves the effect of the optimization way with natural gradient descent. Experimental results show that our optimization method can accelerate the convergence of deep network models and achieve better performance than GD while sharing its computational simplicity.
BibTeX
@misc{liu2021novel,
  author = {Weihua Liu and Xiabi Liu},
  title = {{A Novel Structured Natural Gradient Descent for Deep Learning}},
  year = {2021},
  eprint = {2109.10100},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.10100},
}
Figure