Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning

Donald GoldfarbarXiv 2023 · DOI · arXiv

TL;DR

This paper introduces a novel layer-wise adaptive step-size procedure for first-order optimization methods in deep learning. This approach eliminates the need for manual learning rate tuning by dynamically adjusting step sizes for each layer based on its local curvature, leading to improved performance and comparable computational costs to standard methods.

What Problem Does This Paper Address?

Training deep neural networks (DNNs) often involves minimizing complex, non-convex loss functions. A critical aspect of this optimization process is selecting an appropriate learning rate (LR) for gradient-based methods. Traditional methods either require extensive manual tuning of a global LR, which is time-consuming and suboptimal, or employ adaptive methods that still need a base LR to be set. Existing adaptive methods like Adam or RMSProp adjust step sizes based on gradient statistics but don't fully leverage the layer-specific curvature information inherent in DNNs. This paper aims to address the challenge of automatic, effective learning rate scheduling by proposing a method that adapts step sizes at a per-layer level, exploiting the Hessian's diagonal blocks.

Methods

The core contribution is a layer-wise adaptive step-size procedure designed to work with existing first-order optimization algorithms (like SGD with momentum and AdamW). The method computes an adaptive step size for each layer ($\lambda$) of the neural network, denoted as $t^{(\lambda)}_k$, at each iteration $k$. This step size is derived by leveraging the layer-wise stochastic curvature information contained in the diagonal blocks of the Hessian.

Specifically, for each layer $\lambda$, the algorithm computes:

  • Layer-wise gradient ($g^{(\lambda)}_k$): The gradient of the loss with respect to the parameters of layer $\lambda$.
  • Layer-wise Hessian-vector product ($H^{(\lambda)}_k d^{(\lambda)}_k$): This captures the local curvature information for layer $\lambda$ along the descent direction $d^{(\lambda)}_k$. The paper notes that efficient computation of Hessian-vector products is possible using automatic differentiation techniques.
  • Adaptive step size ($t^{(\lambda)}_k$): This is computed using a closed-form expression that incorporates the gradient, the Hessian-vector product, and a term related to the local curvature. The paper draws inspiration from self-concordant function theory and uses a formula that guarantees a decrease in the objective function.

The overall algorithm then updates the weights of each layer using its respective adaptive step size: $W^{(\lambda)}_{k+1} = W^{(\lambda)}_k - t^{(\lambda)}_k d^{(\lambda)}_k$. This approach is integrated into standard optimizers, creating variants like "SGD-m-LW" and "AdamW-LW". The method aims for computational efficiency, with per-iteration complexity only slightly higher than a standard gradient computation, and memory requirements comparable to first-order methods.

Datasets

The paper evaluates its proposed methods on a variety of standard deep learning benchmarks:

  • CNNs: CIFAR-10, CIFAR-100, and SVHN datasets were used with VGG16, ResNet-18, and DenseNet models.
  • GCNs: Cora, CiteSeer, and PubMed citation network datasets were used for node classification.
  • Autoencoders: MNIST and FACES datasets were used for autoencoder problems.

Key Insights and Ideas

  • Layer-wise curvature matters: The paper demonstrates that exploiting the local curvature of the loss function per layer is crucial for effective optimization, going beyond global adaptive methods.
  • Eliminating LR tuning: The primary goal of removing the need for manual learning rate tuning is achieved, simplifying the optimization process for practitioners.
  • Computational efficiency: The proposed method achieves adaptive step sizes with only a modest increase in computational cost and memory footprint compared to standard first-order methods.
  • Theoretical guarantees: The paper provides theoretical convergence guarantees for an idealized version of SGD with layer-wise step sizes, showing linear convergence in the full-batch setting.
  • Stationary learning rates per layer: Experimental results suggest that the computed layer-wise learning rates tend to stabilize, implying that fixed, layer-specific learning rates could be determined and used.

Key Results

The proposed layer-wise adaptive methods (SGD-m-LW and AdamW-LW) consistently outperform their standard counterparts (SGD-m and AdamW) that use fixed or scheduled learning rates across various tasks (CNNs, GCNs, Autoencoders).

  • Performance: In experiments on CNNs (CIFAR-10, VGG16), SGD-m-LW and AdamW-LW achieved comparable or better optimization and generalization performance than fine-tuned standard methods. For GCNs, they were competitive with SOTA methods. For autoencoders, they showed slightly better optimization performance.
  • Efficiency: While slightly slower than their non-layer-wise counterparts (around 1.8x in CNN experiments), they are competitive with other advanced methods like KFAC and Shampoo.
  • Robustness: The methods are shown to be relatively insensitive to the choice of the learning rate update frequency ($T$) and initial warm-up learning rates, indicating good practical robustness.
  • Directional Impact: Cosine similarity experiments show that the layer-wise scaling of directions significantly alters the update directions compared to vanilla methods, contributing to improved performance.

Limitations and Future Work

  • PyTorch Autograd Limitations: The authors note that standard PyTorch's autograd is not optimized for efficiently computing layer-wise local norms and other layer-specific functions. Enhancements in this area could lead to significant computational and memory savings.
  • Non-convexity: While theoretical guarantees are provided for an idealized SGD, extending these to general non-convex DNNs and other stochastic optimizers remains an area for future research.
  • Further Theoretical Exploration: Adapting existing theoretical frameworks (e.g., from [13]) to the proposed method and exploring the multi-affine nature of loss functions with ReLU activations are suggested.
  • Broader Architectures and Problems: Extending the LW LR approach to other architectures like RNNs and Transformers, and to different problem classes, is a promising direction.
  • Hessian Approximation: Investigating Hutchinson's method for approximating the diagonal of the Hessian to compute local norms is proposed as an avenue for future work.

Summary updated

Abstract

We propose a new per-layer adaptive step-size procedure for stochastic first-order optimization methods for minimizing empirical loss functions in deep learning, eliminating the need for the user to tune the learning rate (LR). The proposed approach exploits the layer-wise stochastic curvature information contained in the diagonal blocks of the Hessian in deep neural networks (DNNs) to compute adaptive step-sizes (i.e., LRs) for each layer. The method has memory requirements that are comparable to those of first-order methods, while its per-iteration time complexity is only increased by an amount that is roughly equivalent to an additional gradient computation. Numerical experiments show that SGD with momentum and AdamW combined with the proposed per-layer step-sizes are able to choose effective LR schedules and outperform fine-tuned LR versions of these methods as well as popular first-order and second-order algorithms for training DNNs on Autoencoder, Convolutional Neural Network (CNN) and Graph Convolutional Network (GCN) models. Finally, it is proved that an idealized version of SGD with the layer-wise step sizes converges linearly when using full-batch gradients.
BibTeX
@misc{goldfarb2023layer,
  author = {Donald Goldfarb},
  title = {{Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning}},
  year = {2023},
  eprint = {2305.13664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.13664},
}
Figure