An Adaptive Memory Multi-Batch L-BFGS Algorithm for Neural Network Training

Federico Zocco, Seán F. McLoonearXiv 2020 · DOI · OpenReview

TL;DR

This paper introduces an adaptive memory L-BFGS algorithm (MB-AM) for neural network training that intelligently adjusts its use of second-order curvature information. By gradually increasing its "trust" in this information as training progresses, MB-AM aims to improve convergence speed and solution quality compared to standard L-BFGS methods, especially for large-scale problems.

What Problem Does This Paper Address?

Training large neural networks is computationally intensive. While first-order methods like SGD and Adam are popular due to their scalability and regularization techniques, second-order methods like L-BFGS offer potentially faster convergence by using curvature information (approximations of the Hessian matrix). However, L-BFGS can be unreliable early in training when the cost function's shape is far from quadratic. This paper tackles the challenge of making L-BFGS more robust and efficient for neural network training by adaptively controlling its reliance on second-order information throughout the training process.

Methods

The core contribution is the Adaptive Memory Multi-Batch L-BFGS (MB-AM) algorithm. It builds upon the existing multi-batch L-BFGS framework by incorporating two key innovations:

  • Progressive Curvature Trust (dev-increase scheme): Instead of using a fixed memory size for storing curvature pairs (which is a limitation of standard L-BFGS), MB-AM gradually increases its memory size as training progresses. This "dev-increase" scheme is triggered by monitoring the validation loss. Early in training, when the cost function is less predictable, MB-AM uses less curvature information, behaving more like a first-order method. As the validation loss stabilizes (indicating proximity to a minimum), the memory size increases, allowing MB-AM to leverage more second-order information for faster convergence.
  • Periodic Memory Resetting: To further mitigate the impact of potentially unreliable curvature information, the algorithm also incorporates a periodic resetting mechanism for the memory. This is applied when the memory size is below a certain threshold, based on the validation loss development.

The paper also compares MB-AM with two variants: MB-AMR (which includes both adaptive memory and resetting) and MB-R (a constant memory L-BFGS with periodic resetting).

Datasets

The authors evaluated their proposed algorithm on six benchmark datasets:

  • CANCER
  • ETCH
  • WAFERS
  • MNIST-R (a rescaled version of MNIST)
  • CIFAR10
  • TRASH-RA (a rescaled and augmented version of TRASH)

These datasets vary in size and complexity, and are used to train both Multi-Layer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs).

Key Insights and Ideas

  • The "sweet spot" for second-order information: The paper highlights that second-order information is most beneficial when the cost function is locally quadratic, which typically occurs later in training. MB-AM's adaptive approach intelligently capitalizes on this by increasing its use of curvature information only when it's most likely to be beneficial.
  • Balancing convergence and reliability: By gradually increasing memory, MB-AM strikes a balance between the rapid convergence potential of L-BFGS and the reliability needed in the early, non-quadratic stages of training. This is a more nuanced approach than simply using a fixed memory size.
  • The detrimental effect of aggressive resetting: The experiments suggest that while resetting can be useful, aggressive or constant resetting (as in MB-R) can hinder performance, indicating that a more controlled approach is preferred.

Key Results

  • Improved Solution Quality: MB-AM and its variant MB-AMR consistently achieved better or comparable mean Correct Classification Rates (CCR) compared to standard multi-batch L-BFGS (MB) and Adam across several experiments. MB-AM, in particular, showed strong performance in experiments involving CNNs.
  • More Consistent Performance: In terms of rank metric (RNK), which assesses relative performance, MB-AM and MB-AMR demonstrated more consistent performance across experiments than MB and Adam.
  • Slight Computational Advantage: For larger CNN-based problems, MB-AM and MB-AMR were marginally faster than MB, likely due to the lower memory usage in early iterations. However, Adam remained significantly faster overall, especially for larger datasets.

Limitations and Future Work

  • Hyperparameter Tuning: The paper acknowledges that MB-AM introduces three new hyperparameters (α, mo/mmax, and mval) that require empirical tuning. Future work will focus on automating the selection of these parameters.
  • Integration with Progressive Batching: The authors suggest exploring the integration of MB-AM with progressive batching techniques for further improvements.
  • Adam's Dominance in Speed: While MB-AM offers improved accuracy and consistency, Adam remains the preferred choice for speed, especially for very large neural networks. The paper concludes that MB-AM should be considered when Adam's accuracy is not satisfactory.

Summary updated

Abstract

Motivated by the potential for parallel implementation of batch-based algorithms and the accelerated convergence achievable with approximated second order information a limited memory version of the BFGS algorithm has been receiving increasing attention in recent years for large neural network training problems. As the shape of the cost function is generally not quadratic and only becomes approximately quadratic in the vicinity of a minimum, the use of second order information by L-BFGS can be unreliable during the initial phase of training, i.e. when far from a minimum. Therefore, to control the influence of second order information as training progresses, we propose a multi-batch L-BFGS algorithm, namely MB-AM, that gradually increases its trust in the curvature information by implementing a progressive storage and use of curvature data through a development-based increase (dev-increase) scheme. Using six discriminative modelling benchmark problems we show empirically that MB-AM has slightly faster convergence and, on average, achieves better solutions than the standard multi-batch L-BFGS algorithm when training MLP and CNN models.
BibTeX
@misc{zocco2020adaptive,
  author = {Federico Zocco and Seán F. McLoone},
  title = {{An Adaptive Memory Multi-Batch L-BFGS Algorithm for Neural Network Training}},
  year = {2020},
  eprint = {2012.07434},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.07434},
}
Figure