TL;DR
This paper introduces an adaptive memory L-BFGS algorithm (MB-AM) for neural network training that intelligently adjusts its use of second-order curvature information. By gradually increasing its "trust" in this information as training progresses, MB-AM aims to improve convergence speed and solution quality compared to standard L-BFGS methods, especially for large-scale problems.
What Problem Does This Paper Address?
Training large neural networks is computationally intensive. While first-order methods like SGD and Adam are popular due to their scalability and regularization techniques, second-order methods like L-BFGS offer potentially faster convergence by using curvature information (approximations of the Hessian matrix). However, L-BFGS can be unreliable early in training when the cost function's shape is far from quadratic. This paper tackles the challenge of making L-BFGS more robust and efficient for neural network training by adaptively controlling its reliance on second-order information throughout the training process.
Methods
The core contribution is the Adaptive Memory Multi-Batch L-BFGS (MB-AM) algorithm. It builds upon the existing multi-batch L-BFGS framework by incorporating two key innovations:
- Progressive Curvature Trust (dev-increase scheme): Instead of using a fixed memory size for storing curvature pairs (which is a limitation of standard L-BFGS), MB-AM gradually increases its memory size as training progresses. This "dev-increase" scheme is triggered by monitoring the validation loss. Early in training, when the cost function is less predictable, MB-AM uses less curvature information, behaving more like a first-order method. As the validation loss stabilizes (indicating proximity to a minimum), the memory size increases, allowing MB-AM to leverage more second-order information for faster convergence.
- Periodic Memory Resetting: To further mitigate the impact of potentially unreliable curvature information, the algorithm also incorporates a periodic resetting mechanism for the memory. This is applied when the memory size is below a certain threshold, based on the validation loss development.
The paper also compares MB-AM with two variants: MB-AMR (which includes both adaptive memory and resetting) and MB-R (a constant memory L-BFGS with periodic resetting).
Datasets
The authors evaluated their proposed algorithm on six benchmark datasets:
- CANCER
- ETCH
- WAFERS
- MNIST-R (a rescaled version of MNIST)
- CIFAR10
- TRASH-RA (a rescaled and augmented version of TRASH)
These datasets vary in size and complexity, and are used to train both Multi-Layer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs).
Key Insights and Ideas
- The "sweet spot" for second-order information: The paper highlights that second-order information is most beneficial when the cost function is locally quadratic, which typically occurs later in training. MB-AM's adaptive approach intelligently capitalizes on this by increasing its use of curvature information only when it's most likely to be beneficial.
- Balancing convergence and reliability: By gradually increasing memory, MB-AM strikes a balance between the rapid convergence potential of L-BFGS and the reliability needed in the early, non-quadratic stages of training. This is a more nuanced approach than simply using a fixed memory size.
- The detrimental effect of aggressive resetting: The experiments suggest that while resetting can be useful, aggressive or constant resetting (as in MB-R) can hinder performance, indicating that a more controlled approach is preferred.
Key Results
- Improved Solution Quality: MB-AM and its variant MB-AMR consistently achieved better or comparable mean Correct Classification Rates (CCR) compared to standard multi-batch L-BFGS (MB) and Adam across several experiments. MB-AM, in particular, showed strong performance in experiments involving CNNs.
- More Consistent Performance: In terms of rank metric (RNK), which assesses relative performance, MB-AM and MB-AMR demonstrated more consistent performance across experiments than MB and Adam.
- Slight Computational Advantage: For larger CNN-based problems, MB-AM and MB-AMR were marginally faster than MB, likely due to the lower memory usage in early iterations. However, Adam remained significantly faster overall, especially for larger datasets.
Limitations and Future Work
- Hyperparameter Tuning: The paper acknowledges that MB-AM introduces three new hyperparameters (α, mo/mmax, and mval) that require empirical tuning. Future work will focus on automating the selection of these parameters.
- Integration with Progressive Batching: The authors suggest exploring the integration of MB-AM with progressive batching techniques for further improvements.
- Adam's Dominance in Speed: While MB-AM offers improved accuracy and consistency, Adam remains the preferred choice for speed, especially for very large neural networks. The paper concludes that MB-AM should be considered when Adam's accuracy is not satisfactory.
Summary updated