Abstract
Training deep neural networks typically relies on Stochastic Gradient Descent (SGD) or its variants, which are notoriously sensitive to hyperparameters like the learning rate. We introduce Stochastic Model Building (SMB), a novel optimization algorithm that enhances SGD by utilizing local quadratic models to adaptively adjust both step size and search direction. SMB achieves faster convergence and superior generalization across diverse deep learning tasks while demonstrating significant robustness to initial learning rate choices. By leveraging local curvature without the computational overhead of traditional second-order methods, SMB offers a more efficient path to training high-performance models.
Introduction
The performance of deep learning models is heavily dependent on the optimization strategy employed. While SGD is the standard, finding the optimal learning rate and decay schedule often requires exhaustive and energy-intensive grid searches. Adaptive methods like Adam and line-search strategies mitigate some of these issues but often fail to adjust the search direction or introduce significant memory overhead. This paper addresses these limitations by proposing a method that uses local curvature to improve the optimization trajectory without the complexity of classic second-order techniques.
Stochastic Model Building (SMB)
The core contribution is the SMB algorithm, which integrates local quadratic modeling into the stochastic optimization process. Unlike standard gradient descent, SMB performs a "trial" step. If this step fails to satisfy a sufficient decrease condition (the Armijo condition), the algorithm constructs a local quadratic model using gradient information from both the current and trial points.
Key features of SMB include:
- Local Quadratic Modeling: Estimates local curvature to refine the optimization path.
- Group-wise Adaptation: Parameters are organized into groups (e.g., layers), allowing for independent adaptive step lengths across the network.
- Trust Region Constraint: Ensures the updated step remains within a region where the local model is accurate.
- SMBi Variant: A statistically rigorous version that uses independent data batches for model building to ensure convergence.
Experimental Evaluation
The authors evaluated SMB against popular optimizers, including Adam and SLS (Stochastic Line Search), on datasets such as MNIST, CIFAR-10, and CIFAR-100 using architectures like ResNet-34 and DenseNet-121.
Performance and Convergence
On the MNIST benchmark using a Multi-Layer Perceptron (MLP), SMB and its variant SMBi demonstrated superior performance. As shown in the figures below, SMB achieves a faster decay in training loss and reaches higher test accuracy compared to baseline optimizers when measured against wall-clock time.
Robustness and Generalization
A critical advantage of SMB is its robustness to hyperparameter tuning. Experiments on CIFAR-100 with DenseNet-121 showed that SMB consistently outperformed other optimizers in training error and reached the highest test accuracy. Notably, the authors report that SMB's performance remains stable across a wide range of initial learning rates, whereas standard SGD's performance is highly volatile. This suggests that SMB finds more robust, "flatter" minima that generalize better to unseen data.
Discussion and Limitations
While SMB offers significant improvements, there are areas for future development. Currently, the algorithm lacks an internal mechanism for automatic learning rate decay over long training cycles. Additionally, while SMBi has a rigorous convergence analysis, a full theoretical proof for the more efficient standard SMB algorithm is still needed. Finally, although the computational cost is offset by faster convergence, the additional gradient evaluations required during model building present an opportunity for further optimization.
Conclusion
Stochastic Model Building represents a significant step forward in making deep learning optimization more robust and efficient. By adaptively adjusting both the step size and direction through local quadratic models, SMB reduces the need for extensive hyperparameter tuning while achieving state-of-the-art performance across various tasks.
Summary updated