Abstract
The optimization algorithm and its hyperparameters can significantly affect the training speed and resulting model accuracy in machine learning applications. The wish list for an ideal optimizer includes fast and smooth convergence to low error, low computational demand, and general applicability. Our recently introduced continual resilient (CoRe) optimizer has shown superior performance compared to other state-of-the-art first-order gradient-based optimizers for training lifelong machine learning potentials. In this work we provide an extensive performance comparison of the CoRe optimizer and nine other optimization algorithms including the Adam optimizer and resilient backpropagation (RPROP) for diverse machine learning tasks. We analyze the influence of different hyperparameters and provide generally applicable values. The CoRe optimizer yields best or competitive performance in every investigated application, while only one hyperparameter needs to be changed depending on mini-batch or batch learning.
Introduction: The Optimizer Dilemma
In modern machine learning, practitioners often find themselves caught between two worlds. On one hand, Adam has become the industry standard for stochastic mini-batch learning, prized for its adaptive learning rates. On the other hand, Resilient Backpropagation (RPROP) remains a powerhouse for deterministic batch learning due to its robustness—it relies solely on the sign of the gradient, making it immune to vanishing or exploding gradient magnitudes. However, RPROP typically fails in noisy, mini-batch environments, while Adam can struggle with stability in certain regimes.
The Continual Resilient (CoRe) optimizer, introduced by Marco Eckhoff and Markus Reiher, addresses this gap. It is designed as a "universal" optimizer that combines the strengths of both approaches while introducing a novel mechanism to handle lifelong learning and stability.
The CoRe Methodology
CoRe is a hybrid first-order gradient-based algorithm that synthesizes several key concepts:
- Adaptive Learning Rates: Like Adam, it utilizes exponential moving averages of the gradient and its square to scale updates, including bias correction for early training steps.
- Resilient Step Sizes: It adopts RPROP’s logic where step sizes increase when the gradient sign is consistent and decrease when it flips, allowing for rapid convergence and fine-tuning near minima.
- Stability-Plasticity Balance: A unique "importance score" is calculated for each weight. CoRe can "freeze" highly important weights (the plasticity factor) to prevent catastrophic forgetting in lifelong learning scenarios.
- Proportional Weight Decay: Unlike standard L2 regularization, CoRe’s weight decay is proportional to the actual update magnitude, ensuring weights don't collapse or explode too aggressively.
Experimental Evaluation
To test the "all-in-one" claim, the authors benchmarked CoRe against nine state-of-the-art optimizers across a wide variety of machine learning tasks, ranging from computer vision to reinforcement learning and time-series prediction.
The tasks included variational auto-encoding (AED/AEF), image classification (ICD/ICF), reinforcement learning (NR/RA), super-resolution (SR), graph learning (SS), and time-series prediction (TS).
Key Results and Insights
The benchmarking results demonstrate that CoRe is remarkably robust. It achieved the highest final accuracy in 6 out of the 9 PyTorch benchmark tasks and remained within the uncertainty interval of the top performer in the remaining three.
The "One Hyperparameter" Rule
One of the most significant findings is CoRe's ease of use. While many optimizers require extensive hyperparameter tuning for each new task, the authors identified a set of "generally applicable" values for CoRe. In practice, only the maximum step size ($s_{max}$) needs to be adjusted: $10^{-3}$ for mini-batch learning and $1$ for batch learning.
Performance in Challenging Regimes
- Reinforcement Learning: CoRe proved significantly more reliable than RPROP, which failed to learn entirely in these noisy environments.
- Lifelong Learning: In specialized molecular modeling tasks (lMLP), CoRe’s stability-plasticity mechanism allowed it to outperform both Adam and RPROP by effectively retaining knowledge across sequential learning phases.
- Efficiency: The computational overhead is negligible, with the optimizer step taking less than 0.2% of the time required for a single gradient calculation.
Conclusion
The CoRe optimizer represents a significant step toward a universal optimization solution. By bridging the gap between stochastic and deterministic learning regimes and providing a robust mechanism for weight stability, it offers a high-performance alternative to standard optimizers like Adam. While the "stability-plasticity" feature requires some care in standard classification tasks, its benefits for lifelong learning and general robustness make it a powerful tool for diverse machine learning applications.
Reference: Eckhoff, M., & Reiher, M. (2024). CoRe Optimizer: An All-in-One Solution for Machine Learning. arXiv:2307.15663v2 [cs.LG].
Summary updated