Abstract
Gradient Descent (GD) remains the foundational algorithm for unconstrained optimization in machine learning and scientific computing. However, its performance is often hampered by extreme sensitivity to step-size selection and a lack of element-wise adaptivity. This paper introduces the Element-wise Relaxed Scalar Auxiliary Variable (E-RSAV) algorithm, a novel framework that ensures unconditional energy dissipation—guaranteeing stability regardless of step size—while providing parameter-specific learning rates. By integrating a relaxation parameter and an adaptive step-size mechanism based on Steffensen’s method (AE-RSAV), the proposed approach achieves superior convergence rates and robustness across convex, non-convex, and physics-informed neural network (PINN) benchmarks.
Introduction
In the pursuit of minimizing objective functions, standard Gradient Descent faces two primary challenges. First, the "Goldilocks" problem of step sizes: a learning rate that is too high leads to numerical instability and divergence, while one that is too low results in prohibitively slow convergence. Second, GD typically applies a uniform update to all parameters, failing to account for the varying sensitivities of different dimensions within the parameter space.
While adaptive optimizers like Adam address element-wise scaling, they often lack rigorous mathematical guarantees for stability. This work bridges the gap between physics-inspired numerical stability and modern optimization by adapting the Scalar Auxiliary Variable (SAV) method—originally developed for gradient flows in physics—into a general-purpose optimization tool: E-RSAV.
Methodology
The E-RSAV Framework
The core innovation of E-RSAV lies in its treatment of the objective function's "energy." Traditional SAV methods utilize a single scalar to track the global energy of a system. E-RSAV decomposes this by assigning an auxiliary variable to each element of the parameter vector. This allows the algorithm to function as an adaptive optimizer where each parameter effectively possesses its own learning rate.
To ensure the auxiliary variables remain faithful to the original objective function, the authors implement a "relaxation" step. This step uses a parameter ($\eta_0$) to couple the auxiliary variables to the actual loss, ensuring that the modified energy of the system always decreases—a property known as unconditional energy dissipation.
Adaptive Element-wise RSAV (AE-RSAV)
To further accelerate convergence, the authors propose AE-RSAV, which incorporates an adaptive step size derived from Steffensen’s method. This variant monitors an indicator ($\alpha$) that measures the alignment between the modified energy and the original loss function. If the two diverge, the step size is automatically adjusted, allowing the algorithm to navigate complex landscapes with minimal manual tuning.
Experimental Results
Stability in Convex Optimization
The algorithm was tested against high-condition quadratic functions where the Hessian matrix makes standard GD prone to oscillation. While GD diverged (returning NaN) at step sizes of $\Delta t \geq 10$, E-RSAV demonstrated remarkable robustness, maintaining stability and achieving loss values as low as $2.23 \times 10^{-308}$. This confirms the theoretical guarantee of unconditional energy dissipation.
Navigating Non-convex Landscapes
On the classic Rosenbrock "banana" function—a challenging non-convex benchmark characterized by a narrow, curved valley—AE-RSAV significantly outperformed both standard GD and the basic E-SAV. The adaptive mechanism allowed the optimizer to accelerate through flat regions and decelerate to navigate sharp turns, reaching the global minimum with higher precision and fewer iterations.
Physics-Informed Neural Networks (PINNs)
The practical utility of E-RSAV was demonstrated by training a PINN to solve Burger’s Equation, a non-linear partial differential equation from fluid dynamics. The training process for PINNs is notoriously difficult due to the complex loss landscapes involving both data and physical constraints.
- AE-RSAV Final Loss: 0.001332
- Standard GD Final Loss: 0.008827
Beyond lower loss, AE-RSAV more accurately captured the sharp gradients (shocks) inherent in the fluid dynamics solution, where standard methods often smoothed out these critical features.
Discussion and Conclusion
E-RSAV represents a significant step forward in creating "fail-safe" optimization algorithms. By guaranteeing that energy always dissipates, it removes the risk of numerical explosions, making it an ideal candidate for complex, high-stakes simulations.
Theoretical analysis confirms that in univariate cases, the algorithm achieves super-linear convergence. While the computational cost per iteration is higher than standard GD due to the inclusion of Steffensen’s method, the reduction in total iterations and the removal of the need for extensive hyperparameter tuning offer a compelling trade-off. Future work will focus on extending the multivariate super-linear convergence proofs and scaling the algorithm to large-scale deep learning architectures like Transformers.
Summary updated