AEGD: Adaptive Gradient Decent with Energy

Hailiang Liu, Xuping TianarXiv 2020 · arXiv

Abstract

In the landscape of deep learning optimization, practitioners often face a trade-off between the simplicity of Gradient Descent (GD) and the robustness of adaptive methods. Standard GD is highly sensitive to step size, often leading to numerical instability or divergence if the learning rate is not meticulously tuned. Conversely, while adaptive methods like Adam offer stability, they can sometimes compromise generalization. This paper introduces AEGD (Adaptive Gradient Descent with Energy), a novel first-order optimization algorithm inspired by physics. By utilizing an auxiliary energy variable, AEGD achieves unconditional energy stability regardless of the step size, bridging the gap between simple explicit methods and stable implicit schemes.

Introduction

The efficiency of training deep neural networks is fundamentally limited by the "step size limitation." In standard first-order optimization, the step size must be constrained by the curvature of the objective function to prevent divergence. While implicit methods—such as the Proximal Point Algorithm—offer significantly better stability, their computational overhead is often prohibitive for large-scale machine learning because they require solving complex sub-problems at every iteration.

AEGD addresses this challenge by introducing a framework that is as computationally efficient as standard GD but as stable as implicit methods. By borrowing techniques from the study of partial differential equations (PDEs), specifically the Invariant Energy Quadratization (IEQ) strategy, AEGD transforms the optimization landscape into a system that naturally dissipates energy, ensuring stable convergence even under aggressive hyperparameter settings.

Methodology: The Energy Trick

The core of AEGD lies in the transformation of the objective function $f(\theta)$ into an augmented system. The process follows several key steps:

  1. The Energy Variable: The authors define an auxiliary variable, $r = \sqrt{f(\theta) + c}$, where $c$ is a constant chosen to ensure the term remains positive. This variable represents the "energy" of the system.
  2. Augmented System Dynamics: The original gradient flow is reformulated into a system involving both the parameters $\theta$ and the energy variable $r$. This structure is mathematically designed to ensure that the total energy of the system non-increases over time.
  3. Linear Update Rule: Unlike implicit methods that require iterative solvers, AEGD derives a discrete update rule that remains linear. This allows the algorithm to adjust the "effective" step size dynamically based on the current energy state, maintaining stability without heavy computation.
  4. Algorithmic Variants: The framework is flexible, leading to several versions:
  5. Global AEGD: Uses a single energy variable for all parameters.
  6. Element-wise AEGD: Assigns a separate energy variable to each coordinate, similar to the per-parameter scaling found in Adam.
  7. AEGDW: Incorporates decoupled weight decay to optimize performance in deep learning contexts.

Experimental Evaluation

The authors validated AEGD across a spectrum of tasks, ranging from simple synthetic functions to complex deep learning benchmarks.

Synthetic and Clustering Tasks

On strongly convex quadratic functions and the non-convex Rosenbrock function, AEGD demonstrated superior convergence speeds compared to standard GD and Momentum-based methods. Notably, on the Rosenbrock function, AEGD remained stable with step sizes nearly 20 times larger than the maximum allowable limit for standard GD. In K-means clustering tests using the Iris dataset, AEGD proved more robust to initialization than the standard Expectation-Maximization (EM) algorithm, more frequently identifying the global minimum.

Deep Learning Benchmarks

To test generalization in high-dimensional spaces, the authors trained ResNet-56 and SqueezeNet on the CIFAR-10 and CIFAR-100 datasets. AEGD and its weight-decay variant (AEGDW) consistently outperformed Stochastic Gradient Descent with Momentum (SGDM). Specifically, AEGD achieved approximately a 1% increase in test accuracy after learning rate decay, suggesting that its stability does not come at the cost of generalization.

Discussion and Insights

The primary contribution of AEGD is the successful transfer of the IEQ strategy—a tool typically reserved for simulating physical systems like fluid dynamics—to the realm of machine learning. The resulting "unconditional energy stability" allows practitioners to use large base step sizes without the risk of divergence. Visualizations of optimization trajectories show that while standard methods often overshoot or detour in high-curvature regions, AEGD maintains a more direct path toward the minimum.

Furthermore, AEGD simplifies the hyperparameter tuning process. The algorithm performs reliably with a default energy constant ($c=1$) and requires less fine-tuning of the base learning rate compared to other adaptive optimizers.

Limitations and Future Work

Despite its empirical success, some theoretical and practical questions remain. Identifying the exact critical threshold for the base step size $\eta$ to ensure the energy variable $r$ does not vanish is a complex task for general non-convex functions. Additionally, while results on CIFAR are promising, the behavior and generalization of AEGD in massive-scale architectures (e.g., Large Language Models) require further investigation. Future research may also explore the sensitivity of the constant $c$ across different loss landscapes, such as those characterized by extreme flatness or sharpness.

Summary updated

Abstract

We propose AEGD, a new algorithm for first-order gradient-based optimization of non-convex objective functions, based on a dynamically updated energy variable. The method is shown to be unconditionally energy stable, irrespective of the step size. We prove energy-dependent convergence rates of AEGD for both non-convex and convex objectives, which for a suitably small step size recovers desired convergence rates for the batch gradient descent. We also provide an energy-dependent bound on the stationary convergence of AEGD in the stochastic non-convex setting. The method is straightforward to implement and requires little tuning of hyper-parameters. Experimental results demonstrate that AEGD works well for a large variety of optimization problems: it is robust with respect to initial data, capable of making rapid initial progress. The stochastic AEGD shows comparable and often better generalization performance than SGD with momentum for deep neural networks.
BibTeX
@misc{liu2020aegd,
  author = {Hailiang Liu and Xuping Tian},
  title = {{AEGD: Adaptive Gradient Decent with Energy}},
  year = {2020},
  eprint = {2010.05109},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.05109},
}
Figure