In the world of deep learning, choosing an optimizer is often a tug-of-war between speed and stability. Adam is the go-to for rapid convergence, but it often struggles to generalize as well as good old-fashioned Stochastic Gradient Descent (SGD). Meanwhile, modern architectures use normalization layers (like Batch Norm) that introduce "scale invariance," a property that can cause standard momentum-based optimizers to waste energy growing weight norms rather than minimizing loss.
A new paper from researchers at the University of Electronic Science and Technology of China introduces PadamP, a novel optimization algorithm designed to solve these specific headaches. By combining projection gradients with partially adaptive moment estimation, PadamP aims to give researchers the best of both worlds: the "get-up-and-go" of Adam and the robust generalization of SGD.
TL;DR
PadamP is a new optimizer that prevents unnecessary weight growth in scale-invariant layers by using a dynamic projection technique. It bridges the gap between Adam and SGD using a tunable $p$-th power for moment estimation, resulting in faster convergence and superior test accuracy on standard image benchmarks.
What Problem Does This Paper Address?
The authors tackle two main issues in deep learning optimization:
- The Generalization Gap: Adaptive optimizers like Adam often converge quickly but fail to find the "flat minima" that lead to good performance on unseen data.
- Radial Weight Growth: In layers with Batch Normalization or Layer Normalization, the direction of the weights matters more than their magnitude (scale invariance). Standard momentum causes weight norms to increase rapidly over time. This "radial growth" effectively shrinks the effective learning rate, slowing down training and potentially hurting performance.
While previous algorithms like AdamP introduced projection techniques to stop this growth, and Padam introduced partial adaptivity, PadamP is the first to integrate these ideas while refining the mathematical conditions for when and how to project the gradient.
Methods
PadamP is built on three key pillars:
- Partial Adaptivity ($p$-th power): Instead of the standard square root ($p=1/2$) used in Adam to scale gradients, PadamP uses a $p$-th power where $p \in (0, 1/2]$. This allows the algorithm to behave more like SGD (as $p$ approaches 0) or more like Adam (at $p=1/2$), providing a "knob" to tune the adaptivity.
- Projection Gradient ($\Pi_\theta$): To handle scale-invariant weights, the algorithm projects the update direction onto the tangent space of the weights. This effectively removes the "radial" component of the gradient—the part that only increases the weight's norm without changing the network's output.
- Dynamic Projection Condition: Unlike its predecessor AdamP, PadamP incorporates the learning rate ($\eta_t$) into its projection criteria. As the learning rate decays, the condition for projecting becomes more stringent. This makes the projection more adaptive to the different stages of training.
The authors also provide a rigorous convergence proof for non-convex problems that "decouples" the momentum coefficients ($\beta_1$ and $\beta_2$), a theoretical hurdle that many previous papers had to bypass with restrictive assumptions.
Datasets
The researchers evaluated PadamP using two classic image classification benchmarks:
- CIFAR-10: 60,000 images across 10 classes.
- CIFAR-100: The same number of images but spread across 100 more granular classes.
They tested these datasets using two popular architectures: VGG-16 and ResNet-18.
Key Insights and Ideas
- The "P" is for Practicality: By including the learning rate in the projection condition, the algorithm automatically realizes that projection is less necessary in the final stages of training when the objective function is near an optimal point.
- Decoupling Theory from Constraints: Most convergence proofs for Adam-type optimizers require $\beta_1 / \sqrt{\beta_2} < 1$. This paper generalizes the proof, making it applicable to a wider range of real-world hyperparameter settings.
- The Power of $p$: The authors found that a smaller $p$ (like $1/8$) is particularly effective in the later stages of training, suggesting that "less adaptivity" can actually lead to better stability and lower final loss.
Key Results
The experimental results show that PadamP consistently outperforms Adam, AdamP, and SGDM:
- Superior Generalization: On VGG-16 (CIFAR-100), PadamP achieved significantly higher test accuracy than Adam and AdamP, proving it finds better-quality solutions.
- Faster Convergence: PadamP reached 100% training accuracy faster than competing optimizers.
- Stability: The algorithm showed more stable loss curves, particularly when using a strategy of decaying the $p$-value (e.g., shifting from $p=1/4$ to $p=1/8$ at 100 epochs).
- Optimal Settings: For VGG-16, $p=1/5$ was found to be the sweet spot, while $p=1/4$ worked best for ResNet-18.
Limitations and Future Work
While PadamP shows great promise, the authors note a few areas for further exploration:
- Hyperparameter Tuning: While $p$ adds flexibility, it also adds another hyperparameter to tune. The authors suggest an adaptive $p$-decay strategy, but more automated ways to find the optimal $p$ would be beneficial.
- Scaling Up: The current experiments focus on CIFAR datasets. Testing on massive datasets like ImageNet or in different domains like Natural Language Processing (Transformers) would be the next logical step.
- Advanced Techniques: The authors suggest that combining PadamP with variance reduction or line-search methods could further enhance its efficiency.
Summary updated