Abstract
Training deep neural networks (DNNs) effectively requires setting appropriate learning rates for millions of parameters. While popular adaptive methods like Adam rely on the second moment of gradients, they incur significant memory overhead. We introduce Rapidly Adapting Moment Estimation (RAME), a first-moment-based adaptive optimizer. By leveraging the directional information in the moving average of gradients, RAME reduces memory costs while achieving superior convergence and generalization across image classification and semantic segmentation tasks.
Introduction
Standard Stochastic Gradient Descent (SGD) often struggles with the varying scales of updates required by different parameters in deep networks. Existing adaptive methods like Adam and RMSprop address this by tracking the second moment (uncentered variance) of gradients. However, the authors argue that the first moment (momentum) already contains sufficient information about local surface structure, such as curvature. Furthermore, tracking the second moment adds a memory buffer equal to the model size, which becomes a significant bottleneck as models scale.
Methodology
RAME builds upon the "Heavy-Ball" (HB) momentum method by introducing a nonlinear scaling function to the first moment. The parameter update is defined as:
$$x_t = x_{t-1} - \eta_t \frac{m_t}{|m_t|^q + \xi}$$
where $m_t$ is the first moment, $q$ is a hyperparameter (typically 0.125 to 0.25) controlling adaptation aggressiveness, and $\xi$ is a small constant for stability.
Key Innovations
- First-Moment Adaptation: Unlike Adam's $1/\sqrt{v_t}$ scaling, RAME uses $1/(|m_t|^q + \xi)$. This ensures coordinates with large momentum receive smaller relative updates, suppressing oscillations in sharp curvature directions, while those with small momentum are accelerated.
- Memory Efficiency: By eliminating the second-moment buffer, RAME saves a memory buffer the size of the entire model.
- Directional Awareness: The first moment preserves the sign of gradients, which the second moment discards. RAME utilizes this directional information to compute individual learning rates more effectively.
Experimental Results
The authors evaluated RAME on standard benchmarks including CIFAR-10, CIFAR-100, and MS COCO-2017.
Image Classification
On architectures like VGG16 and ResNet20, RAME demonstrated a characteristic learning curve: while it often starts slower than Adam, it eventually achieves lower training loss and higher validation accuracy.
Semantic Segmentation
In more complex tasks like people semantic segmentation on MS COCO using a ResNet152 backend, RAME consistently outperformed both Stochastic Heavy-Ball (SHB) and Adam. It achieved lower validation loss and a higher Intersection over Union (IOU) metric.
Conclusion
RAME presents a memory-efficient alternative to second-moment adaptive optimizers. By nonlinearly scaling the first moment, it achieves competitive or superior performance on diverse deep learning tasks. Future work includes refining hyperparameter tuning for $q$ and exploring hybrid first- and second-moment techniques.
Summary updated