A Method for Enhancing Generalization of Adam by Multiple Integrations

Long Jin, Han Nong, Liangming Chen, Zhenming SuAAAI 2025 · DOI

Introduction

The choice of optimizer is a critical factor in deep learning performance. While Stochastic Gradient Descent (SGD) is renowned for its ability to find "flat minima"—broad regions of the loss landscape that generalize well to unseen data—it often suffers from slow convergence. Conversely, Adam is widely used for its rapid convergence and robustness to sparse gradients but frequently falls into "sharp minima." These narrow valleys exhibit low training error but poor test performance, leading to a significant "generalization gap."

The MIAdam Approach: Integration as a Filter

To bridge this gap, researchers have introduced MIAdam (Multiple Integral Adam). This novel optimizer incorporates a multiple integral term into the standard Adam update rule, drawing inspiration from signal processing.

Conceptual overview of MIAdam: using integration to filter out sharp minima and guide the optimization trajectory toward flatter regions.

In signal processing, integration acts as a low-pass filter, smoothing high-frequency noise. MIAdam treats the optimization trajectory as a signal and sharp minima as high-frequency noise. By applying multiple integrations, the optimizer effectively "ignores" these narrow pits, maintaining a smoother path toward flatter, more robust regions of the loss landscape.

Implementation Details

  • Discrete Summations: In practice, the continuous integrals are approximated using $n$-th order summations of first-order moments.
  • The Switching Mechanism: Theoretical analysis indicates that while the integral term aids in finding flat minima, it can hinder final convergence. MIAdam addresses this by using the integral term during early training phases to "scout" for flat regions, then switching to standard Adam at a predefined point ($\zeta$) to ensure stable convergence.
  • Efficiency: The computational overhead is minimal, requiring only a few additional addition operations per parameter.

Key Insights and Visualizations

The effectiveness of MIAdam is supported by both mathematical proof and empirical visualization. Using diffusion theory, the authors demonstrated that MIAdam has a shorter "mean escape time" from sharp minima compared to standard Adam, making it more likely to exit narrow valleys in favor of wider ones.

Trajectory comparison on a 2D loss landscape: MIAdam3 successfully navigates to a distant flat minimum, while standard Adam remains trapped in a local sharp minimum.

Visualizations on 2D loss landscapes clearly illustrate this behavior. While standard Adam often gets stuck in the nearest sharp minimum, MIAdam’s integrated trajectory allows it to bypass these sub-optimal regions and reach the global, flat minimum. Hessian analysis further confirms that MIAdam consistently converges to regions with lower curvature than its predecessors.

Experimental Results

MIAdam was validated across diverse tasks, including image classification (ImageNet-1k, CIFAR), text classification (BERT/RoBERTa on MR, R8), and robustness tests with label noise.

  • Accuracy Gains: On ImageNet-1k with AlexNet, MIAdam1 achieved a top-1 accuracy of 52.34%, a substantial improvement over Adam's 46.48%.
  • Robustness to Noise: In scenarios with 80% label noise on CIFAR-10, MIAdam1 maintained 67.68% accuracy, significantly outperforming standard Adam and other state-of-the-art variants.
  • Efficiency: Crucially, these generalization improvements were achieved without sacrificing the rapid convergence speed that makes Adam a preferred choice for practitioners.

Conclusion

MIAdam offers a compelling solution to Adam's generalization challenges by leveraging the smoothing properties of multiple integrations. By effectively filtering the loss landscape, it combines the fast convergence of adaptive methods with the superior generalization typically associated with SGD. Future work may explore applying this integration framework to other optimization algorithms to further enhance deep learning performance.

Summary updated

BibTeX
@inproceedings{jin2025method,
  author = {Long Jin and Han Nong and Liangming Chen and Zhenming Su},
  title = {{A Method for Enhancing Generalization of Adam by Multiple Integrations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i4.32435},
}
Figure