No article text was recorded for this version.
adaptive optimizer stability
Version 1 of 1392 papers403 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Robustness to Unbounded Smoothness of Generalized SignSGD
- Blog: Robustness to Unbounded Smoothness of Generalized SignSGD
- Blog: Simple Convergence Proof of Adam From a Sign-like Descent Perspective
- Blog: Adam with model exponential moving average is effective for nonconvex optimization
- Blog: Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions
- Blog: RMSprop converges with proper hyper-parameter
- Blog: Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling
- Blog: Universal Stagewise Learning for Non-Convex Problems with Convergence on Averaged Solutions
- Blog: A new regret analysis for Adam-type algorithms
- Blog: Revisit last-iterate convergence of mSGD under milder requirement on step size
- Blog: Sharp higher order convergence rates for the Adam optimizer
- Blog: On the Convergence of Weighted AdaGrad with Momentum for Training Deep Neural Networks
- Blog: How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks
- Blog: Local Methods with Adaptivity via Scaling
- Blog: Step-Size Stability in Stochastic Optimization: A Theoretical Perspective
- Blog: Critical Bach Size Minimizes Stochastic First-Order Oracle Complexity of Deep Learning Optimizer using Hyperparameters Close to One
- Blog: Efficiency Ordering of Stochastic Gradient Descent
- Blog: ATRI: Mitigating Multilingual Audio Text Retrieval Inconsistencies by Reducing Data Distribution Errors
- Blog: Convergence of Distributed Adaptive Optimization with Local Updates
- Blog: Convergences for Minimax Optimization Problems over Infinite-Dimensional Spaces Towards Stability in Adversarial Training
- Blog: Convergence of adaptive algorithms for constrained weakly convex optimization
- Blog: Torque-Aware Momentum
- Blog: The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods
- Blog: Random Scaling and Momentum for Non-smooth Non-convex Optimization
- Blog: Momentum via Primal Averaging: Theoretical Insights and Learning Rate Schedules for Non-Convex Optimization
- Blog: A Novel Unified Parametric Assumption for Nonconvex Optimization
- Blog: Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models
- Blog: On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization
- Blog: On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent
- Blog: Beyond Lipschitz Smoothness: A Tighter Analysis for Nonconvex Optimization
- Blog: Linear Convergence of Adaptive Stochastic Gradient Descent
- Blog: Adaptive Braking for Mitigating Gradient Delay
- Blog: SDEs for Minimax Optimization
- Blog: A Unified Coded Deep Neural Network Training Strategy Based on Generalized PolyDot Codes for Matrix Multiplication
- Blog: Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration
- Blog: On Scaled Methods for Saddle Point Problems
- Blog: Clapping: Removing Per-sample Storage for Pipeline Parallel Distributed Optimization with Communication Compression
- Blog: Convergence of adaptive algorithms for weakly convex constrained optimization
- Blog: The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
- Blog: On the Global Optimality of Model-Agnostic Meta-Learning
- Blog: Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentum
- Blog: SDEs for Minimax Optimization
- Blog: The surprising agreement between convex optimization theory and learning-rate scheduling for large model training
- Blog: Nonlinearly Preconditioned Gradient Methods under Generalized Smoothness
- Blog: Statistical Guarantees for High-Dimensional Stochastic Gradient Descent
- Blog: The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance
- Blog: Fundamental Convergence Analysis of Sharpness-Aware Minimization
- Blog: Properties of Discrete Sliced Wasserstein Losses
- Blog: On the Convergence of Stochastic Multi-Objective Gradient Manipulation and Beyond
- Blog: AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations
- Blog: AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
- Blog: Get rid of your constraints and reparametrize: A study in NNLS and implicit bias
- Blog: An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes
- Blog: Decentralized SGD and Average-direction SAM are Asymptotically Equivalent
- Blog: Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
- Blog: Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness
- Blog: Optimization for Neural Operators can Benefit from Width
- Blog: Stochastic Approximate Gradient Descent via the Langevin Algorithm
- Blog: Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed
- Blog: Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent
Showing 60 of 392 papers.
History & citation
Current · Permanent link
- Theoretical Foundations for Practical Learning Rate Schedules merged into this direction (51% overlap)
- Theoretical Foundations of Adaptive Optimizers in Deep Learning merged into this direction (74% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_adaptive_optimizer_stability_661940cd,
title = {adaptive optimizer stability},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/adaptive-optimizer-stability/rdr_661940cdeeb4428b8c0151ee2e6dd5ff},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}This exact version (version 1, frozen text):
@misc{lacuna_adaptive_optimizer_stability_661940cd_v1,
title = {adaptive optimizer stability},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/adaptive-optimizer-stability/rdr_661940cdeeb4428b8c0151ee2e6dd5ff/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}