AdaGrad Converges in a Robust Sense: Almost Sure Last-Iterate Rates under Any Stopping Time

Weiming Ou, Zhihao Liu, Xiao WangOpenReview 2026 · Submission to ICLR 2026 · Rejected · OpenReview

Abstract

AdaGrad has become a widely used algorithm for training deep models. Recently, the study of almost sure last-iterate convergence rates in stochastic optimization has attracted increasing attention, as it provides the guarantee of stability and **robustness** for arbitrary single trajectory. While such results are well understood for stochastic gradient descent (SGD), the corresponding analysis for AdaGrad remains limited. In this paper, we establish **almost sure** convergence rates of AdaGrad for the **last-iterate** in the (strongly) convex setting and for the best-iterate in the non-convex setting, both valid under **arbitrary** stopping times and with a flexible dependence on gradient history.
BibTeX
@misc{ou2026adagrad,
  author = {Weiming Ou and Zhihao Liu and Xiao Wang},
  title = {{AdaGrad Converges in a Robust Sense: Almost Sure Last-Iterate Rates under Any Stopping Time}},
  year = {2026},
  url = {https://openreview.net/forum?id=DwWorqSjwv},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}

Loading related content...

Figure