Abstract
It is often observed that stochastic gradient descent (SGD) and its variants implicitly select a solution with good generalization performance; such implicit bias is often characterized in terms of the sharpness of the minima. Kleinberg et al. (2018) connected this bias with the smoothing effect of SGD which eliminates sharp local minima by the convolution using the stochastic gradient noise. We follow this line of research and study the commonly-used averaged SGD algorithm, which has been empirically observed in Izmailov et al. (2018) to prefer a flat minimum and therefore achieves better generalization. We prove that in certain problem settings, averaged SGD can efficiently optimize the smoothed objective which avoids sharp local minima. In experiments, we verify our theory and show that parameter averaging with an appropriate step size indeed leads to significant improvement in the performance of SGD.
Introduction
In deep learning, "flat minima"—broad, shallow valleys in the loss landscape—are widely recognized for their superior generalization capabilities compared to "sharp minima." While Stochastic Gradient Descent (SGD) exhibits an implicit bias toward these flat regions, particularly with large learning rates, this creates a practical paradox: the very noise that helps find flat regions also induces instability, preventing convergence to a stable point. Practitioners frequently employ Averaged SGD or Stochastic Weight Averaging (SWA) to mitigate this. This paper provides a rigorous theoretical framework to explain why the average of these unstable oscillations outperforms the final iterate of a stable, small-step-size SGD.
Theoretical Framework: Objective Smoothing
The core of the analysis lies in viewing SGD not as an optimizer for the raw objective $f(w)$, but for a smoothed objective function $F(v)$. This smoothed objective is defined as the expectation of the original loss function convolved with the stochastic gradient noise. In this view, the noise inherent in SGD acts as a "blurring" filter that erases sharp local minima, leaving only broad, flat valleys.
The "strength" of this smoothing is directly controlled by the step size ($\eta$). A larger step size creates a smoother landscape, potentially removing poor-generalizing local minima that would otherwise trap the optimizer. However, a large step size also means individual iterates ($v_t$) will oscillate wildly within these valleys. The authors prove that the average of these iterates ($v_T$) effectively "denoises" the path, converging closer to the minimizer of the smoothed objective than any single iterate could.
Experimental Validation
The theory was validated using standard image classification benchmarks, including CIFAR-10 and CIFAR-100, across various architectures like ResNet-50 and WideResNet.
Key Results
- Performance Gains: On CIFAR-100 with ResNet-50, Averaged SGD achieved 82.87% accuracy, a significant improvement over the 80.83% achieved by standard SGD.
- Comparison with SAM: Sharpness-Aware Minimization (SAM) is a state-of-the-art method for finding flat minima. The authors demonstrate that simple Averaged SGD, with a properly tuned step size, achieves comparable performance. For instance, on CIFAR-100 with PyramidNet, Averaged SAM reached 86.00%, while standard SAM achieved 84.59%.
- Stability in High-Noise Regimes: Experiments confirm that Averaged SGD remains robust even when the step size is large enough to make standard SGD completely unstable.
Discussion
The "denoising" effect of averaging is particularly crucial in asymmetric valleys, which are common in real-world loss landscapes. While standard SGD might be pushed toward the "walls" of such valleys by noise, averaging pulls the parameters toward the flatter, more robust side.
This perspective suggests that large step sizes are not merely a hurdle to be overcome by annealing, but a vital feature that allows the optimizer to "see" the smoothed objective. Without sufficient noise, the algorithm may remain trapped in sharp minima.
Limitations and Future Work
The current theoretical proofs rely on "one-point strong convexity" for the smoothed objective, an assumption that may not fully capture the complexity of non-convex neural network landscapes. Additionally, the benefits of averaging are sensitive to step size tuning. Future research could extend this analysis to other architectures, such as Transformers, where weight averaging is increasingly prevalent.
Summary updated