Parameter Averaging for SGD Stabilizes the Implicit Bias towards Flat Regions

Atsushi Nitanda, Ryuhei Kikuchi, Shugo Maeda

Abstract

Parameter averaging, such as Stochastic Weight Averaging (SWA), is a widely used technique to improve the generalization of deep learning models. This paper provides a theoretical foundation for its success by framing Stochastic Gradient Descent (SGD) as optimizing a smoothed version of the loss function. We demonstrate that large step sizes intentionally introduce noise that "fills in" sharp local minima, and parameter averaging allows the model to converge to the center of these stable, flat regions, leading to superior generalization.

Introduction

In deep learning, achieving low training loss is insufficient; the ultimate goal is generalization to unseen data. Empirical evidence suggests a "folk theorem": flat minima—regions where the loss is relatively invariant to small parameter perturbations—generalize better than sharp minima. While SGD is known to have an implicit bias toward these regions, techniques like SWA further enhance this by averaging model weights over training. This paper investigates the theoretical mechanism behind why averaging iterates from "unstable," oscillating large-step-size SGD leads to superior solutions.

The Objective Smoothing Perspective

The authors propose an "objective smoothing" view of SGD. In this framework, the algorithm is seen as optimizing a convolved version of the original loss function $f$, denoted as the smoothed objective $F$.

Effect of step size on the smoothed objective $F$. Larger step sizes effectively 'fill in' sharp oscillations in the loss landscape.

The "strength" of this smoothing is directly controlled by the step size $\eta$. A large step size prevents SGD from becoming trapped in narrow, sharp pits because the associated stochastic noise effectively "fills them in." Consequently, the iterates oscillate within wider, flatter valleys. The authors prove that while individual iterates may bounce around the bottom of such a valley, their average converges significantly closer to the true minimizer of the smoothed objective $F$ than any single iterate could.

Key Insights: Noise as a Regularizer

In this framework, stochastic noise is not merely an error but a regularization tool. By utilizing a large step size, noise is amplified to "smooth out" the landscape, making it more amenable to optimization.

Visualization of Averaged SGD (ASGD). Averaging allows iterates with a large step size (blue) to converge to the center of the flat region in the smoothed objective.

The core intuition is analogous to a ball in a bowl-shaped valley: with high energy, the ball bounces back and forth, and any single snapshot finds it far from the center. However, the average position over time is precisely at the optimum. Parameter averaging performs this temporal integration for SGD in flat regions. This necessitates a "Goldilocks" step size: too small, and smoothing is insufficient to escape sharp minima; too large, and oscillations become so violent that even the average is displaced from the optimum.

Empirical Results

The theory was validated using synthetic 1D functions and standard benchmarks including CIFAR-10, CIFAR-100, and Fashion MNIST. Key findings include:

  • Performance Gains: On CIFAR-100 with ResNet-50, Averaged SGD (ASGD) reached 82.87% accuracy, significantly outperforming standard SGD's 80.83%.
  • Computational Efficiency: ASGD achieved results comparable to Sharpness-Aware Minimization (SAM) without the overhead of SAM's dual gradient computations.
  • Landscape Flattening: Measurements of the Hessian trace confirmed that ASGD consistently identifies solutions in regions with lower curvature (flatter) than standard SGD.

Conclusion and Limitations

While the smoothing perspective provides a compelling explanation for the efficacy of parameter averaging, it relies on theoretical assumptions like "one-point strong convexity" for the smoothed objective, which are difficult to verify in high-dimensional neural networks. Furthermore, the benefits of large-step-size smoothing are less pronounced on simpler datasets where the landscape is already relatively smooth. Future work may explore how this perspective applies to non-standard noise distributions and diverse architectures like Transformers.

Summary updated

BibTeX
@inproceedings{nitanda2023parameter,
  author = {Atsushi Nitanda and Ryuhei Kikuchi and Shugo Maeda},
  title = {{Parameter Averaging for SGD Stabilizes the Implicit Bias towards Flat Regions}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=p4RvNzlJX7W},
}
Figure