When Do Flat Minima Optimizers Work?

Jean Kaddour, Linqing Liu, Ricardo Silva, Matt J. Kusner

Introduction

In deep learning, achieving high accuracy on training data is only half the battle; the ultimate goal is generalization to unseen data. A prevailing hypothesis suggests that "flat minima"—regions in the parameter space where the loss remains low even with slight perturbations—generalize significantly better than "sharp minima."

To find these desirable regions, two major optimization algorithms have gained traction: Stochastic Weight Averaging (SWA) and Sharpness-Aware Minimization (SAM). However, because these methods were developed independently and tested on different architectures, practitioners have lacked clear guidance on which to use. This paper provides the first systematic, head-to-head comparison of SWA and SAM across 42 tasks in computer vision, NLP, and graph learning. In doing so, it reveals a critical flaw in SAM's geometry and introduces a highly effective hybrid solution: WASAM (Weight-Averaged SAM).

The Contenders: SWA vs. SAM

The authors investigate three primary optimization strategies:

  1. Stochastic Weight Averaging (SWA): An "implicit" approach. It assumes that standard Stochastic Gradient Descent (SGD) iterates bounce around the perimeter of a flat minimum. By averaging these weights over time, SWA moves the model toward the center of that flat basin.
  2. Sharpness-Aware Minimization (SAM): An "explicit" approach. During every update step, SAM performs a "look-ahead" to find the worst-case loss in a small local neighborhood, then updates the weights to minimize that peak. This explicitly penalizes sharpness but doubles the computational cost.
  3. WASAM (Weight-Averaged SAM): The authors' novel hybrid. It applies the weight-averaging logic of SWA directly to the iterates produced by SAM.

Key Insights: The "Cliff-Edge" Phenomenon

By analyzing the loss surfaces using linear interpolation and Hessian eigenvalues (a mathematical measure of "curviness"), the authors uncovered a striking geometric difference between the optimizers.

Conceptual illustration of SWA vs. SAM

While SAM successfully finds regions with lower overall sharpness (lower Hessian eigenvalues) than SWA, its final iterate often sits dangerously close to a "sharp cliff"—a sudden, steep increase in loss. SWA, by contrast, naturally settles in the center of a basin, safely away from edges, even if that basin is technically less flat than SAM's.

This is where WASAM shines as the "best of both worlds." By averaging the SAM iterates, WASAM pulls the model away from the sharp cliffs and centers it safely within the exceptionally flat regions discovered by SAM.

Domain Sensitivity

The benchmarking also revealed that optimizer efficacy is highly domain-dependent:

  • NLP & Vision Transformers: SAM (and WASAM) are significantly more effective. SWA often fails to improve, or even degrades, Vision Transformer performance.
  • Graph Representation Learning (GRL): SWA frequently outperforms standard SAM on graph tasks.

Key Results and Flatness Quantification

Across the 42 benchmarked tasks, WASAM consistently delivered the best generalization, outperforming the baseline in 39 of them.

  • Generalization Boosts: In NLP, SAM and WASAM provided substantial gains. For instance, WASAM achieved a +1.41 boost on the CoLA task compared to the baseline.
  • Quantifying Flatness: The paper measures the "sharpness" of minima using the dominant Hessian eigenvalue ($\lambda_{max}$). Lower values indicate flatter, more stable minima.
Dominant Hessian eigenvalues for different optimizers

As shown above, on CIFAR-100, SWA more than halved the baseline sharpness. SAM reduced it further, but WASAM slashed the sharpness to an impressive 117, confirming it finds the flattest minima.

  • Robustness: Even on tasks where SWA or SAM individually struggled to improve upon the baseline, WASAM remained robust, rarely degrading performance.

Limitations and Future Directions

While flat-minima optimizers offer clear benefits, they come with trade-offs:

  • Hyperparameter Sensitivity: Both methods introduce new hyperparameters (e.g., SAM's neighborhood radius $\rho$, SWA's averaging schedule) that require careful, dataset-specific tuning.
  • Computational Cost: SAM and WASAM require two forward/backward passes per step, making them twice as expensive as standard SGD or SWA. The authors argue the generalization gains often justify this cost, particularly in NLP.
  • Unexplained Failures: On certain tasks (like the OGB-Proteins dataset), none of the flat-minima optimizers beat the baseline. This suggests that in some domains, training loss flatness does not perfectly correlate with test accuracy, highlighting the need for novel flatness metrics.

Ultimately, this comprehensive benchmark demystifies flat-minima optimizers, offering practitioners a clear roadmap for deploying them and establishing WASAM as a powerful new default for robust deep learning.

Summary updated

Abstract

Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic and adaptive gradient-based optimizers. Two methods have received significant attention due to their scalability: 1. Stochastic Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem.
BibTeX
@inproceedings{kaddour2022when,
  author = {Jean Kaddour and Linqing Liu and Ricardo Silva and Matt J. Kusner},
  title = {{When Do Flat Minima Optimizers Work?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=zYjlwYDwLcS},
}
Figure