Questions for Flat-Minima Optimization of Modern Neural Networks

Jean Kaddour, Linqing Liu, Ricardo Silva, Matt J. Kusner

Abstract

Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic and adaptive gradient-based optimizers. Two methods have received significant attention due to their scalability:

  1. Stochastic Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem.

Introduction

Standard Stochastic Gradient Descent (SGD) often finds solutions that minimize training loss but may generalize poorly to unseen data. A growing body of research suggests that "flat" minima—regions where the loss remains low despite small weight perturbations—correlate with better generalization. While algorithms like Stochastic Weight Averaging (SWA) and Sharpness-Aware Minimization (SAM) have gained popularity for finding these regions, their relative strengths across diverse domains remained largely unexplored. This paper presents a systematic benchmark of these methods across 42 tasks in vision, language, and graph learning, revealing significant domain-dependent performance and introducing a hybrid approach, WASAM, that enhances robustness.

Methods: SWA, SAM, and WASAM

The study evaluates three primary flat-minima optimization strategies:

  1. Stochastic Weight Averaging (SWA): This method keeps a running average of model weights during the final stages of training. The intuition is that averaging iterates will pull the solution toward the center of a flat basin, rather than leaving it at a potentially sharp local minimum.
  2. Sharpness-Aware Minimization (SAM): SAM explicitly seeks flat regions by minimizing the maximum loss within a neighborhood of the current weights. This involves calculating a worst-case perturbation and updating parameters based on the gradient at that perturbed point, effectively penalizing "sharp" peaks in the loss landscape.
  3. Weight-Averaged SAM (WASAM): A novel hybrid proposed by the authors. Recognizing that SAM can sometimes oscillate near the boundaries of flat regions, WASAM applies SWA-style weight averaging to SAM's iterates, aiming to pull the final solution into the stable center of the flat basin.
Conceptual illustration of SWA and SAM on loss landscapes. SWA averages iterates to find the center of a flat region, while SAM seeks a point robust to local perturbations.

Benchmarking Across Domains

The authors conducted an extensive evaluation across 42 tasks:

  • Computer Vision (CV): Supervised and self-supervised tasks on CIFAR-10, CIFAR-100, and ImageNette.
  • Natural Language Processing (NLP): Question answering (NQ, TriviaQA) and the GLUE benchmark.
  • Graph Representation Learning (GRL): Various Open Graph Benchmark (OGB) datasets including Proteins, Products, Code2, and Citation2.

To analyze the geometry of the solutions, the authors visualized loss landscapes through linear interpolation and quantified "flatness" using the dominant eigenvalues of the Hessian matrix.

Key Insights: The "Edge" Problem and Domain Sensitivity

A critical discovery is that SAM, despite its explicit sharpness penalty, often produces solutions that sit precariously at the edge of sharp cliffs in the loss landscape. This "edge problem" can lead to instability. The proposed WASAM method effectively mitigates this by averaging SAM's iterates, pulling the model away from these cliffs into flatter, more robust regions.

Comparison of loss and accuracy/F1 curves for SWA, SAM, and WASAM. WASAM (star) consistently finds flatter regions compared to SAM (cross), which often sits near sharp cliffs.

Furthermore, the benchmark revealed a stark domain split:

  • SWA excels in Graph Representation Learning but struggles with Transformer-based NLP tasks.
  • SAM dominates in NLP but shows limited effectiveness in GRL.
  • Architecture Sensitivity: On the OGB-Citation2 dataset, the choice of layer (GCN vs. SAGE) completely reversed the relative performance of SWA and SAM, highlighting that the optimizer's success is deeply tied to the model architecture.

Results and Robustness

WASAM emerged as the most robust optimizer, matching or exceeding the best individual method in 39 out of 42 cases. Quantitatively, Hessian analysis on CIFAR-100 showed that WASAM achieved a flatness score of 117, significantly lower (and thus flatter) than the baseline (673), SWA (265), and SAM (237).

The study also found that flat-minima optimizers are surprisingly effective in self-supervised learning, a domain for which they were not originally designed. Overall, applying these optimizers is a low-risk, high-reward strategy: performance drops were negligible (max -0.30%), while gains reached up to +2.60%.

Conclusion

This systematic study clarifies the strengths and weaknesses of SWA and SAM across diverse deep learning domains. By identifying the "edge problem" in SAM and proposing WASAM as a robust solution, the authors provide a practical guide for optimizer selection. Future work remains to explore these methods in reinforcement learning and to better align training loss flatness with actual test performance.

Summary updated

Abstract

Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic and adaptive gradient-based optimizers. Two methods have received significant attention due to their scalability: 1. Stochastic Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem.
BibTeX
@inproceedings{kaddour2022questions,
  author = {Jean Kaddour and Linqing Liu and Ricardo Silva and Matt J. Kusner},
  title = {{Questions for Flat-Minima Optimization of Modern Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=zYjlwYDwLcS},
}
Figure