An Adaptive Policy to Employ Sharpness-Aware Minimization

Weisen Jiang, Hansi Yang, Yu Zhang, James T. Kwok

Abstract

Sharpness-aware minimization (SAM), which searches for flat minima by min-max optimization, has been shown to be useful in improving model generalization. However, since each SAM update requires computing two gradients, its computational cost and training time are both doubled compared to standard empirical risk minimization (ERM). Recent state-of-the-arts reduce the fraction of SAM updates and thus accelerate SAM by switching between SAM and ERM updates randomly or periodically. In this paper, we design an adaptive policy to employ SAM based on the loss landscape geometry. Two efficient algorithms, AE-SAM and AE-LookSAM, are proposed. We theoretically show that AE-SAM has the same convergence rate as SAM. Experimental results on various datasets and architectures demonstrate the efficiency and effectiveness of the adaptive policy.

Introduction

Deep neural networks often settle into "sharp" minima in the loss landscape, which are associated with poor generalization. Sharpness-Aware Minimization (SAM) addresses this by explicitly seeking flatter regions of the loss surface. However, the primary bottleneck of SAM is its computational overhead: it requires two gradient computations per update, effectively doubling the training time compared to standard Empirical Risk Minimization (ERM).

Previous attempts to mitigate this cost have relied on "blind" strategies, such as applying SAM periodically or at random intervals. These methods are inherently inefficient as they ignore the local geometry of the loss landscape, potentially wasting expensive SAM updates on already-flat regions or missing sharp regions where SAM is most needed. This paper introduces an Adaptive policy to Employ SAM (AE-SAM), which uses geometry-aware triggers to apply SAM only when the loss landscape is locally sharp.

Methodology: Geometry-Aware Adaptive Triggering

The core of the AE-SAM approach is a statistical trigger that determines when to pay the "SAM tax." Since direct computation of the Hessian is computationally prohibitive, the authors utilize the squared stochastic gradient norm as a proxy for local sharpness.

Statistical Modeling

Empirical observations indicate that these gradient norms follow a Normal distribution throughout the training process. By employing an Exponential Moving Average (EMA), the algorithm tracks the mean ($\mu_t$) and variance ($\sigma_t^2$) of these norms in real-time without requiring additional gradient calls.

Distributions and Q-Q plots of squared gradient norms at different training stages.

The Trigger Mechanism

SAM is triggered only when the current gradient norm significantly exceeds the historical average, indicating a sharp region: $$\|\nabla \mathcal{L}\|^2 \ge \mu_t + c_t \sigma_t$$ where $c_t$ is a threshold. To optimize efficiency, the authors use a linear schedule for $c_t$, prioritizing ERM for speed in early training and increasing SAM frequency as the model nears convergence to boost generalization. This policy was also integrated with LookSAM (resulting in AE-LookSAM), which reuses previous SAM directions to further reduce costs.

Experimental Results

The researchers evaluated AE-SAM and AE-LookSAM across multiple benchmarks, including CIFAR-10/100, ImageNet, and corrupted datasets with label noise.

Efficiency and Accuracy

On CIFAR-100 with a ResNet-18 architecture, AE-SAM achieved 80.48% accuracy using only approximately 50% of the SAM updates, outperforming standard SAM (80.17%) which uses 100% SAM updates. AE-LookSAM further improved efficiency, achieving comparable results to standard SAM while utilizing only ~20% of the SAM updates.

Evolution of the expected squared gradient norm across training for various architectures.

Robustness to Noise

AE-LookSAM demonstrated exceptional robustness in high-noise environments. In scenarios with 80% label noise, it maintained a testing accuracy of 77.03% on ResNet-32, whereas standard ERM collapsed to 29%. This suggests that geometry-aware updates are particularly effective at preventing the model from memorizing noisy data in sharp minima.

Conclusion

AE-SAM provides a theoretically sound and empirically effective method for accelerating Sharpness-Aware Minimization. By treating sharp regions as outliers that require specialized optimization, the adaptive policy achieves superior generalization with roughly half the computational overhead of standard SAM. Future work may explore per-layer adaptive policies to further refine the efficiency of geometry-aware optimization.

Summary updated

Abstract

Sharpness-aware minimization (SAM), which searches for flat minima by min-max optimization, has been shown to be useful in improving model generalization. However, since each SAM update requires computing two gradients, its computational cost and training time are both doubled compared to standard empirical risk minimization (ERM). Recent state-of-the-arts reduce the fraction of SAM updates and thus accelerate SAM by switching between SAM and ERM updates randomly or periodically. In this paper, we design an adaptive policy to employ SAM based on the loss landscape geometry. Two efficient algorithms, AE-SAM and AE-LookSAM, are proposed. We theoretically show that AE-SAM has the same convergence rate as SAM. Experimental results on various datasets and architectures demonstrate the efficiency and effectiveness of the adaptive policy.
BibTeX
@inproceedings{jiang2023adaptive,
  author = {Weisen Jiang and Hansi Yang and Yu Zhang and James T. Kwok},
  title = {{An Adaptive Policy to Employ Sharpness-Aware Minimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=6Wl7-M2BC-},
}
Figure