An SDE for Modeling SAM: Theory and Insights

Enea Monzio Compagnoni, Luca Biggio, Antonio Orvieto, Frank Norbert Proske, Hans Kersting, Aurelien Lucchi

Introduction

Sharpness-Aware Minimization (SAM) has emerged as a powerful optimizer in deep learning, specifically designed to find "flat" minima that correlate with better generalization. Despite its empirical success, the discrete, two-step nature of SAM—involving an adversarial ascent step followed by a gradient descent step—makes it difficult to analyze using standard optimization theory. This research introduces a rigorous continuous-time framework using Stochastic Differential Equations (SDEs) to decode SAM’s dynamics and explain its unique geometry-seeking behavior.

The Theoretical Framework

To bridge the gap between discrete updates and theoretical analysis, the authors derive SDEs that approximate the trajectories of SAM and its variants. They focus on three versions:

  1. SAM: The full version with gradient normalization.
  2. USAM: A simplified "unnormalized" version often used in previous theoretical work.
  3. DNSAM (Deterministic Normalization SAM): A new variant introduced by the authors that serves as a more accurate theoretical proxy for SAM than USAM.

Through mathematical derivation, the authors prove that these SDEs are "order 1 weak approximations." This means the distribution of the SDE's path closely matches the distribution of the actual discrete-time algorithm over time, providing a reliable foundation for studying SAM's long-term behavior.

Implicit Regularization and Curvature-Induced Noise

The SDE derivation reveals that SAM is effectively minimizing a "regularized" loss function. It does not simply follow the gradient of the loss; it also seeks to minimize the expected norm of the gradient, which naturally pulls the optimizer toward flatter regions.

A key insight from this framework is the role of curvature-induced noise. The SDE shows that SAM introduces an additional noise term that depends on the Hessian (the curvature) of the loss landscape. In sharp minima where curvature is high, this noise becomes massive, making the optimizer unstable and forcing it to leave the region. In flat minima, the noise is low, allowing the optimizer to settle. This provides a formal mathematical explanation for how SAM avoids sharp regions that typically lead to poor generalization.

The Saddle Point Trap

While SAM is designed to avoid sharp minima, the SDE analysis reveals a potential pitfall: the "Saddle Point Trap." If the ascent radius ($ ho$) is set too high, the adversarial step can dominate the dynamics. The authors prove that under certain conditions, SAM can actually be attracted to saddle points or even local maxima—points that standard Stochastic Gradient Descent (SGD) would typically bypass. This counter-intuitive behavior suggests that SAM might oscillate around or get stuck near saddles if hyperparameters are poorly tuned.

Experimental Validation

The authors validated their theoretical SDE models against actual discrete-time runs using several benchmarks, including:

  • Convex Quadratic Landscapes: For fundamental sanity checks.
  • Iris and Breast Cancer Databases: Using linear and nonlinear Multi-Layer Perceptrons (MLPs).
  • Linear Autoencoders: Specifically chosen to study dynamics near known saddle points.

The results confirmed that the derived SDEs—particularly the DNSAM variant—accurately track the loss evolution and parameter trajectories of the real algorithms. Experiments with linear autoencoders specifically confirmed the "Saddle Point Trap," showing that SAM fails to escape the origin when initialized too close with a large $ ho$, whereas SGD escapes easily.

Conclusion and Future Work

By providing the first rigorous SDE framework for SAM, this research clarifies why the optimizer succeeds and identifies critical failure modes. The discovery that DNSAM is a superior theoretical proxy to USAM suggests that future theoretical work should shift its focus toward normalized variants. While the paper identifies the dangers of a large ascent radius, future work could use these SDEs to derive universal scaling rules to automatically balance learning rates, batch sizes, and the SAM radius for optimal performance.

Summary updated

Abstract

We study the SAM (Sharpness-Aware Minimization) optimizer which has recently attracted a lot of interest due to its increased performance over more classical variants of stochastic gradient descent. Our main contribution is the derivation of continuous-time models (in the form of SDEs) for SAM and two of its variants, both for the full-batch and mini-batch settings. We demonstrate that these SDEs are rigorous approximations of the real discrete-time algorithms (in a weak sense, scaling linearly with the learning rate). Using these models, we then offer an explanation of why SAM prefers flat minima over sharp ones -- by showing that it minimizes an implicitly regularized loss with a Hessian-dependent noise structure. Finally, we prove that SAM is attracted to saddle points under some realistic conditions. Our theoretical results are supported by detailed experiments.
BibTeX
@inproceedings{compagnoni2023sde,
  author = {Enea Monzio Compagnoni and Luca Biggio and Antonio Orvieto and Frank Norbert Proske and Hans Kersting and Aurelien Lucchi},
  title = {{An SDE for Modeling SAM: Theory and Insights}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=hYPLU1nFZRX},
}
Figure