An SDE for Modeling SAM: Theory and Insights

Enea Monzio Compagnoni, Luca Biggio, Antonio Orvieto, Frank Norbert Proske, Hans Kersting, Aurelien Lucchi

TL;DR

This paper derives the first rigorous continuous-time Stochastic Differential Equation (SDE) models for Sharpness-Aware Minimization (SAM) and its variants. By analyzing these continuous models, the authors provide a clean mathematical explanation for why SAM favors flat minima and uncover a surprising pitfall: under realistic conditions, SAM can actually be attracted to and slowed down by saddle points.

What Problem Does This Paper Address?

Sharpness-Aware Minimization (SAM) has emerged as a highly effective optimizer for improving generalization in deep learning by actively seeking out flat minima. However, understanding why it works so well and how it behaves dynamically has been a major theoretical challenge.

Previous theoretical analyses often simplified SAM into deterministic Ordinary Differential Equations (ODEs) or assumed an unrealistically small perturbation radius ($\rho \to 0$). These simplifications completely miss the stochastic noise inherent to mini-batch training and fail to capture SAM's behavior in practical settings where $\rho$ is larger. This paper fills that gap by providing a mathematically rigorous continuous-time framework that accounts for both stochasticity and realistic perturbation regimes.

Methods

The authors leverage a mathematical framework to construct continuous-time SDE approximations that match the discrete-time updates of SAM and two of its variants: Unnormalized SAM (USAM) and a newly proposed variant called Deterministic Normalization SAM (DNSAM).

They focus on the realistic "moderate perturbation" regime where the perturbation radius scales as $\rho = O(\sqrt{\eta})$ (where $\eta$ is the learning rate). By analyzing these SDEs on quadratic loss landscapes, they decompose the optimizer's trajectory into a deterministic "drift" term (representing implicit regularization) and a stochastic "diffusion" term (representing noise).

Datasets

Because the theoretical framework requires computing full Hessians at every step, empirical validation was performed on smaller, well-controlled benchmarks rather than massive models:

  • Iris Database: Used for evaluating deep linear classification.
  • Breast Cancer Database: Used for evaluating deep nonlinear classification.
  • Synthetic Benchmarks: A convex quadratic landscape and a Teacher-Student model consisting of deep linear and nonlinear MLPs.
  • Linear Autoencoder: Specifically chosen to study optimization behavior around known saddle points.

Key Insights and Ideas

  • Curvature-Induced Noise: The derived SDEs show that SAM doesn't just implicitly regularize the loss function; it also injects noise that is directly scaled by the Hessian (curvature) of the loss landscape. In sharp minima, this noise explodes, violently kicking the optimizer out. In flat minima, the noise dies down, allowing the optimizer to settle.
  • The Saddle Point Trap: Contrary to the common intuition that stochastic noise helps optimizers escape bad critical points, the paper proves that SAM can actually be attracted to saddle points and maxima if the perturbation radius $\rho$ is too large.
  • DNSAM as a Better Proxy: The authors show that dropping the gradient normalization (as USAM does) alters the dynamics too much. Their proposed DNSAM retains a deterministic normalization factor, making it a much more accurate theoretical proxy for studying the true SAM algorithm.

Key Results

  • The derived SDEs achieve an order-1 weak approximation, meaning they accurately track the probability distribution of the discrete algorithms over time. As shown in Figure 2, the modeling error remains exceptionally low in the moderate $\rho$ regime, whereas a naive SGD SDE approximation fails significantly.
Error comparison for different models
  • The authors empirically confirm the "saddle point pitfall." When initialized near a quadratic saddle, SAM and its variants exhibit a cyclical "pull and push" dynamic (Figure 4), getting pulled toward the critical point before volatility spikes eventually push them away, drastically slowing down optimization compared to standard SGD.
Cyclical dynamics near a quadratic saddle

Limitations and Future Work

  • Computational Scale: The requirement to compute the full network Hessian restricts empirical validation to relatively small models and simple landscapes.
  • Hyperparameter Regimes: The analysis primarily targets the moderate perturbation regime ($\rho = O(\sqrt{\eta})$), leaving very large learning rates or alternative scaling choices ($\rho < \eta$) less explored.
  • Future Directions: The authors suggest using these SDE models to derive optimal hyperparameter scaling rules (e.g., balancing learning rate, batch size, and $\rho$) to maximize both training speed and generalization without falling into saddle point traps.

Summary updated

Abstract

We study the SAM (Sharpness-Aware Minimization) optimizer which has recently attracted a lot of interest due to its increased performance over more classical variants of stochastic gradient descent. Our main contribution is the derivation of continuous-time models (in the form of SDEs) for SAM and two of its variants, both for the full-batch and mini-batch settings. We demonstrate that these SDEs are rigorous approximations of the real discrete-time algorithms (in a weak sense, scaling linearly with the learning rate). Using these models, we then offer an explanation of why SAM prefers flat minima over sharp ones~--~by showing that it minimizes an implicitly regularized loss with a Hessian-dependent noise structure. Finally, we prove that SAM is attracted to saddle points under some realistic conditions. Our theoretical results are supported by detailed experiments.
BibTeX
@inproceedings{compagnoni2023sde,
  author = {Enea Monzio Compagnoni and Luca Biggio and Antonio Orvieto and Frank Norbert Proske and Hans Kersting and Aurelien Lucchi},
  title = {{An SDE for Modeling SAM: Theory and Insights}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=hYPLU1nFZRX},
}
Figure