TL;DR
This paper derives the first rigorous continuous-time Stochastic Differential Equation (SDE) models for Sharpness-Aware Minimization (SAM) and its variants. By analyzing these continuous models, the authors provide a clean mathematical explanation for why SAM favors flat minima and uncover a surprising pitfall: under realistic conditions, SAM can actually be attracted to and slowed down by saddle points.
What Problem Does This Paper Address?
Sharpness-Aware Minimization (SAM) has emerged as a highly effective optimizer for improving generalization in deep learning by actively seeking out flat minima. However, understanding why it works so well and how it behaves dynamically has been a major theoretical challenge.
Previous theoretical analyses often simplified SAM into deterministic Ordinary Differential Equations (ODEs) or assumed an unrealistically small perturbation radius ($\rho \to 0$). These simplifications completely miss the stochastic noise inherent to mini-batch training and fail to capture SAM's behavior in practical settings where $\rho$ is larger. This paper fills that gap by providing a mathematically rigorous continuous-time framework that accounts for both stochasticity and realistic perturbation regimes.
Methods
The authors leverage a mathematical framework to construct continuous-time SDE approximations that match the discrete-time updates of SAM and two of its variants: Unnormalized SAM (USAM) and a newly proposed variant called Deterministic Normalization SAM (DNSAM).
They focus on the realistic "moderate perturbation" regime where the perturbation radius scales as $\rho = O(\sqrt{\eta})$ (where $\eta$ is the learning rate). By analyzing these SDEs on quadratic loss landscapes, they decompose the optimizer's trajectory into a deterministic "drift" term (representing implicit regularization) and a stochastic "diffusion" term (representing noise).
Datasets
Because the theoretical framework requires computing full Hessians at every step, empirical validation was performed on smaller, well-controlled benchmarks rather than massive models:
- Iris Database: Used for evaluating deep linear classification.
- Breast Cancer Database: Used for evaluating deep nonlinear classification.
- Synthetic Benchmarks: A convex quadratic landscape and a Teacher-Student model consisting of deep linear and nonlinear MLPs.
- Linear Autoencoder: Specifically chosen to study optimization behavior around known saddle points.
Key Insights and Ideas
- Curvature-Induced Noise: The derived SDEs show that SAM doesn't just implicitly regularize the loss function; it also injects noise that is directly scaled by the Hessian (curvature) of the loss landscape. In sharp minima, this noise explodes, violently kicking the optimizer out. In flat minima, the noise dies down, allowing the optimizer to settle.
- The Saddle Point Trap: Contrary to the common intuition that stochastic noise helps optimizers escape bad critical points, the paper proves that SAM can actually be attracted to saddle points and maxima if the perturbation radius $\rho$ is too large.
- DNSAM as a Better Proxy: The authors show that dropping the gradient normalization (as USAM does) alters the dynamics too much. Their proposed DNSAM retains a deterministic normalization factor, making it a much more accurate theoretical proxy for studying the true SAM algorithm.
Key Results
- The derived SDEs achieve an order-1 weak approximation, meaning they accurately track the probability distribution of the discrete algorithms over time. As shown in Figure 2, the modeling error remains exceptionally low in the moderate $\rho$ regime, whereas a naive SGD SDE approximation fails significantly.
- The authors empirically confirm the "saddle point pitfall." When initialized near a quadratic saddle, SAM and its variants exhibit a cyclical "pull and push" dynamic (Figure 4), getting pulled toward the critical point before volatility spikes eventually push them away, drastically slowing down optimization compared to standard SGD.
Limitations and Future Work
- Computational Scale: The requirement to compute the full network Hessian restricts empirical validation to relatively small models and simple landscapes.
- Hyperparameter Regimes: The analysis primarily targets the moderate perturbation regime ($\rho = O(\sqrt{\eta})$), leaving very large learning rates or alternative scaling choices ($\rho < \eta$) less explored.
- Future Directions: The authors suggest using these SDE models to derive optimal hyperparameter scaling rules (e.g., balancing learning rate, batch size, and $\rho$) to maximize both training speed and generalization without falling into saddle point traps.
Summary updated