TL;DR
Monge SAM (M-SAM) is a novel optimization algorithm that achieves reparameterization invariance in Sharpness-Aware Minimization (SAM) by leveraging the Monge metric of the loss surface. This "conservative" approach significantly enhances robustness to hyperparameters and prevents the optimizer from being trapped by suboptimal saddle points or local maxima.
The Problem: Parameter Sensitivity in SAM
Sharpness-Aware Minimization (SAM) has gained popularity for its ability to find "flat" minima, which typically generalize better to unseen data. It achieves this by seeking points with low loss even under small adversarial perturbations. However, SAM relies on the Euclidean metric, making it sensitive to how a model is parameterized. Simple weight rescaling—common in networks with Batch Normalization or ReLU—can change the measured "sharpness" without altering the underlying function. This lack of invariance makes SAM sensitive to hyperparameter tuning and prone to attraction by saddle points or local maxima.
Methods: Geometry-Aware Optimization
To address these flaws, the authors introduce Monge SAM (M-SAM). Rather than treating the parameter space as a flat Euclidean plane, M-SAM treats it as a Riemannian manifold where distance is defined by the Monge metric: $G(\theta) = I + \nabla \ell(\theta) \nabla \ell(\theta)^\top$.
This metric effectively "stretches" the space in directions of rapid loss change. A key advantage of the Monge metric is its analytical efficiency; the Sherman-Morrison formula allows for a simple inverse, avoiding the heavy computational costs associated with other invariant methods like Fisher SAM. The resulting M-SAM perturbation is a "conservative" version of SAM's: $$\delta^*_{M-SAM} = \frac{1}{\sqrt{1 + \|\nabla \ell(\theta)\|^2}} \delta^*_{SAM}$$ In steep regions (large gradients), M-SAM automatically reduces the perturbation size, behaving more like standard Gradient Descent. In flatter regions, it recovers the behavior of SAM.
Key Insights
- Conservative Perturbations: Standard SAM's fixed-radius perturbation can "explode" in steep terrain, pushing models toward saddle points. M-SAM’s gradient-inverse scaling prevents this instability.
- Loss Geometry vs. Probability: Unlike Fisher SAM, which relies on Information Geometry and probabilistic assumptions, M-SAM depends solely on the loss geometry. This makes it universally applicable to any loss function.
- Hyperparameter Robustness: M-SAM is significantly harder to "break" during grid searches, maintaining stability where standard SAM might diverge.
Key Results
The efficacy of M-SAM was validated across toy manifolds, CIFAR-10 fine-tuning, and large-scale CLIP representational alignment tasks.
- Saddle Point Avoidance: In 2D experiments, M-SAM reached global minima from starting points where SAM remained stuck at the origin.
- Superior Alignment: In CLIP tasks, M-SAM achieved a mutual kNN similarity score ($S_{kNN}$) of 0.446, outperforming SAM (0.405) and Adam (0.388).
- Divergence Resistance: M-SAM demonstrated remarkable stability during fine-tuning. While SAM's loss exploded to ~4.85 under high learning rates, M-SAM remained stable at ~0.58.
Limitations and Future Work
While M-SAM offers significant improvements, it currently relies on a first-order Taylor expansion. Future research could explore incorporating higher-order curvature (the Hessian) for even more precise landscape navigation. Additionally, while empirical results on mini-batches are promising, further theoretical analysis of M-SAM in purely stochastic settings is warranted.
Summary updated