Monge SAM: Robust Reparameterization-Invariant Sharpness-Aware Minimization Based on Loss Geometry

Albert Kjøller Jacobsen, Georgios ArvanitidisarXiv 2025 · DOI · arXiv

TL;DR

Monge SAM (M-SAM) is a novel optimization algorithm that achieves reparameterization invariance in Sharpness-Aware Minimization (SAM) by leveraging the Monge metric of the loss surface. This "conservative" approach significantly enhances robustness to hyperparameters and prevents the optimizer from being trapped by suboptimal saddle points or local maxima.

The Problem: Parameter Sensitivity in SAM

Sharpness-Aware Minimization (SAM) has gained popularity for its ability to find "flat" minima, which typically generalize better to unseen data. It achieves this by seeking points with low loss even under small adversarial perturbations. However, SAM relies on the Euclidean metric, making it sensitive to how a model is parameterized. Simple weight rescaling—common in networks with Batch Normalization or ReLU—can change the measured "sharpness" without altering the underlying function. This lack of invariance makes SAM sensitive to hyperparameter tuning and prone to attraction by saddle points or local maxima.

Methods: Geometry-Aware Optimization

To address these flaws, the authors introduce Monge SAM (M-SAM). Rather than treating the parameter space as a flat Euclidean plane, M-SAM treats it as a Riemannian manifold where distance is defined by the Monge metric: $G(\theta) = I + \nabla \ell(\theta) \nabla \ell(\theta)^\top$.

Comparison of optimization trajectories: Monge SAM (red) avoids the saddle point that traps standard SAM (blue).

This metric effectively "stretches" the space in directions of rapid loss change. A key advantage of the Monge metric is its analytical efficiency; the Sherman-Morrison formula allows for a simple inverse, avoiding the heavy computational costs associated with other invariant methods like Fisher SAM. The resulting M-SAM perturbation is a "conservative" version of SAM's: $$\delta^*_{M-SAM} = \frac{1}{\sqrt{1 + \|\nabla \ell(\theta)\|^2}} \delta^*_{SAM}$$ In steep regions (large gradients), M-SAM automatically reduces the perturbation size, behaving more like standard Gradient Descent. In flatter regions, it recovers the behavior of SAM.

Key Insights

  • Conservative Perturbations: Standard SAM's fixed-radius perturbation can "explode" in steep terrain, pushing models toward saddle points. M-SAM’s gradient-inverse scaling prevents this instability.
  • Loss Geometry vs. Probability: Unlike Fisher SAM, which relies on Information Geometry and probabilistic assumptions, M-SAM depends solely on the loss geometry. This makes it universally applicable to any loss function.
  • Hyperparameter Robustness: M-SAM is significantly harder to "break" during grid searches, maintaining stability where standard SAM might diverge.

Key Results

The efficacy of M-SAM was validated across toy manifolds, CIFAR-10 fine-tuning, and large-scale CLIP representational alignment tasks.

Stability comparison: M-SAM (red) converges more reliably than SAM (blue) across different learning rates.
  • Saddle Point Avoidance: In 2D experiments, M-SAM reached global minima from starting points where SAM remained stuck at the origin.
  • Superior Alignment: In CLIP tasks, M-SAM achieved a mutual kNN similarity score ($S_{kNN}$) of 0.446, outperforming SAM (0.405) and Adam (0.388).
  • Divergence Resistance: M-SAM demonstrated remarkable stability during fine-tuning. While SAM's loss exploded to ~4.85 under high learning rates, M-SAM remained stable at ~0.58.

Limitations and Future Work

While M-SAM offers significant improvements, it currently relies on a first-order Taylor expansion. Future research could explore incorporating higher-order curvature (the Hessian) for even more precise landscape navigation. Additionally, while empirical results on mini-batches are promising, further theoretical analysis of M-SAM in purely stochastic settings is warranted.

Summary updated

Abstract

Recent studies on deep neural networks show that flat minima of the loss landscape correlate with improved generalization. Sharpness-aware minimization (SAM) efficiently finds flat regions by updating the parameters according to the gradient at an adversarial perturbation. The perturbation depends on the Euclidean metric, making SAM non-invariant under reparametrizations, which blurs sharpness and generalization. We propose Monge SAM (M-SAM), a reparametrization invariant version of SAM by considering a Riemannian metric in the parameter space induced naturally by the loss surface. Compared to previous approaches, M-SAM works under any modeling choice, relies only on mild assumptions while being as computationally efficient as SAM. We theoretically argue that M-SAM varies between SAM and gradient descent (GD), which increases robustness to hyperparameter selection and reduces attraction to suboptimal equilibria like saddle points. We demonstrate this behavior both theoretically and empirically on a multi-modal representation alignment task.
BibTeX
@misc{jacobsen2025monge,
  author = {Albert Kjøller Jacobsen and Georgios Arvanitidis},
  title = {{Monge SAM: Robust Reparameterization-Invariant Sharpness-Aware Minimization Based on Loss Geometry}},
  year = {2025},
  eprint = {2502.08448},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.08448},
}
Figure