RSAM: Learning on Manifolds with Riemannian Sharpness-Aware Minimization

Tuan Truong, Hoang-Phi Nguyen, Tung Pham, Minh-Tuan Tran, Mehrtash Harandi, Dinh Phung, Trung LeSubmission to ICLR 2024 · Withdrawn

TL;DR

Riemannian Sharpness-Aware Minimization (RSAM) extends flat-minima optimization to parameters constrained to Riemannian manifolds (such as Stiefel manifolds enforcing orthogonality). By restricting the sharpness perturbation to the manifold’s intrinsic geometry, RSAM achieves a tighter theoretical generalization bound ($\mathcal{O}(\sqrt{d})$ rather than Euclidean SAM's $\mathcal{O}(\sqrt{k})$) and yields substantial performance improvements in supervised and self-supervised vision tasks.

What Problem Does This Paper Address?

Deep neural networks are prone to overfitting due to complex, high-dimensional, non-convex loss landscapes. Sharpness-Aware Minimization (SAM) mitigates this by seeking flat minima—optimizing for parameters whose local neighborhood also has low loss.

However, modern deep learning frequently imposes structural parameter constraints, such as orthogonality, full rank, or symmetric positive definiteness (SPD). Under these constraints, parameters lie on a curved sub-manifold $\mathcal{M}$ (with intrinsic dimension $d$) embedded in a higher-dimensional Euclidean space $\mathbb{R}^k$ ($d \ll k$). Standard SAM does not account for this geometry: it either perturbs weights off the manifold into invalid parameter regimes or relies on soft penalty regularizers that are brittle, difficult to tune, and fail to guarantee true constraint satisfaction.

Methods

The authors introduce RSAM, generalizing SAM’s minimax formulation to Riemannian manifolds:

  1. Manifold Sharpness Definition: The perturbation neighborhood is defined on the manifold via a retraction mapping $R_\theta$ from a $\rho$-ball on the tangent space $T_\theta \mathcal{M}$: $$B_\theta(\rho; \mathcal{M}) = R_\theta(\{ \epsilon \in T_\theta\mathcal{M} : \|\epsilon\|_2 \le \rho \})$$
  2. Tangent Perturbation & Projection: Directly solving the inner worst-case perturbation on the tangent space using an orthonormal basis is computationally prohibitive ($\mathcal{O}(p^2)$ scaling and nested loops). Instead, RSAM solves a relaxed perturbation problem in the local Riemannian metric space: $$\bar{\epsilon} = \rho \frac{\operatorname{grad}_\theta(\mathcal{L}_B(\theta))^\top D_\theta}{\|\operatorname{grad}_\theta(\mathcal{L}_B(\theta))^\top D_\theta\|_2}$$ and projects it onto the tangent space: $\epsilon^* = \operatorname{Proj}_\theta(\bar{\epsilon})$.

  3. Riemannian Optimization Step:

  4. Ascend step: Retract the perturbed tangent vector onto the manifold: $\theta^* = R_\theta(\epsilon^*)$.
  5. Descend step: Compute the Riemannian gradient at the perturbed point and update via retraction: $\theta_{t+1} = R_{\theta_t}(-\eta \operatorname{grad}_\theta \mathcal{L}_B(\theta^*))$.
  6. Concrete Applications on Stiefel Manifolds ($St(p, n)$):
  7. Orthogonal CNNs: Ensuring convolutional filter banks remain strictly orthonormal ($W_\ell^\top W_\ell = I_d$) throughout training without penalty terms.
  8. Metric Learning for Supervised Contrastive Learning (SupCon): Introducing an R-Stiefel layer (projecting representations via $U S$, where $U$ lies on the Stiefel manifold and $S$ is diagonal) to model learnable Mahalanobis distances.
       Tangent Space T_θ M
          [ ε* = Proj_θ(ε) ]
                 |
      Retraction | R_θ(ε*)
                 v
  Manifold M: [ θ ] ------------> [ θ* ]  (Ascend to local worst case)

Datasets

The authors evaluate RSAM across standard image classification and fine-grained benchmarks:

  • CIFAR-10 and CIFAR-100
  • FGVC-Aircraft (fine-grained aircraft classification)
  • MNIST (used in an ablation study for autoencoder orthogonal representation learning)

Note: No new datasets were introduced.

Key Insights and Ideas

  • Intrinsic Dimension Tightens Generalization: Theoretical bounds on the generalization gap for standard SAM scale with the ambient Euclidean dimension $\mathcal{O}(\sqrt{k})$. By confining the perturbation to the manifold $\mathcal{M}$, RSAM bounds generalization error by the intrinsic dimension $\mathcal{O}(\sqrt{d})$, which is dramatically smaller ($d \ll k$).
  • Projection Approximation Bypasses Manifold Basis Construction: Computing an orthonormal basis of the tangent space for every step scales poorly and is not readily available for arbitrary manifolds. The authors prove that finding an unconstrained directional step and projecting it onto the tangent space matches the exact closed-form solution's performance while running ~1.75× faster.
  • Hard Constraints Outperform Soft Regularizers: In an autoencoder ablation, regularizing orthogonality with a penalty term ($\lambda \|W^\top W - I\|_F^2$) created extreme sensitivity: smaller $\lambda$ caused severe orthogonality violations, while larger $\lambda$ led to sharper minima. RSAM eliminates this trade-off by strictly staying on the manifold, achieving lower loss and lower sharpness.

Key Results

  • Supervised Learning (ResNet-34 & ResNet-50): Imposing orthogonality on a single middle convolutional layer via RSAM consistently outperformed standard SGD and SAM:
  • CIFAR-100 (ResNet-50): 77.78% (RSAM) vs. 75.04% (SAM) and 74.62% (SGD).
  • CIFAR-10 (ResNet-50): 96.32% (RSAM) vs. 95.39% (SAM) and 94.56% (SGD).
  • FGVC-Aircraft (ResNet-50): 84.68% (RSAM) vs. 83.01% (SAM) and 82.44% (SGD).
  • Self-Supervised Learning (SupCon):
  • Linear evaluation accuracy on CIFAR-100 reached 81.62% with ResNet-50—outperforming SupCon + SAM (76.73%) and SupCon + SGD (75.29%) by nearly 5–6%.
  • Computational Overhead: RSAM requires only about ~6% additional wall-clock runtime per epoch over Euclidean SAM (e.g., 52.6s vs. 49.1s on CIFAR-100 with ResNet-34).
  • Loss Landscape Geometry: Hessian analysis on CIFAR-100 confirmed that RSAM finds flatter minima than SAM, exhibiting a smaller top Hessian eigenvalue (10.86 vs. 13.32) and a more concentrated spectral distribution.

Limitations and Future Work

  • Double Forward-Backward Overhead: Like standard SAM, RSAM requires two forward-backward computations per iteration, rendering it approximately $2\times$ slower than basic SGD.
  • Limited Manifold Exploration: Although motivated by multiple manifold classes (e.g., SPD, Grassmann), the empirical experiments focus almost exclusively on Stiefel manifolds (orthogonal convolutional weights and contrastive projection heads).
  • Partial Architecture Application: In the supervised experiments, orthogonality was enforced on only a single convolutional layer in the middle of the network rather than across all layers. Extending Riemannian sharpness optimization to entire multi-layer networks across diverse manifolds remains an open direction.

Summary updated

Abstract

Nowadays, understanding the geometry of the loss landscape shows promise in enhancing a model's generalization ability. In this work, we draw upon prior works that apply geometric principles to optimization and present a novel approach to improve robustness and generalization ability for constrained optimization problems. Indeed, this paper aims to generalize the Sharpness-Aware Minimization (SAM) optimizer to Riemannian manifolds. In doing so, we first extend the concept of sharpness and introduce a novel notion of sharpness on manifolds. To support this notion of sharpness, we present a theoretical analysis characterizing generalization capabilities with respect to manifold sharpness, which demonstrates a tighter bound on the generalization gap, a result not known before. Motivated by this analysis, we introduce our algorithm, Riemannian Sharpness-Aware Minimization (RSAM). To demonstrate RSAM's ability to enhance generalization ability, we evaluate and contrast our algorithm on a broad set of problems, such as image classification and contrastive learning across different datasets, including CIFAR100, CIFAR10, and FGVCAircraft. Our code is publicly available at \url{https://t.ly/RiemannianSAM}.
BibTeX
@misc{truong2023rsam,
  author = {Tuan Truong and Hoang-Phi Nguyen and Tung Pham and Minh-Tuan Tran and Mehrtash Harandi and Dinh Phung and Trung Le},
  title = {{RSAM: Learning on Manifolds with Riemannian Sharpness-Aware Minimization}},
  year = {2023},
  eprint = {2309.17215},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.17215},
}

Loading related content...

Figure