Riemannian SAM: Sharpness-Aware Minimization on Riemannian Manifolds

Jihun Yun, Eunho YangNeurIPS 2023 · OpenReview

Executive Summary

Sharpness-Aware Minimization (SAM) has become a go-to technique for improving model generalization by seeking out "flat" minima in the loss landscape. This paper introduces Riemannian SAM (RiemSAM), extending this powerful optimization strategy to non-Euclidean geometries like hyperbolic and orthogonal manifolds. The authors provide both a rigorous convergence analysis and state-of-the-art results in knowledge graph completion and machine translation.

The Challenge of Non-Euclidean Optimization

Deep learning is increasingly moving beyond flat Euclidean space. Geometric Deep Learning (GDL) is now widely used to model hierarchical data in hyperbolic space or to ensure numerical stability using orthogonal neural networks (Stiefel manifolds).

While Riemannian versions of standard optimizers (like R-SGD and R-Adam) exist, the field has lacked a way to apply "sharpness-aware" optimization to these curved spaces. In Euclidean space, SAM works by perturbing weights to find a region where the loss is consistently low. However, you cannot simply "add" a perturbation to a point on a manifold—doing so would cause the point to fall off the surface. This paper bridges that gap, defining how to measure and minimize sharpness while strictly respecting the underlying geometry of the data.

Formulating Riemannian SAM

The authors formulate Riemannian SAM by shifting the perturbation step into the tangent space of the manifold. Key components of their approach include:

  1. Tangent Space Perturbation: Instead of perturbing the model parameters directly, the algorithm finds the steepest ascent direction within the tangent space (a local flat approximation of the manifold).
  2. Retractions: To move from the tangent space back onto the manifold, they use "retractions"—computationally efficient generalizations of exponential maps.
  3. Vector Transport: After calculating the gradient at a perturbed point, they use "vector transport" to move that gradient back to the original point's tangent space so a valid descent step can be taken.
  4. Lorentz SAM: They derive a specific, novel instantiation for the Lorentz model (a type of hyperbolic geometry), which is particularly effective for hierarchical data.
  5. Unification: They demonstrate that Fisher SAM (a previously proposed Euclidean variant) is actually a special case of Riemannian SAM acting on a statistical manifold.

Key Insights and Theoretical Contributions

  • Alignment in Preconditioned Space: A major theoretical contribution is the insight that for SAM to converge on a manifold, the gradients do not need to align in Euclidean space. Instead, they must align within the preconditioned space defined by the Riemannian metric. As illustrated in their experiments, while Euclidean gradients might diverge, the Riemannian ones stay closely aligned.
Gradient Alignment
  • Theoretical Gap Bridged: By providing a convergence proof for the general Riemannian framework, the authors also establish the first formal convergence guarantees for existing variants like Fisher SAM.
  • Generalization over Accuracy: Like its Euclidean counterpart, RiemSAM aims for the "flattest" error rather than just the lowest training error, consistently leading to better performance on unseen test data.

Empirical Results

The authors evaluated RiemSAM on two primary tasks: Knowledge Graph Completion (using WN18RR and FB15k-237 datasets with a HYBONET architecture) and Machine Translation (using IWSLT ’14 and WMT ’14 benchmarks with a Lorentz Transformer).

  • Superior Generalization: In Knowledge Graph Completion, RiemSAM consistently outperformed Riemannian Adam (rAdam). For example, on WN18RR with 32 dimensions, the Mean Reciprocal Rank (MRR) improved from 48.8 to 49.3.
  • State-of-the-Art Hyperbolic Translation: The Lorentz Transformer trained with RiemSAM achieved a BLEU score of 20.1 on WMT ’14, significantly outperforming both Euclidean Transformers (17.0) and the same hyperbolic model trained with rAdam (19.3).
  • Flatter Minima: Empirical metrics confirm that RiemSAM successfully finds flatter minima on the manifold compared to Euclidean SAM, as evidenced by a lower Hessian trace and reduced manifold-aware sharpness.
Hessian Trace and Sharpness
  • Efficiency: While RiemSAM requires two gradient computations per iteration, the "wall-clock" time was only about 1.6x to 1.8x longer than standard rAdam, a trade-off often considered acceptable for the resulting gains in generalization.

Limitations and Future Work

The primary limitation of RiemSAM is the computational overhead of the double-gradient update. The authors suggest exploring "Efficient SAM" techniques, such as stochastic weight perturbations, to mitigate this cost in the future. Additionally, while they proved convergence, deriving formal generalization error bounds specifically for Riemannian manifolds remains an open area for research. Finally, testing RiemSAM on a wider variety of manifolds (like Grassmannians) could further demonstrate its versatility beyond the hyperbolic spaces heavily featured in this work.

Summary updated

Abstract

Contemporary advances in the field of deep learning have embarked upon an exploration of the underlying geometric properties of data, thus encouraging the investigation of techniques that consider general manifolds, for example, hyperbolic or orthogonal neural networks. However, the optimization algorithms for training such geometric deep learning models still remain highly under-explored. In this paper, we introduce Riemannian SAM by generalizing conventional Euclidean SAM to Riemannian manifolds. We successfully formulate the sharpness-aware minimization on Riemannian manifolds, leading to one of a novel instantiation, Lorentz SAM. In addition, SAM variants proposed in previous studies such as Fisher SAM can be derived as special examples under our Riemannian SAM framework. We provide the convergence analysis of Riemannian SAM under a less aggressively decaying ascent learning rate than Euclidean SAM. Our analysis serves as a theoretically sound contribution encompassing a diverse range of manifolds, also providing the guarantees for SAM variants such as Fisher SAM, whose convergence analyses are absent. Lastly, we illustrate the superiority of Riemannian SAM in terms of generalization over previous Riemannian optimization algorithms through experiments on knowledge graph completion and machine translation tasks.
BibTeX
@inproceedings{yun2023riemannian,
  author = {Jihun Yun and Eunho Yang},
  title = {{Riemannian SAM: Sharpness-Aware Minimization on Riemannian Manifolds}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=strvrjSi3C},
}
Figure