Riemannian Optimization for LoRA on the Stiefel Manifold

JuneYoung Park, Minjae Kang, Seongbae Lee, Haegang Lee, Seongwan Kim, Jaeho LeeEMNLP 2025

TL;DR

This paper introduces Stiefel-LoRA, a novel method for fine-tuning large language models (LLMs) that optimizes LoRA's low-rank update matrices on the Stiefel manifold. By enforcing orthogonality constraints, Stiefel-LoRA achieves superior performance and parameter efficiency compared to standard LoRA with AdamW, unlocking LoRA's full potential.

What Problem Does This Paper Address?

Large Language Models (LLMs) are incredibly powerful but computationally expensive to fine-tune. Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA offer a solution by reducing the number of trainable parameters. LoRA achieves this by representing weight updates as a product of two low-rank matrices, A and B. However, standard optimization of these matrices using algorithms like AdamW in Euclidean space can lead to inefficiencies, such as redundant or unstable update directions in matrix B. This underutilizes the model's representational capacity, potentially requiring higher ranks (more parameters) or resulting in suboptimal performance. The paper addresses the question of whether applying geometric constraints, specifically orthogonality, to LoRA's update matrices can unlock greater efficiency and performance.

Methods

The core innovation is Stiefel-LoRA, which applies an orthogonality constraint to the LoRA update matrix B. This constraint forces the columns of B to be orthonormal, meaning B resides on the Stiefel manifold. This geometric approach ensures that the low-rank update effectively spans an r-dimensional subspace, maximizing its representational capacity.

The optimization process involves:

  1. Euclidean Gradient Calculation: Computing standard gradients for both matrices A and B.
  2. Manifold Projection: Projecting the Euclidean gradient of B onto the tangent space of the Stiefel manifold.
  3. Retraction: Using a QR decomposition-based retraction to update B, ensuring it remains on the Stiefel manifold.
  4. Momentum Integration: Adapting momentum-based optimizers like Adam to work with the Stiefel manifold by first computing the Euclidean momentum update and then projecting it onto the tangent space.

Datasets

The paper evaluates Stiefel-LoRA on several benchmark datasets across different NLP tasks:

  • Commonsense Reasoning: BoolQ, PIQA, SIQA, HellaSwag, ARC-e, ARC-c, OBQA.
  • Reading Comprehension: SQuAD, QuAC.
  • Mathematics: GSM8K, MATH.

The models used for evaluation are from the LLaMA family: LLaMA-3.2-1B, LLaMA-3.2-3B, and LLaMA3-8B.

Key Insights and Ideas

  • Orthogonality is Key: The paper strongly suggests that the limitations of standard LoRA with AdamW stem from the lack of explicit geometric constraints. Enforcing orthogonality on the LoRA update matrix B is crucial for maximizing its effectiveness.
  • Stiefel Manifold for Efficiency: Optimizing B on the Stiefel manifold ensures that its columns are linearly independent, leading to a full effective rank equal to the nominal rank. This prevents redundancy and maximizes the utilization of the low-rank approximation.
  • Complementary to DoRA: Stiefel-LoRA shows strong performance when combined with DoRA (Weight-Decomposed Low-Rank Adaptation), indicating that geometric constraints and weight decomposition can be complementary.
  • Importance of Co-adaptation: When matrix A is randomly initialized and fixed, Stiefel-LoRA's performance advantage diminishes. This highlights that for Stiefel-LoRA to shine, both A and B need to co-adapt, with A providing meaningful input for B to refine.

Key Results

  • Superior Performance: Stiefel-LoRA consistently outperforms standard LoRA with AdamW across all tested commonsense reasoning, reading comprehension, and mathematics benchmarks, for all model scales.
  • Enhanced Parameter Efficiency: The effective rank analysis shows that Stiefel-LoRA consistently utilizes the full specified rank (e.g., 16 dimensions), whereas AdamW-optimized LoRA often uses fewer dimensions on average (e.g., 12). This means Stiefel-LoRA achieves better results with the same or fewer parameters.
  • Faster Convergence: While not explicitly quantified with numbers in the main text, the paper implies faster convergence due to more stable and efficient optimization dynamics.
  • Specific Gains: For complex reasoning tasks like ARC-c and HellaSwag, Stiefel-LoRA shows particularly pronounced improvements. For larger models on the MATH dataset, the gains are also significant.

Limitations and Future Work

  • Model Scope: The study primarily used LLaMA base models. Future work could explore instruction-tuned models, which are more prevalent in practical LLM applications.
  • Qualitative Analysis: The paper lacks qualitative analysis of generated text.
  • Adaptive Rank Allocation: Investigating adaptive rank allocation methodologies is suggested for future research.
  • Independent Basis Vectors: Further exploration of the resulting independent basis vectors from the orthogonal constraints is also proposed.

Summary updated

BibTeX
@inproceedings{park2025riemannian,
  author = {JuneYoung Park and Minjae Kang and Seongbae Lee and Haegang Lee and Seongwan Kim and Jaeho Lee},
  title = {{Riemannian Optimization for LoRA on the Stiefel Manifold}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure