TL;DR
This paper introduces a Riemannian preconditioner for Low-Rank Adaptation (LoRA) that significantly stabilizes fine-tuning and eliminates the need for complex, separate learning rate schedules for the $A$ and $B$ matrices. By incorporating a computationally "minuscule" $r \times r$ matrix inversion into each gradient step, the authors achieve faster convergence, better performance on LLMs and diffusion models, and remarkable robustness to hyperparameter choices.
What Problem Does This Paper Address?
Low-Rank Adaptation (LoRA) has become the industry standard for Parameter-Efficient Fine-Tuning (PEFT). By freezing a model's weights and only training two small matrices ($A$ and $B$) whose product approximates the weight update, LoRA drastically reduces storage and memory requirements.
However, LoRA training is notoriously sensitive to hyperparameters. Recent theoretical work (like LoRA+) has shown that for stable "feature learning" in large models, the learning rate for matrix $B$ must be significantly larger than for matrix $A$ (often by a factor of $2^4$). This forces researchers into a tedious joint hyperparameter search. Furthermore, standard optimizers like AdamW don't account for the unique geometric structure of low-rank matrix factorization, leading to suboptimal convergence and potential training instability.
Methods
The authors propose Riemannian Preconditioned LoRA. Instead of treating $A$ and $B$ as flat vectors, they treat the optimization as occurring on a "quotient manifold" of low-rank matrices.
The core contribution is a modified gradient update (Equation 1 in the paper): $$A_{t+1} = A_t - \alpha(B_t^T B_t)^{-1}(\nabla_{A_t} \mathcal{L})$$ $$B_{t+1} = B_t - \alpha(\nabla_{B_t} \mathcal{L})(A_t A_t^T)^{-1}$$
Where $\mathcal{L}$ is the loss and $\alpha$ is the learning rate. By scaling the gradients with these $r \times r$ preconditioners, the update effectively acts as a gradient projector, aligning the LoRA update more closely with what a full-parameter fine-tuning step would look like.
Crucially, because the LoRA rank $r$ is typically very small (e.g., 4, 8, or 16), inverting these matrices adds virtually zero overhead to training time or memory. The authors implement this as "Scaled AdamW" and "Scaled GD," requiring only about four lines of code change in standard optimizer implementations.
Datasets
The authors evaluated their method across a diverse set of tasks:
- Large Language Models (LLMs): Fine-tuning GPT-2 (Small and Medium) and Mistral 7B (4-bit quantized) on the E2E, WebNLG, DART, and GLUE benchmarks.
- Diffusion Models: Fine-tuning Stable Diffusion V1.5 for object generation and the Mix-of-Show variant for high-quality face generation.
Key Insights and Ideas
- Unifying Learning Rates: The preconditioner mathematically balances the training dynamics of $A$ and $B$. This allows researchers to use a single learning rate for both matrices while still achieving the "stable feature learning" that previously required separate, tuned ratios.
- Geometric Awareness: By deriving the preconditioner from a Riemannian metric, the optimizer becomes "aware" of the matrix factorization structure. This leads to a convergence rate that is independent of the data's condition number.
- Extreme Robustness: The method is remarkably resilient to high learning rates. In diffusion tasks, standard AdamW often collapses and produces "black images" when the learning rate is too high; Scaled AdamW continues to produce high-quality, prompt-adherent images under the same conditions.
Key Results
- Performance Boost: Scaled AdamW consistently outperformed standard AdamW across all LLM evaluation metrics (BLEU, ROUGE-L, METEOR, etc.). For example, on the E2E challenge with GPT-2, Scaled AdamW improved the BLEU score from 68.9 to 69.6.
- Negligible Overhead: Runtime comparisons (Figure 2) show that the scaled optimizers run just as fast as their unscaled counterparts, even as the number of training steps increases.
- Qualitative Superiority: In diffusion models, the method captured prompts more accurately (e.g., generating a blue vase when fine-tuned on red ones) and was far more robust to the "fusion coefficient" used when merging LoRA weights back into the base model.
Limitations and Future Work
- Invertibility: The preconditioners $(A A^T)^{-1}$ and $(B^T B)^{-1}$ require the matrices to be non-singular. The authors handle this by adding a small damping term ($\delta I$), but the optimal choice of $\delta$ is a remaining hyperparameter.
- Infinite-Width Assumption: The theoretical proof for stable feature learning relies on the infinite-width neural network regime. While empirical results on finite models are strong, extending the theory to finite-width settings is an open area.
- Broader PEFT Application: The current work focuses on LoRA. Future research could explore whether similar Riemannian preconditioning can benefit other PEFT methods like DoRA (Weight-Decomposed Low-Rank Adaptation).
Summary updated