Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models

Fangzhao Zhang, Mert Pilanci

TL;DR

This paper introduces a Riemannian preconditioner for Low-Rank Adaptation (LoRA) that significantly stabilizes fine-tuning and eliminates the need for complex, separate learning rate schedules for the $A$ and $B$ matrices. By incorporating a computationally "minuscule" $r \times r$ matrix inversion into each gradient step, the authors achieve faster convergence, better performance on LLMs and diffusion models, and remarkable robustness to hyperparameter choices.

What Problem Does This Paper Address?

Low-Rank Adaptation (LoRA) has become the industry standard for Parameter-Efficient Fine-Tuning (PEFT). By freezing a model's weights and only training two small matrices ($A$ and $B$) whose product approximates the weight update, LoRA drastically reduces storage and memory requirements.

However, LoRA training is notoriously sensitive to hyperparameters. Recent theoretical work (like LoRA+) has shown that for stable "feature learning" in large models, the learning rate for matrix $B$ must be significantly larger than for matrix $A$ (often by a factor of $2^4$). This forces researchers into a tedious joint hyperparameter search. Furthermore, standard optimizers like AdamW don't account for the unique geometric structure of low-rank matrix factorization, leading to suboptimal convergence and potential training instability.

Methods

The authors propose Riemannian Preconditioned LoRA. Instead of treating $A$ and $B$ as flat vectors, they treat the optimization as occurring on a "quotient manifold" of low-rank matrices.

The core contribution is a modified gradient update (Equation 1 in the paper): $$A_{t+1} = A_t - \alpha(B_t^T B_t)^{-1}(\nabla_{A_t} \mathcal{L})$$ $$B_{t+1} = B_t - \alpha(\nabla_{B_t} \mathcal{L})(A_t A_t^T)^{-1}$$

Where $\mathcal{L}$ is the loss and $\alpha$ is the learning rate. By scaling the gradients with these $r \times r$ preconditioners, the update effectively acts as a gradient projector, aligning the LoRA update more closely with what a full-parameter fine-tuning step would look like.

Crucially, because the LoRA rank $r$ is typically very small (e.g., 4, 8, or 16), inverting these matrices adds virtually zero overhead to training time or memory. The authors implement this as "Scaled AdamW" and "Scaled GD," requiring only about four lines of code change in standard optimizer implementations.

Datasets

The authors evaluated their method across a diverse set of tasks:

  • Large Language Models (LLMs): Fine-tuning GPT-2 (Small and Medium) and Mistral 7B (4-bit quantized) on the E2E, WebNLG, DART, and GLUE benchmarks.
  • Diffusion Models: Fine-tuning Stable Diffusion V1.5 for object generation and the Mix-of-Show variant for high-quality face generation.

Key Insights and Ideas

  • Unifying Learning Rates: The preconditioner mathematically balances the training dynamics of $A$ and $B$. This allows researchers to use a single learning rate for both matrices while still achieving the "stable feature learning" that previously required separate, tuned ratios.
  • Geometric Awareness: By deriving the preconditioner from a Riemannian metric, the optimizer becomes "aware" of the matrix factorization structure. This leads to a convergence rate that is independent of the data's condition number.
  • Extreme Robustness: The method is remarkably resilient to high learning rates. In diffusion tasks, standard AdamW often collapses and produces "black images" when the learning rate is too high; Scaled AdamW continues to produce high-quality, prompt-adherent images under the same conditions.
Figure 1. Generation results for prompt “a blue ⟨Vvase⟩” after fine-tuning on 6 red vase images of the Stable Diffusion V1.5 model. No black images are observed for our method (scaled AdamW)’s generation and AdamW generates only black images for large learning rates. Our method generates photos better capturing the prompt and is more robust to learning rate changes. See Section 6.4.1 for experimental details.

Key Results

  • Performance Boost: Scaled AdamW consistently outperformed standard AdamW across all LLM evaluation metrics (BLEU, ROUGE-L, METEOR, etc.). For example, on the E2E challenge with GPT-2, Scaled AdamW improved the BLEU score from 68.9 to 69.6.
  • Negligible Overhead: Runtime comparisons (Figure 2) show that the scaled optimizers run just as fast as their unscaled counterparts, even as the number of training steps increases.
Figure 2. Runtime for LoRA fine-tuning GPT-2 medium model with different optimizers. Our scaled methods introduce negligible runtime overhead and train as fast as unscaled methods. See Section 6.2 for experimental details. Here we set r = 4.
  • Qualitative Superiority: In diffusion models, the method captured prompts more accurately (e.g., generating a blue vase when fine-tuned on red ones) and was far more robust to the "fusion coefficient" used when merging LoRA weights back into the base model.
Figure 3. Generation results for prompt “a pencil sketch of ⟨Vpotter⟩” by Mix-of-Show model with different optimizers and various learning rates. Our method (scaled AdamW) generates photos better capturing the prompt, i.e., a pencil sketch, and is more robust to learning rate choices. See Section 6.4.2 for experimental details.

Limitations and Future Work

  • Invertibility: The preconditioners $(A A^T)^{-1}$ and $(B^T B)^{-1}$ require the matrices to be non-singular. The authors handle this by adding a small damping term ($\delta I$), but the optimal choice of $\delta$ is a remaining hyperparameter.
  • Infinite-Width Assumption: The theoretical proof for stable feature learning relies on the infinite-width neural network regime. While empirical results on finite models are strong, extending the theory to finite-width settings is an open area.
  • Broader PEFT Application: The current work focuses on LoRA. Future research could explore whether similar Riemannian preconditioning can benefit other PEFT methods like DoRA (Weight-Decomposed Low-Rank Adaptation).

Summary updated

Abstract

Low-Rank Adaptation (LoRA) emerges as a popular parameter-efficient fine-tuning (PEFT) method, which proposes to freeze pretrained model weights and update an additive low-rank trainable matrix. In this work, we study the enhancement of LoRA training by introducing an $r \times r$ preconditioner in each gradient step where $r$ is the LoRA rank. We theoretically verify that the proposed preconditioner stabilizes feature learning with LoRA under infinite-width NN setting. Empirically, the implementation of this new preconditioner requires a small change to existing optimizer code and creates virtually minuscule storage and runtime overhead. Our experimental results with both large language models and text-to-image diffusion models show that with this new preconditioner, the convergence and reliability of SGD and AdamW can be significantly enhanced. Moreover, the training process becomes much more robust to hyperparameter choices such as learning rate. The new preconditioner can be derived from a novel Riemannian metric in low-rank matrix field. Code can be accessed at https://github.com/pilancilab/Riemannian_Preconditioned_LoRA.
BibTeX
@inproceedings{zhang2024riemannian,
  author = {Fangzhao Zhang and Mert Pilanci},
  title = {{Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=IwqE4QqBew},
}
Figure