Abstract
In this paper, we show that Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate for A and B does not allow efficient feature learning. We then show that this suboptimality of LoRA can be corrected simply by setting different learning rates for the LoRA adapter matrices A and B with a well-chosen ratio. We call this proposed algorithm LoRA+. In our extensive experiments, LoRA+ improves performance (1-2% improvements) and finetuning speed (up to ~2X SpeedUp), at the same computational cost as LoRA.
Introduction
Low-Rank Adaptation (LoRA) has become the industry standard for parameter-efficient fine-tuning (PEFT) of Large Language Models. By freezing the pre-trained weights and training two small, low-rank matrices ($A$ and $B$), LoRA significantly reduces the memory and computational requirements of adaptation. However, the original LoRA implementation applies a uniform learning rate to both adapter matrices.
This paper introduces LoRA+, an optimized approach that recognizes the distinct functional roles of the two adapter matrices. The authors demonstrate that as models grow wider—possessing larger embedding dimensions—the standard uniform learning rate becomes a bottleneck, preventing the model from reaching its full potential in feature learning.
The Scaling Problem in Large-Width Models
The core of the paper's argument lies in the "infinite-width" scaling theory, similar to the $\mu$P (Maximal Update Parameterization) framework. In standard LoRA, the update to the weight matrix is represented as $\Delta W = BA$. As the embedding dimension (width) of the model increases, the gradient updates for $A$ and $B$ do not scale in a way that promotes efficient feature learning.
Specifically, using the same learning rate for both matrices often results in "lazy training." In this regime, the model fails to adapt its internal representations effectively to the new task, relying too heavily on the pre-trained features. The authors prove that to achieve "efficient" feature learning in wide networks, the learning rate for the expansion matrix ($B$) must be significantly higher than that of the compression matrix ($A$).
The LoRA+ Algorithm
LoRA+ is a straightforward modification to the standard LoRA pipeline that decouples the learning rates for the two adapter matrices:
- Decoupled Learning Rates: Instead of a single $\eta$, the algorithm sets $\eta_A$ for matrix $A$ and $\eta_B$ for matrix $B$.
- The Ratio ($\lambda$): The relationship is defined as $\eta_B = \lambda \eta_A$, where $\lambda$ is a fixed ratio greater than 1.
- Implementation: This change requires zero additional memory or FLOPs. It is a purely algorithmic adjustment to the optimizer's update step.
For practitioners, this simplifies hyperparameter tuning. Rather than a complex 2D grid search for two independent learning rates, one can fix a recommended ratio (such as $\lambda=16$) and perform a standard 1D search for the base learning rate $\eta_A$.
Experimental Results
The researchers validated LoRA+ across a variety of architectures and benchmarks, including RoBERTa-base, GPT-2, and Llama-7b.
Performance Gains
LoRA+ consistently outperformed standard LoRA by 1% to 2% across various GLUE tasks, including MNLI, QQP, SST2, and QNLI. On the MMLU (Massive Multitask Language Understanding) benchmark, Llama-7b saw a 1.3% improvement when using decoupled learning rates compared to the best-tuned standard LoRA baseline.
Training Efficiency
One of the most significant advantages of LoRA+ is its convergence speed. The algorithm achieved peak accuracy up to 2x faster than standard LoRA. For instance, on the MNLI task with RoBERTa, LoRA+ reached the target accuracy in 10,000 steps, whereas standard LoRA required nearly 20,000 steps to achieve the same result.
Key Insights
- Architectural Asymmetry: Matrix $A$ compresses high-dimensional input into a low-rank bottleneck, while Matrix $B$ expands it back. Their geometric roles are fundamentally different, necessitating different step sizes for optimal updates.
- Task Difficulty: The benefits of LoRA+ are most pronounced on "hard" tasks that require substantial feature adaptation. On simpler tasks where the model can rely on pre-existing knowledge, the performance gap is smaller.
- Initialization Sensitivity: The optimal ratio $\lambda$ is influenced by the initialization strategy. Standard practice (initializing $B$ to zero) requires a different ratio than random initialization to maintain stability and efficiency.
Conclusion and Future Work
LoRA+ provides a "free lunch" for model fine-tuning, offering better performance and faster convergence without increasing the computational budget. As the industry moves toward even larger models with massive embedding dimensions, the suboptimality of standard LoRA will likely become more acute, making LoRA+ an essential tool for efficient adaptation. Future research may explore automated methods for determining the optimal ratio $\lambda$ for specific architectures and how these scaling laws interact with newer quantization techniques like QLoRA.
Summary updated