NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models

Chenlu Guo, Yi Chang, Yuan Wu

TL;DR

NLoRA introduces a more efficient way to fine-tune Large Language Models (LLMs) by moving from LoRA’s two-matrix structure to a three-matrix "Structured LoRA" (SLoRA) initialized via the Nyström method. This approach achieves significantly higher accuracy on reasoning and coding tasks than standard LoRA while being orders of magnitude faster to initialize than SVD-based alternatives like PiSSA.

What Problem Does This Paper Address?

Fine-tuning LLMs is notoriously resource-heavy. Low-Rank Adaptation (LoRA) solved this by freezing the main model and only training two small "adapter" matrices ($A$ and $B$). However, LoRA has two main drawbacks:

  1. Slow Convergence: It takes a long time to reach peak performance.
  2. Expensive Initialization: Recent variants like PiSSA try to speed up convergence by using Singular Value Decomposition (SVD) to initialize the matrices. While effective, SVD is computationally expensive ($O(mn^2)$), making it a bottleneck for very large weight matrices.

The authors sought a way to get the performance boost of smart initialization without the heavy computational tax of SVD.

Methods

The paper introduces three nested innovations:

  1. StructuredLoRA (SLoRA): Instead of the standard $\Delta W = AB$ (two matrices), the authors insert a small $r \times r$ intermediate matrix $N$, resulting in $\Delta W = ANB$. This "sandwich" structure increases the model's expressive power with negligible parameter overhead.
  2. NyströmLoRA (NLoRA): To initialize the $A, N,$ and $B$ matrices, they use the Nyström method. This technique approximates a large matrix by sampling a subset of its rows and columns. It reduces the initialization complexity to $O(mr + r^2 + rn)$, which is much faster than SVD.
  3. IntermediateTune (IntTune): A "minimalist" fine-tuning strategy where $A$ and $B$ are frozen after Nyström initialization, and only the tiny intermediate matrix $N$ is trained.

Datasets

The researchers evaluated their methods across a broad spectrum of tasks:

  • Natural Language Generation (NLG): GSM8K and MATH (mathematical reasoning); HumanEval and MBPP (coding); and MT-Bench (dialogue/instruction following).
  • Natural Language Understanding (NLU): The GLUE benchmark, including tasks like MNLI (inference), SST-2 (sentiment), and CoLA (linguistic acceptability).
  • Models: They tested on LLaMA 2-7B, Mistral-7B, DeBERTa-v3-base, and RoBERTa-large.

Key Insights and Ideas

  • The Efficiency of Sampling: The "aha moment" is that you don't need a perfect SVD to initialize adapters. The Nyström method’s approximation is "good enough" to capture the essential structure of the pre-trained weights, allowing the model to start training from a much better vantage point than random initialization.
  • The Centrality of the Intermediate Matrix: IntTune reveals a fascinating property: for many generation tasks, the "directions" of adaptation (captured in $A$ and $B$) can be fixed early on. Simply tuning how these directions interact (via $N$) is enough to outperform standard LoRA.
  • Task-Specific Sensitivity: The authors found that NLG tasks (like coding and math) benefit more from this high-level "central" tuning, whereas NLU tasks (like sentiment analysis) still require the finer-grained control provided by training the larger $A$ and $B$ matrices.

Key Results

  • Massive Gains in Reasoning: On the GSM8K math benchmark, NLoRA achieved 57.70% accuracy, a staggering improvement over LoRA’s 42.30%, using nearly the same number of parameters.
  • Initialization Speedup: For a rank-8 adapter, NLoRA initializes in just 7.27 seconds, compared to 7723 seconds (over 2 hours) for PiSSA’s SVD-based approach.
  • Extreme Parameter Efficiency: IntTune (tuning only the $N$ matrix) used only 1.25% of LoRA's trainable parameters (4M vs 320M) yet still boosted average NLG performance by 7.45% over the LoRA baseline.
  • Memory Savings: IntTune reduced GPU memory allocation by approximately 10% compared to LoRA during training.

Limitations and Future Work

  • NLU Performance Gap: While NLoRA performs well on NLU, the "minimalist" IntTune approach struggles to match LoRA on these tasks. The authors suggest NLU requires capturing low-level token semantics that frozen $A$ and $B$ matrices might miss.
  • Modality Expansion: The current study is limited to text. The authors suggest that extending this Nyström-based approach to vision or multi-modal tasks is a promising next step.
  • Hybrid Approaches: There is potential to combine NLoRA with other recent LoRA improvements, such as DoRA (Weight-Decomposed Low-Rank Adaptation), to further push the boundaries of PEFT.

Summary updated

Abstract

Parameter-efficient fine-tuning (PEFT) is essential for adapting large language models (LLMs), with low rank adaptation (LoRA) being the most popular approach.However, LoRA suffers from slow convergence, and some recent LoRA variants, such as PiSSA, primarily rely on Singular Value Decomposition (SVD) for initialization, leading to expensive computation.To mitigate these problems, we resort to Nystrm method, which follows a three-matrix manipulation.Therefore, we first introduce StructuredLoRA (SLoRA), investigating to introduce a small intermediate matrix between the low-rank matrices A and B. Secondly, we propose NystrmLoRA (NLoRA), which leverages Nystrm-based initialization for SLoRA to improve its effectiveness and efficiency.Finally, we propose IntermediateTune (IntTune) to explore fine-tuning exclusively the intermediate matrix of NLoRA to furthermore boost LLMs' efficiency.We evaluate our methods on 5 natural language generation (NLG) tasks and 8 natural language understanding (NLU) tasks.On GSM8K, SLoRA and NLoRA achieve accuracies of 56.48% and 57.70%, surpassing LoRA by 33.52% and 36.41% with only 3.67M additional trainable parameters.IntTune boosts average NLG performance over LoRA by 7.45% while using only 1.25% of its parameters.These results demonstrate the efficiency and effectiveness of our approach in enhancing model performance with minimal parameter overhead.
BibTeX
@misc{guo2025nlora,
  author = {Chenlu Guo and Yi Chang and Yuan Wu},
  title = {{NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models}},
  year = {2025},
  eprint = {2502.14482},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.14482},
  howpublished = {arXiv},
}
Figure