High-Rank Updates for Parameter-Efficient Fine-Tuning

This direction is merged into another direction: low-rank expressivity bottleneck (low-rank representation bottleneck). Its last version stays readable and citable here.

Version 1 of 130 papers49 conceptsfrozen 2026-10-04

Standard Parameter-Efficient Fine-Tuning (PEFT), exemplified by LoRA, relies on a fundamental hypothesis: weight updates in large models reside in a "low-rank" subspace. By representing an update $\Delta W$ as the product of two thin matrices $A$ and $B$ ($W = W_0 + AB$), we can tune millions of parameters by updating only a few thousand.

However, recent research suggests this low-rank constraint is a bottleneck. While efficient, it limits the model's ability to learn complex, high-frequency features or store novel information during continual pretraining. A new direction in PEFT seeks to decouple parameter efficiency from rank deficiency, enabling high-rank or even full-rank updates without increasing the trainable parameter count.

The Multiplicative Power of Kronecker Products

The most prominent strategy for breaking the rank bottleneck is the use of Kronecker products. In traditional LoRA, the rank of the update is limited by the inner dimension $r$ of the two matrices. In contrast, a Kronecker-based update represents $\Delta W$ as $A \otimes B$.

As Xuehai He and colleagues demonstrate in KAdaptation, the rank of a Kronecker product is the product of the ranks of its factors ($rank(A) \times rank(B)$), while the number of parameters is merely the sum of the parameters in $A$ and $B$. This allows for high-rank expressiveness with minimal overhead. Shyam Marjit applies this in DiffuseKronA to capture the intricate spatial details required for personalizing diffusion models, which standard low-rank methods often blur.

By summing multiple Kronecker products, as explored by Jarom D. Hogue in KDL and Mohammadreza Sadeghi in MoKA, researchers can precisely control the trade-off between capacity and efficiency, often achieving full-rank transformations that outperform LoRA on complex reasoning tasks.

Element-wise Interactions and Hadamard Products

Another way to bypass the "narrow straw" of low-rank bottlenecks is through the Hadamard (element-wise) product. Methods like HiRA by Qiushi Huang apply a low-rank trainable matrix as an element-wise modulator to the frozen pre-trained weights.

The intuition here is that the pre-trained weights already contain high-rank information; by using a Hadamard product, the update inherits the rank of the original weight matrix. Similarly, Raghav Singhal introduces ABBA-Adapters, which reparameterizes updates as the Hadamard product of two independent low-rank pairs. This allows the effective rank to scale up to the square of the individual ranks, capturing nuanced patterns that additive low-rank methods miss.

Full Rank via Structured Kernels

A third approach replaces the general matrices of LoRA with structured matrices, such as Circulant or Toeplitz forms. These matrices are mathematically "unrestricted-rank"—they can be full-rank—but they are defined by a very small number of parameters (often a single vector or "kernel").

In SURM, Avinava Dubey argues that weight updates do not inherently need to be low-rank to be efficient. By using structured matrices, one can achieve a 12x reduction in parameters compared to standard adapters while gaining significant accuracy in vision tasks. Aochuan Chen takes this further in C3A by using circular convolutions powered by Fast Fourier Transforms (FFT). This design completely decouples the rank of the update from the parameter count, allowing for full-rank adaptations with lower memory overhead than LoRA.

Spanning the Space with Random Bases

If the goal is to reach a high-rank update, why not use a fixed, high-rank foundation? Soroush Abbasi Koohpayegani introduces NOLA, which represents weight updates as a linear combination of fixed, random basis matrices.

By learning only the scalar coefficients for these bases, the number of trainable parameters becomes independent of the model's hidden dimensions. Paul Albert extends this logic in RandLoRA, showing that aggregating multiple scaled random matrices can bridge the performance gap between PEFT and full fine-tuning.

Why High Rank Matters

The shift from low-rank to high-rank adaptation is driven by a recurring finding: rank is the currency of memory. As Ting Jiang notes in MoRA, low-rank updates struggle with "knowledge-intensive" tasks like continual pretraining because they lack the capacity to store new information distributions.

By using mechanisms like Kronecker products, Hadamard products, or circulant structures, these researchers have shown that we can maintain the "efficiency" of PEFT (low parameter count, low optimizer state memory) while reclaiming the "expressivity" of full fine-tuning (high rank, complex feature capture). The result is a generation of adapters that are not just smaller, but fundamentally more capable of handling the complexity of modern foundation models.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_high_rank_updates_for_parameter_efficien_5790e929,
  title = {High-Rank Updates for Parameter-Efficient Fine-Tuning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-rank-updates-for-parameter-efficient-fine-tuning/rdr_5790e92930a64b8ab3692c89ff044448},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_high_rank_updates_for_parameter_efficien_5790e929_v1,
  title = {High-Rank Updates for Parameter-Efficient Fine-Tuning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-rank-updates-for-parameter-efficient-fine-tuning/rdr_5790e92930a64b8ab3692c89ff044448/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}