Standard Parameter-Efficient Fine-Tuning (PEFT), exemplified by LoRA, relies on a fundamental hypothesis: weight updates in large models reside in a "low-rank" subspace. By representing an update $\Delta W$ as the product of two thin matrices $A$ and $B$ ($W = W_0 + AB$), we can tune millions of parameters by updating only a few thousand.
However, recent research suggests this low-rank constraint is a bottleneck. While efficient, it limits the model's ability to learn complex, high-frequency features or store novel information during continual pretraining. A new direction in PEFT seeks to decouple parameter efficiency from rank deficiency, enabling high-rank or even full-rank updates without increasing the trainable parameter count.
The Multiplicative Power of Kronecker Products
The most prominent strategy for breaking the rank bottleneck is the use of Kronecker products. In traditional LoRA, the rank of the update is limited by the inner dimension $r$ of the two matrices. In contrast, a Kronecker-based update represents $\Delta W$ as $A \otimes B$.
As Xuehai He and colleagues demonstrate in KAdaptation, the rank of a Kronecker product is the product of the ranks of its factors ($rank(A) \times rank(B)$), while the number of parameters is merely the sum of the parameters in $A$ and $B$. This allows for high-rank expressiveness with minimal overhead. Shyam Marjit applies this in DiffuseKronA to capture the intricate spatial details required for personalizing diffusion models, which standard low-rank methods often blur.
By summing multiple Kronecker products, as explored by Jarom D. Hogue in KDL and Mohammadreza Sadeghi in MoKA, researchers can precisely control the trade-off between capacity and efficiency, often achieving full-rank transformations that outperform LoRA on complex reasoning tasks.
Element-wise Interactions and Hadamard Products
Another way to bypass the "narrow straw" of low-rank bottlenecks is through the Hadamard (element-wise) product. Methods like HiRA by Qiushi Huang apply a low-rank trainable matrix as an element-wise modulator to the frozen pre-trained weights.
The intuition here is that the pre-trained weights already contain high-rank information; by using a Hadamard product, the update inherits the rank of the original weight matrix. Similarly, Raghav Singhal introduces ABBA-Adapters, which reparameterizes updates as the Hadamard product of two independent low-rank pairs. This allows the effective rank to scale up to the square of the individual ranks, capturing nuanced patterns that additive low-rank methods miss.
Full Rank via Structured Kernels
A third approach replaces the general matrices of LoRA with structured matrices, such as Circulant or Toeplitz forms. These matrices are mathematically "unrestricted-rank"—they can be full-rank—but they are defined by a very small number of parameters (often a single vector or "kernel").
In SURM, Avinava Dubey argues that weight updates do not inherently need to be low-rank to be efficient. By using structured matrices, one can achieve a 12x reduction in parameters compared to standard adapters while gaining significant accuracy in vision tasks. Aochuan Chen takes this further in C3A by using circular convolutions powered by Fast Fourier Transforms (FFT). This design completely decouples the rank of the update from the parameter count, allowing for full-rank adaptations with lower memory overhead than LoRA.
Spanning the Space with Random Bases
If the goal is to reach a high-rank update, why not use a fixed, high-rank foundation? Soroush Abbasi Koohpayegani introduces NOLA, which represents weight updates as a linear combination of fixed, random basis matrices.
By learning only the scalar coefficients for these bases, the number of trainable parameters becomes independent of the model's hidden dimensions. Paul Albert extends this logic in RandLoRA, showing that aggregating multiple scaled random matrices can bridge the performance gap between PEFT and full fine-tuning.
Why High Rank Matters
The shift from low-rank to high-rank adaptation is driven by a recurring finding: rank is the currency of memory. As Ting Jiang notes in MoRA, low-rank updates struggle with "knowledge-intensive" tasks like continual pretraining because they lack the capacity to store new information distributions.
By using mechanisms like Kronecker products, Hadamard products, or circulant structures, these researchers have shown that we can maintain the "efficiency" of PEFT (low parameter count, low optimizer state memory) while reclaiming the "expressivity" of full fine-tuning (high rank, complex feature capture). The result is a generation of adapters that are not just smaller, but fundamentally more capable of handling the complexity of modern foundation models.