The explosion of Large Language Models (LLMs) has made Parameter-Efficient Fine-Tuning (PEFT) a necessity. The dominant paradigm, Low-Rank Adaptation (LoRA), works by freezing the original weights $W$ and adding a trainable product of two small matrices, $A$ and $B$. While memory-efficient, this approach forces all model updates into a tiny, static subspace.
Recent research suggests that this "low-rank" constraint is not just a shortcut—it is a fundamental bottleneck that prevents models from reaching their full potential in complex reasoning, creative generation, and large-scale pre-training. A new wave of methods is now moving beyond static low-rank updates to achieve "full-rank" expressiveness with "low-rank" memory costs.
The Expressivity Gap
The core problem with standard LoRA is its rigidity. By choosing a fixed rank $r$ at the start of training, researchers like Yongchang Hao, Yanshuai Cao, and Lili Mou argue in FLORA that the model is confined to a single, unchanging subspace. This lack of "subspace diversity" prevents the model from exploring the full parameter landscape.
This limitation manifests in several ways:
- Reasoning Failure: In tasks like Reinforcement Learning with Verifiable Rewards (RLVR), Qingyu Yin and colleagues found that extreme compression creates an information bottleneck, preventing models from learning the self-correction behaviors needed for complex reasoning Evaluating Parameter Efficient Methods for RLVR.
- Distribution Collapse: In image generation, Kartikeya Bhardwaj and Shubhankar Borse show that fixed-rank updates cause "copy-cat" behavior, where the model loses its creative spark and produces nearly identical outputs FouRA: Fourier Low-Rank Adaptation.
- Pre-training Barriers: While LoRA is fine for simple adaptation, it is often insufficient for pre-training from scratch, where the model needs to capture high-frequency details and broad representational power GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection.
Dynamic Subspaces and Gradient Compression
To break this bottleneck, researchers are reimagining LoRA not as a structural change to the model, but as a form of gradient compression.
In FLORA, the authors periodically resample the projection matrices. By changing the subspace throughout training, the model can accumulate information across different dimensions over time, effectively achieving a high-rank update while only ever storing low-rank states in memory.
Similarly, Jiawei Zhao and the team behind GaLore project the gradients into a low-rank subspace rather than the weights themselves. Because the projection matrix is updated during training, GaLore allows the model to explore the full parameter space. This allowed them to pre-train a 7B model on a single consumer GPU, a feat previously impossible with standard full-parameter training.
Accumulating Rank through Iteration
Another strategy to bypass the bottleneck is to treat model adaptation as a "residual" problem. Instead of finding one perfect low-rank update, these methods stack or merge multiple updates.
- Chain of LoRA (COLA): Chengwei Qin and colleagues proposed an iterative framework that trains a LoRA module, merges it into the base weights, and then initializes a new LoRA module to learn the remaining "residual" errors Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning. This allows the model to learn a high-rank transformation through a sequence of low-rank steps.
- Delta-LoRA: Bojia Zi introduced a method that uses the difference (delta) between low-rank matrices to update the "frozen" pre-trained weights directly Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices. This bridges the gap to full fine-tuning by allowing the entire weight matrix to evolve without the memory cost of full-rank gradients.
Rethinking the Low-Rank Structure
Some researchers argue that the $A \times B$ matrix factorization itself is the problem. They propose alternative structures that offer more "expressive bang for the buck."
Aozhong Zhang and the authors of DiaBlo: Diagonal Blocks Are Sufficient For Finetuning suggest that updating diagonal blocks of the weight matrix is more effective than low-rank decomposition. This approach avoids the structural bottleneck of $A \times B$ products, leading to more stable convergence in reasoning and coding tasks.
Other work, such as FRoD by Guoan Wan, uses "rotational degrees" and a globally shared basis to achieve full-rank expressiveness. By training only a small percentage of parameters that "rotate" the existing knowledge, they match full-model performance in just a few epochs.
Toward Memory-Efficient Full-Rank Training
The field is converging on a shared realization: low-rank is a useful constraint for memory, but a poor constraint for learning.
Whether through dynamic rank adjustment Hyeon Ju Shin et al., Riemannian optimization on fixed-rank manifolds RAdaGrad/RAdamW, or coordinate block selection BlockLLM, the goal is the same. We are moving away from "adapting" models via tiny side-channels and toward "training" models in ways that are as expressive as full fine-tuning but as light as LoRA.