Training Large Language Models (LLMs) is notoriously memory-intensive, largely because of the "optimizer bottleneck." While a model might have billions of parameters, adaptive optimizers like Adam require storing two additional values—momentum and variance—for every single parameter. This effectively triples the memory footprint of the model's weights.
To solve this, researchers have turned to low-rank optimization. Methods like LoRA and GaLore project gradients into a low-dimensional subspace, allowing the optimizer to track history for only a fraction of the total parameters. However, these subspaces are not static; they must evolve to capture the full complexity of the learning process. This creates a geometric problem: when the subspace shifts, the optimizer’s "memory" (its moments) becomes mathematically misaligned with the new coordinate system.
The Problem of Subspace Drift
In standard full-rank training, the optimizer's coordinate system is fixed. In low-rank training, however, we are essentially looking at the model through a narrow "window" (the subspace). As training progresses, we need to move this window to follow the gradient.
The issue, as identified by Nurbek Tastan and Samuel Horváth in LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning, is that the optimizer's internal states—momentum and variance—become "stale." If the low-rank basis shifts but the momentum buffer remains in the old basis, the next update will point in a direction that is no longer representable or relevant. This "computational shock" leads to loss spikes, training instability, and a performance gap between low-rank and full-parameter fine-tuning.
Aligning the Optimizer's Memory
The core insight of recent research is that we must mathematically "rotate" or "project" the optimizer's history whenever the subspace changes. Instead of resetting the optimizer—which discards valuable information—we transform the existing moments to align with the new geometry.
Several researchers have proposed different flavors of this alignment:
- Basis Rotation and Projection: Jui-Nan Yen and Si Si introduced LoRA Done RITE, which uses matrix preconditioning to ensure "transformation invariance." By rotating the first and second moments into the current orientation of the model's weight factors, they maintain the integrity of the gradient signal across steps.
- Projection-Aware Updates: In LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics, Thomas Robert and Dan Alistarh develop update rules that explicitly account for periodic basis changes. They show that rotating states during transitions prevents the accuracy loss typically associated with subspace shifts.
- Continuous Adaptation: Rather than periodic jumps, Zhengbo Wang suggests in Taming Momentum that updating the low-rank factors at every single step eliminates discontinuities entirely, keeping the compressed state perpetually aligned with the gradient.
Maintaining Temporal Consistency
Beyond simple rotation, the stability of the subspace itself matters. If the projection matrix jumps wildly from one iteration to the next, no amount of rotation can save the momentum.
Jinqi Xiao and Shen Sang argue in COAP that we must ensure "temporal consistency" between successive projection matrices. By treating the subspace update as a mini-optimization problem that considers historical momentum, they prevent the abrupt shifts that destabilize training. Similarly, Yehonathan Refael demonstrates in AdaRankGrad that preserving optimizer history through these transitions is what allows low-rank methods to finally match the convergence speed of full-rank Adam.
Why Geometric Alignment Works
The intuition is simple: an optimizer is a system with physical momentum. If you suddenly change the coordinate system of the world without adjusting the momentum vector, the "object" (the model weights) will fly off in a nonsensical direction.
By mathematically aligning the optimizer's internal state with the evolving low-rank manifold—a process Steffen Schotthöfer describes as respecting the "curved" geometry of the low-rank parameter space—these methods allow us to reap the memory benefits of low-rank training without the stability penalties. The result is a training process that is as robust as full fine-tuning but requires only a fraction of the hardware.