Aligning Optimizer States Across Dynamic Subspaces

39 concepts20 papers1 proposal

Training large language models (LLMs) is heavily bottlenecked by memory. Standard adaptive optimizers like AdamW store first moments (momentum) and second moments (variance) for every single parameter, often doubling or tripling the memory footprint required by the model weights alone.

To bypass this bottleneck, a powerful strategy is to project gradients into a low-dimensional subspace, tracking optimizer states only within this compressed coordinate system. However, a fundamental challenge arises: the optimal low-rank subspace shifts as the model trains. As Tianyuan Zou et al. demonstrate in PE-SGD: Differentially Private Deep Learning via Evolution of Gradient Subspace for Text, a fixed subspace quickly becomes obsolete as the model's optimization trajectory evolves, causing the projection to miss critical gradient updates.

To maintain high performance, the projection subspace must change dynamically. But changing the subspace introduces a new mathematical crisis: subspace staleness.

The Dilemma of Subspace Staleness

When an optimizer redefines its low-rank projection matrix, the historical optimizer states stored in memory suddenly become mathematically misaligned with the new coordinate system.

Old Subspace Coordinates              New Subspace Coordinates
       [ Momentum Vector ]                   [ ??? ]
               \                                \
                \  -- (Subspace Shifts) -->      \  Historical states 
                 \                                \ become misaligned!

Early low-rank training frameworks handled this transition using one of two blunt strategies, both of which degrade convergence:

  1. The Reset Strategy: The optimizer states are completely wiped and reset to zero whenever the subspace is updated, often paired with a brief learning rate warmup to stabilize training (as seen in the framework by Aashiq Muhamed et al. [GRASS: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients]). While this avoids erratic updates, it discards valuable historical gradient information, forcing the optimizer to rebuild its momentum from scratch.
  2. The Anchor Strategy: The old optimizer states are retained without modification. As Vladislav Lialin et al. observe in ReLoRA: High-Rank Training Through Low-Rank Updates, these unaligned states act as a "subspace anchor," dragging the model back into previously explored directions and preventing it from effectively utilizing the new coordinate space.

As Pouria Mahdavinia and Mehrdad Mahdavi note in Low-rank Momentum Factorization for Memory Efficient Training, this misalignment leads to severe training instability and sub-optimal convergence compared to full-parameter training.

Mechanism: Projection-Aware State Transformation

To resolve this dilemma, a new paradigm of projection-aware optimization has emerged. Instead of discarding history or allowing geometric drift, these methods mathematically transform historical optimizer states so that they remain valid within the new subspace.

First-Moment Transport

When the low-rank basis shifts, the historical first-moment vector (the running velocity of the gradients) must be projected or rotated into the new coordinate system. Xin Yu et al. implement this in AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections by explicitly projecting historical momentum into the current low-rank subspace at each training step.

Similarly, the authors of SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training map first-order moments from the previous subspace to the current one, ensuring that the moving average of gradients remains continuous and relevant even as the representational view shifts.

Second-Moment Alignment

Aligning the first moment is only half the battle. In adaptive optimizers like Adam, updates are scaled by the second moment (variance). Junjie Wang et al. point out a critical failure mode in Conda: Column-Normalized Adam for Training Large Language Models Faster: if the second-moment scaling factors (the denominator) operate in a mismatched coordinate system relative to the first-moment estimates (the numerator), the optimizer calculates erratic step sizes, causing immediate training divergence at high learning rates.

To prevent this, modern projection-aware optimizers—such as those designed by Thomas Robert et al. [LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics] and Yehonathan Refael et al. [AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning]—simultaneously map both moments into the new subspace during rank or basis transitions.

[Old Subspace Moments] 
       │
       ▼
[Projection Matrix Transformation] ──► [Compensate for Lost Info]
       │                                         │
       ▼                                         ▼
[New Subspace Moments] ────────────────► [Stable Parameter Update]

Why It Works: Geometric Continuity

The reason projection-aware state transformation works so effectively can be understood through the lens of differential geometry. Low-rank parameter spaces are not flat; they form curved manifolds.

As Steffen Schotthöfer et al. explain in A geometric framework for momentum-based optimizers for low-rank training, when weight updates and internal optimizer states diverge, the optimizer suffers from "geometric drift." By utilizing projection matrices to rotate historical moments into the current coordinate system—often combined with a mathematical compensation term to account for information lost during the transformation (Jui-Nan Yen et al. [LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization])—the optimizer maintains a mathematically consistent trajectory along the loss landscape.

By tracking these gradient subspaces on a smooth Grassmannian manifold, frameworks like the one proposed by Sahar Rajabi et al. [SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training] eliminate the discrete, abrupt jumps in learning progress that typically cause computational shock. This geometric alignment ultimately allows low-rank training methods to closely mimic the convergence behavior, trajectory, and final accuracy of full-parameter optimization while maintaining a fraction of the memory footprint (Yongchang Hao et al. [Flora: Low-Rank Adapters Are Secretly Gradient Compressors]; Nurbek Tastan et al. [LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning]).

Key Foundational Work

SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training (2025) — contributed 5 concept elements to this direction.
AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (2025) — contributed 3 concept elements to this direction.
LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics (2025) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_subspace_momentum_transfer_7762db8c,
  title = {cross-subspace momentum transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-subspace-momentum-transfer/rdr_7762db8cc9f94e6aa0abb2b136e25838},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_subspace_momentum_transfer_7762db8c_v1,
  title = {cross-subspace momentum transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-subspace-momentum-transfer/rdr_7762db8cc9f94e6aa0abb2b136e25838/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake