Exploring Full Parameter Spaces via Dynamic Low Rank Subspaces

23 concepts18 papers1 proposal

When training modern large language models, the primary bottleneck is often not the model weights themselves, but the massive memory overhead required by adaptive optimizers like Adam. Storing historical gradient momentum and variance requires several times the memory of the static model parameters.

To mitigate this, parameter-efficient methods like Low-Rank Adaptation (LoRA) freeze the original weights and train a low-dimensional bottleneck. However, static low-rank constraints inherently limit the model's expressivity. Because the optimization is permanently trapped within a fixed subspace, the model cannot learn features outside that initial slice of the parameter space.

Dynamic subspace optimization resolves this tension. The core intuition is simple: while an individual optimization step can be computed within a low-dimensional subspace to save memory, the subspace itself can be rotated, resampled, or updated over time. By changing the projection basis throughout training, a sequence of low-rank steps can accumulate into a full-rank update trajectory, allowing the model to explore the entire parameter landscape without ever materializing full-rank optimizer states.

Static Subspace (e.g., LoRA):
Step 1: [─── Subspace A ───] ➔ Step 2: [─── Subspace A ───] ➔ Cumulative: [─── Subspace A ───] (Trapped)

Dynamic Subspace:
Step 1: [─── Subspace A ───] ➔ Step 2: [─── Subspace B ───] ➔ Cumulative: [────── Full Rank ──────] (Explored)

The Limitations of Rigid Bottlenecks

As highlighted by Choromanski et al., the intrinsic dimensionality of an optimization landscape changes dynamically as training progresses. Forcing a network to operate within a fixed hyperparameter-defined subspace fails to capture these geometric shifts.

When an optimizer is restricted to a static set of adaptors, it frequently stalls. Refael et al. note that this restriction prevents the model from escaping suboptimal local minima, severely harming generalization in both pre-training and fine-tuning scenarios. In the spectral domain, training on a static projection causes diminishing returns; the optimizer struggles to capture fine-grained details associated with smaller eigenvalues, causing convergence to plateau (Zhang et al.).


Mechanism 1: Random Resampling and Subspace Rotation

The most straightforward way to break out of a frozen subspace is to periodically redraw the projection matrices completely at random.

In Random Bases Descent (RBD), Gressmann et al. demonstrate that re-drawing random projection bases at every optimization step provides the necessary directional variety to navigate complex loss landscapes. Because the hardware-accelerated random projections change constantly, the optimizer explores a much broader range of weight-space directions than any fixed projection could allow.

Taking a similar view, Hao et al. introduce FLORA, which frames low-rank adaptation as a form of gradient compression. By periodically resampling the random projection matrices used during training, the individual low-rank updates gradually accumulate across different subspaces. Over many iterations, the cumulative weight updates span a high-rank space, effectively bypassing the expressive limitations of a fixed low-rank bottleneck and matching full-parameter training performance.


Mechanism 2: Data-Driven and Gradient-Aligned Subspaces

Rather than relying on purely random rotations, a more targeted approach constructs the subspace using the structural properties of the gradients themselves.

The foundational framework for this is GaLore, proposed by Zhao et al.. Instead of projecting the weights, GaLore computes a Singular Value Decomposition (SVD) on the full-rank gradient matrix periodically during training to derive a low-rank projection matrix. By updating these projection matrices at scheduled intervals, the training trajectory pivots through a sequence of different low-rank subspaces. As the model weights evolve, the projection matrix adapts, ensuring that the compressed gradients remain aligned with the most significant directions of the optimization trajectory.

Full Gradient ➔ [ Periodic SVD ] ➔ Low-Rank Projection Basis ➔ Memory-Efficient Adam Update
                                           ▲
                                   Evolving Over Time

Several variations build directly on this data-driven alignment:

  • Layerwise Importance: Yin et al. introduce OwLore, which periodically recalculates low-rank subspaces but prioritizes "outlier-rich" layers, allocating more tracking capacity to the layers experiencing the most volatile optimization trajectories.
  • Weight Normalization: Huang utilizes a scheduled re-computation of gradient weight-normalized projections to ensure the low-rank subspace tracks evolving gradient dynamics smoothly as the model converges.
  • Black-Box Adaptation: In settings where gradients are unavailable, Liu et al. show that iteratively learning and updating the basis vectors of a transformation matrix allows for highly query-efficient gradient estimation in black-box visual prompting.

Mechanism 3: Escaping Stagnation via Complement Spaces

A key challenge with gradient-aligned methods is the risk of self-reinforcing stagnation. If an optimizer only tracks the dominant eigenvectors of the current gradient, it might completely ignore emerging parameter directions that currently have small gradients but are crucial for escaping a plateau.

To solve this, Gong et al. propose periodically swapping out some of the leading eigenvectors for vectors drawn from the random complement space. By forcing the optimizer to integrate these orthogonal directions, the algorithm actively explores new regions of the loss landscape that standard low-rank tracking methods miss.

Similarly, Zhang et al. replace rigid dominant subspace selection with a randomized importance sampling strategy (SARA). Instead of deterministically picking the top singular vectors, they sample directions stochastically based on singular values, ensuring that minor but vital directional components are occasionally explored. In the context of parameter-efficient fine-tuning, Zhang et al. introduce UORA, which selectively reinitializes "dead" or underperforming dimensions within frozen projection matrices, allowing the model to dynamically discover optimal low-rank updates without expanding the parameter footprint.


Alternative Geometries and Update Frequencies

The frequency and mathematical formulation of the subspace update represent another crucial design axis. While methods like GaLore update the subspace at infrequent, discrete intervals to minimize SVD overhead, Mahdavinia and Mahdavi argue that infrequent updates allow stale momentum to accumulate outside the active subspace. Their approach, MoFaSGD, maintains a dynamically updated low-rank SVD representation of gradient momentum that changes at every iteration, ensuring continuous alignment with the true gradient flow.

Furthermore, the update process can be framed through alternative geometric lenses:

  • Grassmann Manifolds: Ramasinghe et al. treat the low-dimensional subspace as a point on a Grassmann manifold—the mathematical space of all linear subspaces of a given dimension. By performing optimization directly on this manifold, they continuously align the basis with the most significant gradient directions, enabling communication-efficient decentralized training.
  • Structured Sparsity: Rather than dense low-rank projections, Muhamed et al. project gradients into a structured sparse subspace. The active subspace is periodically updated by re-selecting parameter rows based on row norms or stochastic sampling, mimicking the behavior of coordinate descent across the full weight matrix.
  • Local Geometry Refinement: In black-box scenarios, Cosson et al. show that re-evaluating and updating the active subspace whenever descent progress stalls allows the algorithm to adapt fluidly to changing local low-rank structures along the optimization trajectory.

By shifting the optimization paradigm from a single, static low-dimensional bottleneck to a sequence of dynamically evolving subspaces, these methods successfully decouple the memory footprint of an individual training step from the total expressive capacity of the model.

Key Foundational Work

Improving Neural Network Training in Low Dimensional Random Bases (2020) — contributed 2 concept elements to this direction.
From Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox Optimization (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_subspace_optimization_8bc2546a,
  title = {dynamic subspace optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-subspace-optimization/rdr_8bc2546ae3334954be3cb96de20915fc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_subspace_optimization_8bc2546a_v1,
  title = {dynamic subspace optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-subspace-optimization/rdr_8bc2546ae3334954be3cb96de20915fc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake