Dynamic Subspace Projection for Memory Efficient Training

This direction is split into other directions: dynamic subspace optimization. Its last version stays readable and citable here.

Version 1 of 140 papers59 conceptsfrozen 2026-10-04

Training large language models (LLMs) is notoriously memory-intensive, primarily because of the "optimizer state" bottleneck. For every parameter in a model, optimizers like Adam must store auxiliary data (moments), effectively tripling or quadrupling the memory required for weights alone. While Low-Rank Adaptation (LoRA) reduces this by training a tiny fraction of parameters, it often fails to match the performance of full-parameter fine-tuning because it is trapped within a fixed, static subspace.

A new research direction, exemplified by methods like GaLore and Flora, overcomes this by treating low-rank updates not as a static constraint, but as a dynamic projection. By periodically rotating or re-sampling the subspace used for gradient updates, these methods allow a model to eventually explore its entire parameter space while only ever storing a fraction of the optimizer states at any given moment.

The Frozen Subspace Problem

To understand why dynamic subspaces matter, we must first look at the limitations of standard low-rank methods. In a traditional LoRA setup, we choose a low-rank bottleneck and keep it fixed throughout training. This is equivalent to saying, "I will only allow the model to learn in these specific 128 directions."

As Haochen Zhang and colleagues point out in their work on SARA, this creates a "frozen subspace" bottleneck. If the optimal path to a solution requires moving in a direction outside that initial 128-dimensional slice, the model simply cannot go there. This rigidity is why memory-efficient methods have historically lagged behind full-parameter training.

Temporal Rank Expansion

The core intuition behind this new direction is temporal rank expansion. While a single update might be low-rank (e.g., rank 128), the sum of many different low-rank updates can be full-rank.

Imagine trying to describe a 3D object using only 2D shadows. A single shadow (a projection) loses information. However, if you rotate the object and take many shadows from different angles, you can eventually reconstruct the entire 3D shape.

Jiawei Zhao and the authors of GaLore apply this to gradients. By periodically re-computing the projection matrix—typically via a Singular Value Decomposition (SVD) of the current gradient—the optimizer "pivots" to a new subspace. Over the course of training, the optimization trajectory spans the full rank of the weight matrix. This allows a 7B parameter model to be trained on a single consumer GPU (like an RTX 4090) with the same expressive power as full-parameter training.

Mechanisms for Subspace Rotation

Researchers have proposed several ways to manage this "rotation" to ensure the optimizer doesn't get stuck:

  • SVD-Based Re-projection: GaLore and GaLore 2 by Jane Xu use SVD to find the "principal components" of the gradient. They update the projection basis at fixed intervals to track the shifting trajectory of the weights.
  • Randomized Resampling: Yongchang Hao and the creators of Flora show that even randomly resampling the projection matrices can work. Because the "intrinsic dimension" of the model's landscape is low, random subspaces are likely to capture enough signal to make progress, and changing them frequently ensures no part of the parameter space is permanently ignored.
  • Importance Sampling: To avoid the overhead of frequent SVDs, Haochen Zhang introduced SARA, which uses importance sampling to select subspaces. This prevents the optimizer from becoming "greedy" and only focusing on the most dominant directions, which can lead to stagnation.
  • Basis Mixing: Wenbo Gong and Meyer Scetbon suggest in their work on Alice that we should periodically mix the current projection basis with random components from the "complement" space. This forces the optimizer to "scout" for emerging gradient directions that haven't become dominant yet.

Why It Works: The Low-Dimensional Landscape

This approach is supported by the "Low Dimensional Landscape Hypothesis," which Lei Tan and Xiaolin Huang have explored extensively. They demonstrate that DNNs can often be trained in subspaces as small as 40 dimensions.

The reason dynamic projection is so effective is that while the entire parameter space is huge, the "useful" directions at any specific moment in training are very few. Luke Melas-Kyriazi and Franklyn Wang argue in their work on Intrinsic Gradient Compression that by changing the subspace periodically, we exploit this low intrinsic dimensionality without being permanently constrained by it.

Beyond Memory: Generalization and Communication

While memory efficiency is the primary driver, dynamic subspaces offer secondary benefits:

  1. Generalization: Yehonathan Refael shows in LORENZA that rotating the subspace acts as a form of regularization, helping models avoid sharp local minima and generalize better.
  2. Communication Efficiency: In decentralized or federated learning, communicating full gradients is impossible. Sameera Ramasinghe and colleagues use Subspace Networks to compress gradients into a shared low-dimensional subspace that evolves over time, allowing billion-parameter models to be trained over standard internet connections.

By shifting from "learning a small sub-model" (LoRA) to "learning the full model through a moving window" (GaLore/Flora), this research direction provides a path toward training state-of-the-art models on accessible, consumer-grade hardware.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_subspace_projection_for_memory_e_c9611f54,
  title = {Dynamic Subspace Projection for Memory Efficient Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-subspace-projection-for-memory-efficient-training/rdr_c9611f545aff496fa976292991e5428a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_subspace_projection_for_memory_e_c9611f54_v1,
  title = {Dynamic Subspace Projection for Memory Efficient Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-subspace-projection-for-memory-efficient-training/rdr_c9611f545aff496fa976292991e5428a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}