Low-Rank Gradient Projection for Memory-Efficient LLM Training

This direction is merged into another direction: optimizer state memory overhead. Its last version stays readable and citable here.

Version 1 of 180 papers81 conceptsfrozen 2026-10-04

The primary obstacle to training Large Language Models (LLMs) is not the size of the model weights themselves, but the "memory wall" created by the optimizer. While a 7-billion parameter model might only require 14GB of VRAM to store its weights in half-precision, training it with a standard optimizer like AdamW can require up to 100GB. This is because AdamW maintains "optimizer states"—running averages of gradients and their squares—for every single parameter.

To break this wall, a new research direction focuses on Low-Rank Gradient Projection. This approach rests on a powerful intuition: although LLMs have billions of parameters, the gradients used to update them during training are highly redundant and reside in a much lower-dimensional subspace. By projecting these gradients into a low-rank space, researchers can store optimizer states for a tiny fraction of the total parameters, enabling full-parameter training on consumer-grade hardware.

The Geometry of Redundant Gradients

In standard training, a weight matrix $W \in \mathbb{R}^{m \times n}$ has a corresponding gradient matrix $G$ of the same size. Optimizers like Adam store two additional matrices of size $m \times n$ to track momentum and variance. Jiawei Zhao, Anima Anandkumar, and their colleagues introduced GaLore (Gradient Low-Rank Projection) to exploit the fact that $G$ is typically low-rank.

Instead of tracking states for the full $G$, GaLore computes a low-rank projection $P^\top G Q$, where $P$ and $Q$ are projection matrices. The optimizer then only needs to track states for this compressed representation. This reduces the memory footprint of optimizer states by up to 65.5% while still allowing the model to learn across all its parameters—unlike Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA, which freeze the original weights and only train a small adapter.

Evolving the Subspace: From SVD to Online PCA

A challenge with low-rank projection is that the "optimal" subspace for gradients changes as the model learns. Early versions of this idea required periodic Singular Value Decomposition (SVD) to find new projection matrices, which is computationally expensive.

To solve this, Kaizhao Liang and Qiang Liu proposed Online Subspace Descent (OSD). Instead of stopping training to perform SVD, OSD uses a continuous online PCA approach to update the subspace on the fly. Similarly, SubTrack++ treats the subspace as a point on a Grassmannian manifold, using geometric updates to track the gradient's movement smoothly. These methods ensure that the compressed optimizer states always capture the most relevant information for the current stage of training.

Quantization and Random Projections

The efficiency of low-rank projection can be further amplified by combining it with quantization. AJAY KUMAR JAISWAL and Zhenyu Zhang developed Q-GaLore, which quantizes the projection matrices themselves to INT4. This allows a LLaMA-7B model to be trained from scratch on a single 16GB GPU.

Other researchers have explored even simpler ways to define the subspace. Hanqing Zhu and Zhenyu Zhang introduced APOLLO, which uses random projections to approximate the gradient structure. By avoiding the need to compute SVD or PCA entirely, APOLLO achieves AdamW-level performance with a memory footprint closer to simple SGD.

Eliminating States Entirely

While low-rank methods shrink optimizer states, some researchers argue for eliminating them entirely. Chao Ma and Edward Meeds proposed SWAN, a "stateless" optimizer that uses row-wise normalization and whitening to achieve the stability of Adam without storing any historical gradient data.

For fine-tuning, Kai Lv and Qipeng Guo introduced LOMO (LOw-Memory Optimization). LOMO fuses the gradient computation and parameter update into a single step during the backward pass. By updating parameters as soon as their gradients are calculated, LOMO avoids storing a full gradient map or any momentum buffers, reducing memory usage to roughly 10% of standard methods.

Why It Works: The Low-Rank Hypothesis

The success of these methods across papers like SLTrain and SARA suggests a fundamental property of neural network optimization: the "effective" dimensionality of the task is much smaller than the parameter count.

Whether through GrassWalk's randomized subspaces or HiFT's hierarchical block updates, the research community is converging on a shared conclusion: we can train massive models by focusing only on the most important directions of change. This shift from "dense" optimization to "subspace" optimization is what finally makes the training of multi-billion parameter models accessible to researchers without massive industrial compute clusters.

Papers in this version

Showing 60 of 80 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_low_rank_gradient_projection_for_memory__6a466ce6,
  title = {Low-Rank Gradient Projection for Memory-Efficient LLM Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-rank-gradient-projection-for-memory-efficient-llm-training/rdr_6a466ce68748440aafc1b996f81e3af9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_low_rank_gradient_projection_for_memory__6a466ce6_v1,
  title = {Low-Rank Gradient Projection for Memory-Efficient LLM Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-rank-gradient-projection-for-memory-efficient-llm-training/rdr_6a466ce68748440aafc1b996f81e3af9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}