When training large language models or vision transformers, the primary hardware bottleneck is often not the model weights themselves, but the auxiliary memory consumed by the optimizer. Adaptive optimizers like AdamW require tracking historical gradient statistics—specifically, the first moment (momentum) and the second moment (variance)—for every single parameter in the network.
For a weight matrix of size $m \times n$, an adaptive optimizer must store two additional tensors of size $m \times n$. In mixed-precision training, these states are typically maintained in 32-bit floating-point precision to ensure numerical stability. Consequently, the optimizer states alone can consume up to four times the memory of the model weights, creating a severe "memory wall" that restricts full-parameter training on consumer hardware.
However, a fundamental question arises: Does every individual parameter truly require its own independent coordinate in the optimizer state? Recent insights demonstrate that gradients and optimizer states possess massive structural redundancy. By projecting these states into low-dimensional subspaces or factorizing them along tensor dimensions, we can compress the optimizer's footprint by 60% to 98% without degrading convergence or final model accuracy.
Gradient Subspace Projection
One powerful strategy to bypass the memory wall is to perform optimization entirely within a low-rank subspace. Instead of tracking statistics for the full-rank gradient matrix $G \in \mathbb{R}^{m \times n}$, we can project $G$ into a compact subspace of rank $r$ (where $r \ll \min(m, n)$).
In GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, Jiawei Zhao et al. show that the high-dimensional weight gradient can be mapped into a low-rank space using projection matrices derived via Singular Value Decomposition (SVD). The adaptive optimizer then tracks momentum and variance exclusively for this compressed representation. This shifts the optimizer state memory requirement from $O(m \times n)$ to $O(r(m + n))$, enabling full-parameter pre-training of 7B models on a single consumer GPU.
While computing SVD periodically introduces computational overhead, subsequent methods optimize this step. For instance, Kaizhao Liang et al. introduce Online Subspace Descent (Memory-Efficient LLM Training with Online Subspace Descent), which replaces intermittent SVD with continuous online Principal Component Analysis (PCA). This approach dynamically updates the projection matrix at every step, narrowing the performance gap on complex downstream reasoning tasks. To guarantee stable convergence during these subspace shifts, Thomas Robert et al. develop projection-aware update rules in LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics, ensuring that the error introduced by subspace tracking is safely managed via generalized feedback mechanisms.
To eliminate the memory required to store unique projection bases for every single layer, Mher Safaryan proposes Trion (Trion: FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of LLMs), a method that dynamically selects columns from a fixed Discrete Cosine Transform (DCT) matrix, eliminating the cumulative memory overhead of storing distinct layer-wise bases.
Tensor Factorization and Dimension Sharing
An alternative to explicit subspace projection is tensor factorization, which decomposes the full-rank optimizer state into low-rank components directly along its architectural dimensions.
The foundational intuition for this approach was demonstrated by Rohan Anil et al. with the SM3 optimizer (Memory Efficient Adaptive Optimization). Instead of tracking an element-wise second moment for a matrix, the authors group parameters by rows and columns, tracking only per-dimension statistics. The full second-moment matrix is approximated on the fly by taking the minimum of the corresponding row and column vectors, reducing the storage complexity from quadratic ($O(m \times n)$) to linear ($O(m + n)$).
Building on this concept, Son Nguyen et al. introduce Factorized Hamiltonian Descent (Memory-Efficient Optimization with Factorized Hamiltonian Descent), which applies rank-1 factorization to both the first and second moments of the optimizer. This dual-moment factorization represents the weight updates using two smaller vectors, slashing the optimizer's memory footprint by up to 90% while preserving standard adaptive convergence properties. To push memory savings even further, Bingrui Li et al. combine low-rank matrix factorization of the second moment with 4-bit quantization of the first moment in Memory Efficient Optimizers with 4-bit States, resolving numerical instability through a specialized rank-1 normalization technique.
Structural Block Adaptivity
A third perspective asks whether every layer or block in a neural network actually requires a unique, dynamic learning rate. Deconstructing the mechanics of adaptive optimizers reveals that much of the fine-grained, per-parameter tracking is redundant.
In Deconstructing What Makes a Good Optimizer for Autoregressive Language Models, Rosie Zhao et al. demonstrate that for the vast majority of a network's layers, the ratios of adaptive learning rates can be frozen at their initial values without degrading performance. Dynamic adaptivity is primarily critical only for the final output layers and normalization layers.
Leveraging this insight, Yushun Zhang et al. design Adam-mini (Adam-mini: Use Fewer Learning Rates To Gain More). Instead of assigning a unique learning rate to every parameter, Adam-mini assigns a single learning rate to entire structural blocks (such as an entire attention head or an MLP block). This effectively eliminates the memory footprint of the second-order optimizer state while retaining first-order momentum, cutting the memory overhead of AdamW by half while increasing training throughput.
Similarly, Dayal Singh Kalra et al. develop SlimAdam (When Can You Get Away with Low Memory Adam?) by utilizing a layer-wise Signal-to-Noise Ratio (SNR) analysis. The authors show that one can safely replace per-parameter second moments with their mean values along specific architectural dimensions (such as MLP output dimensions) while selectively maintaining granular scaling only where variance remains high, such as token embeddings and attention heads.
Why It Works: The Low-Rank Reality of Gradients
The reason these compression strategies work so well without stalling training lies in the geometry of the neural network loss landscape. Although a model may contain billions of parameters, the actual trajectory of the gradients during optimization does not explore a high-dimensional space uniformly.
Gradients are highly correlated due to the underlying network structure and the nature of backpropagation. For example, parameters associated with the same neuron or attention head share highly correlated activation patterns. This structural redundancy means that the effective rank of the gradient matrix is remarkably low throughout training.
By constraining the optimizer states to a low-rank subspace or a block-wise structure, these methods act as a lossy but highly effective filter. They discard the high-frequency orthogonal noise that does not contribute to the true descent direction, while preserving the low-rank principal components that drive convergence. Ultimately, compressing the optimizer state decouples memory consumption from the raw parameter count, allowing large-scale models to break through the memory wall.