When training a Large Language Model (LLM), the most severe bottleneck is often not the size of the model weights themselves, but the hidden "memory tax" imposed by the optimizer.
To understand why, consider training a 7-billion parameter model using standard 16-bit precision (FP16 or BF16). The model weights require 14 GB of video memory (VRAM), and their gradients require another 14 GB. However, if you optimize this model using Adam—the industry standard adaptive optimizer—the hardware requirements skyrocket.
Adam tracks two auxiliary statistics for every single parameter to adjust learning rates dynamically: the first moment (momentum) and the second moment (variance). To maintain numerical stability and prevent rounding errors from destroying training progress, these states are stored in 32-bit floating-point (FP32) precision. This requires 4 bytes for momentum and 4 bytes for variance per parameter, totaling 8 bytes. For a 7B model, this translates to 56 GB of VRAM dedicated solely to optimizer states.
This state overhead frequently triples the total memory footprint of the model weights alone, creating a critical "memory wall" that restricts large-scale training to expensive, high-end industrial hardware (Zhao et al., 2024; Alistarh et al., 2024).
To dismantle this memory wall, a prominent research direction focuses on compressing, projecting, or entirely eliminating optimizer states without sacrificing the convergence stability of adaptive optimization.
Downsizing the State Space via Subspace Projection
The core insight behind subspace methods is that gradient trajectories in deep networks are highly redundant. While a weight matrix might contain millions of individual parameters, the effective updates often lie within a much lower-dimensional subspace (Rosie Zhao et al., 2024). Instead of tracking historical statistics for every single parameter in the full-rank weight space, we can project the gradients into a compact subspace before passing them to the optimizer.
In GaLore, Jiawei Zhao et al. operationalized this by projecting a full-rank gradient matrix $G \in \mathbb{R}^{m \times n}$ into a low-rank matrix $P \in \mathbb{R}^{m \times r}$ (where $r \ll \min(m, n)$). The optimizer only maintains momentum and variance tensors for the downsized matrix $P$, drastically reducing the state memory. Once the optimizer computes the low-rank update step, it is projected back into the original dimensions to update the weights.
Full-Rank Gradient (G) ───> [ Project to Subspace ] ───> Low-Rank Gradient (P)
│
▼
Full-Rank Weights <─── [ Project Back ] <─── Low-Rank Step <─── [ Adam Optimizer (Small States) ]
Because the optimal gradient subspace shifts as the model learns, the projection matrix must change over time. GaLore periodically pauses to recompute the subspace using Singular Value Decomposition (SVD), which introduces a brief computational overhead. To make this process smoother, Kaizhao Liang et al. introduced Online Subspace Descent (OSD). OSD replaces discrete SVD steps with a continuous online Principal Component Analysis (PCA) mechanism, tracking the moving gradient subspace dynamically without training interruptions.
However, relying strictly on the most dominant subspace can trap optimization in a "frozen subspace," ignoring minor gradient directions that are critical for long-term convergence. To resolve this, Haochen Zhang et al. developed SARA, an importance sampling strategy that randomizes subspace selection to guarantee exploration and provable convergence. Similarly, Yongchang Hao et al. proposed Flora, which treats low-rank adaptation as a dynamic gradient compressor that periodically resamples projection matrices to allow high-rank updates with sublinear memory usage.
Compressing States Directly via Block-wise Quantization
Instead of altering the mathematical dimensions of the optimizer states, another approach attacks the bit-width of the states directly. Why do optimizer states require 32 bits when weights can survive at 16 or even 4 bits? The reason is variance: gradient statistics span multiple orders of magnitude across different layers and iterations. Naive quantization causes underflow, destroying the subtle historical differences the optimizer relies on to scale updates.
To preserve this wide dynamic range at lower bit-widths, Mike Lewis et al. introduced 8-bit optimizers using block-wise quantization. Instead of quantizing an entire state tensor globally, the authors segment the tensor into small, independent blocks (e.g., 2048 elements). Each block is quantized to 8 bits using its own dynamically calculated scaling factor.
$$\text{Block Quantization: } X_{\text{8-bit}} = \text{round}\left(\frac{X}{\text{max}(|X_{\text{block}}|)} \times 127\right)$$
This isolation ensures that a massive outlier gradient in one part of the network only compresses the dynamic range of its immediate neighbors, leaving the precision of smaller gradients elsewhere intact.
Pushing this concept to its logical extreme, Bingrui Li et al. achieved stable 4-bit optimizer states. The authors resolved the "zero-point problem"—where quantization errors systematically bias the moving averages over millions of steps—by introducing a novel rank-1 normalization technique that preserves optimization trajectories even under aggressive bit-width compression.
Bypassing States via Alternative Optimization
The most radical strategy for eliminating optimizer state memory is to alter the training loop so that tracking historical states becomes unnecessary. This is particularly effective during fine-tuning, where the model is already close to a solution and may not require the full per-parameter adaptivity of Adam.
One approach is to eliminate the temporal storage of gradients and states entirely through immediate execution. In LOMO (LOw-Memory Optimization), Kai Lv et al. fused the backward pass with the parameter update. Instead of calculating and storing the entire gradient map for all layers before updating the model, LOMO updates each layer's parameters immediately after computing its specific gradient during backpropagation. Once the layer is updated, its gradient is instantly cleared from memory. By pairing this fused update with a stabilized version of Stochastic Gradient Descent (SGD), LOMO completely bypasses the need for auxiliary state storage.
Taking state elimination a step further, Sadhika Malladi et al. developed MeZO, a zeroth-order optimization framework that eliminates backpropagation entirely. MeZO estimates gradients using only forward passes by applying a small random perturbation to the model weights and measuring the resulting change in the loss function.
$$\nabla f(W) \approx \frac{f(W + \epsilon z) - f(W - \epsilon z)}{2\epsilon} z$$
Because the random seed $z$ can be re-generated on the fly, the training loop requires no gradient tensors, no activation caching for backpropagation, and zero optimizer states. This collapses the training memory footprint down to match the footprint of simple inference, allowing consumer-grade hardware to fine-tune massive architectures.