Do Optimizer Momenta Obey an Area Law? Matrix-Product-State Momentum for Memory-Efficient LLM Training

Training a large language model with Adam costs far more memory than the model itself, because the optimizer keeps a running average of past gradients (the "momentum") and of their squares for every parameter. A line of work called dynamic subspace optimization reduces this cost. It stores the momentum only inside a small, low-rank slice of each weight matrix and keeps moving that slice as training proceeds, so the accumulated updates still reach every direction. The best of these methods, MoFaSGD, keeps a rank-$r$ singular value decomposition of the momentum and updates it at every step. It trains LLaMA-3.1-8B in 29.4 GB instead of AdamW's 70.8 GB. The price is quality: it still trails full AdamW by 4.2% on Tulu3, and its own analysis finds that the top 32 singular directions hold only about 80% of the momentum's energy. The remaining gap is therefore not a memory problem. It is a problem of how much momentum one can store per byte.

We propose to spend the same bytes differently. Each singular vector of a 4096×4096 momentum matrix is a 4096-long vector. In MoFaSGD it is stored densely. We would instead reshape it into a small grid and store it as a matrix product state (MPS), also called a tensor train: a chain of small three-way tensors whose connecting dimensions ("bond dimensions") limit how much correlation can flow between parts of the index. Physicists use this format because many physical states obey an "area law" and need only small bonds. If momentum singular vectors are similarly compressible, the same memory could hold a rank-256 momentum where MoFaSGD holds rank 32. The first job of this project is to measure whether that is true, which is cheap to do. The second is to build the optimizer only if the measurement says yes.

Core Claim

Fold the row index of an $m \times n$ momentum matrix as $m = a \cdot b$ and the column index as $n = c \cdot d$, and order the four modes as (row-coarse, row-fine, column-fine, column-coarse). An MPS on this chain has three bonds. The middle bond separates row modes from column modes, so in canonical form its Schmidt decomposition (the tensor-network name for an SVD across a cut) is exactly the matrix SVD of the momentum:

$$M_t = U_t \Sigma_t V_t^\top, \qquad U_t = \mathrm{TT}(A^{(1)}, A^{(2)}), \quad V_t = \mathrm{TT}(A^{(3)}, A^{(4)}).$$

Here $M_t$ is the momentum at step $t$, $\Sigma_t$ is the diagonal matrix of $\chi_{mid}$ singular values on the middle bond, and $U_t, V_t$ are the $m \times \chi_{mid}$ and $n \times \chi_{mid}$ singular-vector matrices. Each is stored as two cores joined by a small outer bond $\chi_{out}$. Setting $\chi_{out}$ to its maximum recovers MoFaSGD exactly, so the method is a strict generalization of it.

Take a 4096×4096 layer folded as $64 \times 64$ on each side. A dense rank-32 $U$ costs about 131k numbers. A tensor-train $U$ with $\chi_{out} = 8$ and $\chi_{mid} = 256$ costs $64 \cdot 8 + 8 \cdot 64 \cdot 256 \approx 131$k. That is the same memory for eight times the rank.

The claim to test is this: at equal optimizer memory, an MPS momentum captures more gradient energy than a truncated SVD, and it closes part of MoFaSGD's gap to AdamW. The claim is false if the singular vectors of real momenta have near-maximal "entanglement" across the within-row cut. In that case the tensor train buys nothing, and we would report that as a negative result.

Relation to Prior Work

The proposal borrows its optimizer structure from dynamic subspace methods. GaLore and GaLore 2 project gradients onto a subspace recomputed by SVD every few hundred steps. SARA shows that always choosing the dominant directions freezes the subspace, and that sampling directions in proportion to their singular values helps. MoFaSGD removes periodic SVDs by tracking the momentum on the low-rank manifold every step. All of these store singular vectors as dense columns. We keep MoFaSGD's per-step tracking and its normalized update, and change only how the singular vectors are stored. SARA's finding also motivates us: if narrow subspaces stall training, a cheaper way to widen the subspace is valuable.

The tangent-space geometry comes from Cai et al.. They prove linear convergence of Riemannian gradient descent on the manifold of tensors with fixed tensor-train rank for static tensor completion. We reuse their tangent projection and retraction, but apply them to a stochastic, non-stationary momentum stream, where their guarantees do not transfer.

Three tensorized methods constrain the design by contrast. Tensor-GaLore projects gradients with a Tucker decomposition for neural-operator weights. Its dense core grows exponentially with tensor order, and it was tested only on low-order PDE tensors. CoMERA stores the weights themselves in tensor-train form and adapts ranks. TT-LoRA and TRAC use static tensor-train adapters, and TRAC explicitly freezes a random core. All four either restrict the model's weights or never move their subspace. Here the weights stay dense and unconstrained, and only the optimizer state is tensorized and moving.

Tensor Train Random Projection shows that even rank-1 tensor-train projections preserve distances in expectation. That supports a cheap variant in which new directions are drawn at random but kept in tensor-train form.

Research Program

Diagnostic stage. The first stage is purely a measurement, and it gates the rest. We pre-train LLaMA-60M and LLaMA-130M on C4 with AdamW, following the GaLore protocol, and save momentum buffers at a dozen checkpoints for every layer type. For each buffer we compute:

  • the singular spectrum at each cut of the folded tensor under several mode orderings (row-then-column, interleaved, and random permutations as a control);
  • the energy captured versus stored parameters for SVD and for tensor trains.

Mode ordering matters a great deal for MPS, so the orderings are part of the question, not an afterthought. We also examine attention projections folded along the head axis, where a natural three-way structure (input × heads × head dimension) exists.

Go/no-go criterion. We proceed only if, at matched memory, the tensor-train form captures clearly more energy than SVD, for example lifting captured energy well above the roughly 80% MoFaSGD reports at rank 32.

Optimizer stage. The second stage builds MPS-MoFa. Each step does the following:

  1. Project the incoming gradient onto the tangent space of the current tensor train, following Cai et al. This costs on the order of $mn\chi$, comparable to MoFaSGD's $mnr$ projection.
  2. Form the momentum update in the tangent space.
  3. Retract by tensor-train rounding, which costs on the order of $N d \chi^3$ with $N = 4$ modes.
  4. Move the chain into canonical form at the middle bond and apply the normalized update:

$$W_{t+1} = W_t - \eta\, U_t V_t^\top.$$

Here $\eta$ is the learning rate. The update replaces the singular values with ones, as MoFaSGD and Muon do.

To address known kernel-launch overhead from small tensor contractions, we fold into only four modes of size 64 so that every contraction is a GEMM of reasonable size. We also borrow CoMERA's approach of recording contraction paths with CUDA Graphs.

Bond-sampling variant. A second variant ports SARA's idea to the middle bond. It samples which Schmidt directions to keep in proportion to their weights, and tests whether a "frozen bond" analogue of the frozen subspace appears.

Baselines

Naive and quality ceiling. The naive reference is full AdamW with dense states. It sets the quality ceiling and the memory cost. We run it with the GaLore C4 recipe (same token budget, sequence length and learning-rate sweep). Beating it is not the goal; the goal is to shrink the gap to it.

MoFaSGD. The strongest competitor is MoFaSGD at rank $r$. It is the closest method: same per-step tracking, same update rule. The only difference is dense versus tensor-train singular vectors. Reported results are 29.4 GB peak memory on LLaMA-3.1-8B, +0.8% average over GaLore, and −4.2% versus AdamW on Tulu3. On FineWeb NanoGPT it beats GaLore, but no reported absolute perplexity was found on Lacuna for MoFaSGD on FineWeb. We rerun it ourselves at ranks 16, 32 and 64, and match MPS-MoFa's optimizer memory to each rank within 5%. Each method gets the same learning-rate grid of equal size.

GaLore and SARA. For C4 perplexity, the published reference points are GaLore-SARA-Adam at 24.21 and GaLore-Adam at 24.88 validation perplexity on LLaMA-130M, from SARA. We reproduce both under our pipeline before comparing. Beating them means lower validation perplexity at equal optimizer memory, repeated over three seeds, with the margin larger than the seed spread.

Ablations isolating the key ingredient. The first ablation is "dense-at-high-rank": MoFaSGD at rank $\chi_{mid}$ with full dense factors, using more memory. It shows how much of any gain comes simply from higher rank, and how much the tensor-train compression of singular vectors gives back. The second is the same MPS-MoFa with a random mode ordering. If random ordering does as well, the gain is not coming from structure. The third, on attention layers only, is Tensor-GaLore's Tucker projection at matched memory.

Why not just…?

Why not just raise MoFaSGD's rank or quantize its factors? This is the real threat. Optimizer state is already a small share of total memory, so a rank-64 MoFaSGD may cost little extra. Quantization is orthogonal to our change and could be stacked on it, so it is not a rival. Raising the rank is exactly our matched-memory comparison. We report quality against total peak memory, not just optimizer memory. If rank-64 dense MoFaSGD matches MPS-MoFa at a negligible total-memory increase, the practical case collapses to settings where many layers are memory-bound. We would say so.

Why not just use Tucker as Tensor-GaLore does? For a four-mode fold with ranks near 64, a dense Tucker core has about $64^4 \approx 1.7 \times 10^7$ entries. That is larger than the matrix itself. Tucker is viable only for low-order, low-rank folds such as the three-way attention tensor, which is precisely where we include it as a baseline.

Why not just train tensor-train weights as CoMERA or TT-LoRA do? These methods restrict what the model can represent. TT-LoRA and TRAC are fine-tuning-only and never move their subspace. CoMERA's compressed models are a different product. Our weights stay dense, so the result is an ordinary checkpoint. The comparison that matters is against dense-weight optimizers.

Why not just use random tensor-train projections, as in TTRP? Random projections are cheap and distance-preserving, but they ignore gradient statistics. We include rank-1 random tensor-train bases resampled every step as a variant. If this variant matches the gradient-tracked version, that is a simpler and useful finding.

Proposed Experiments

Stage 1: the diagnostic. This is the decisive first experiment: plots of captured momentum energy against stored numbers, for SVD versus tensor trains, across layers, training time and mode orderings, on LLaMA-60M and LLaMA-130M C4 checkpoints. It costs a few GPU-days.

Stage 2: C4 pre-training. If the diagnostic passes, we pre-train LLaMA-60M, 130M and 350M on C4 against every baseline above. We report validation perplexity at matched optimizer memory, captured-energy traces during training, throughput in tokens per second, and peak memory. We also run NanoGPT on FineWeb to compare with MoFaSGD's own setting.

Stage 3: instruction tuning. Only if MPS-MoFa improves perplexity at 350M do we run Tulu3-SFT on LLaMA-3.1-8B. The target is to narrow the reported −4.2% gap to AdamW while staying near 29.4 GB.

Negative outcomes. A flat entanglement profile at Stage 1, or a Stage 2 result where the random-ordering ablation equals the structured one, would each end the project with a clear negative answer about whether LLM momenta have area-law structure.