Most ways of getting a small, fast network still pass through a large one. Some methods train a full-size model and compress it afterwards with a singular value decomposition (SVD), which splits a weight matrix into a product of two thin matrices. Others, such as Trained Rank Pruning and its successors, keep full-size weights during training and repeatedly snap them to low rank. In both cases the full $m \times n$ weight matrices and their full-size optimizer states are held in memory for the whole run. The obvious fix is to store every layer as a product $W = UV^\top$ of two thin factors from the first step. That fix is known to train badly. The factors drift out of balance, and the thin model gets stuck early. The optimizers people use, AdamW and Muon, were also never designed with a factored layer in mind.
We propose a small wrapper that sits around any base optimizer and trains networks that are factored from start to finish. It never forms or decomposes a full weight matrix. The wrapper does three things, and all of them cost only $O((m+n)r^2)$ per layer, where $r$ is the factor width. It periodically puts each factor pair into a canonical "balanced" form. It carries the base optimizer's momentum buffers through that change of coordinates so that updates do not jump. It starts at a generous rank and shrinks it toward a target during training, the stochastic analogue of the "rank continuation" strategy from classical matrix recovery. The deliverable is a compact model whose training-time memory scales with the compressed size rather than the dense one.
Core Claim
The failure of factored-from-scratch training is mainly an optimizer–parameterization mismatch, and a wrapper can repair it. It is not an inherent capacity limit that forces a detour through dense weights.
The mismatch has two sources. First, the product $UV^\top$ is unchanged if $U$ is multiplied by any invertible $r\times r$ matrix $T$ and $V$ by $T^{-\top}$. The optimizer's trajectory, however, depends heavily on which of these equivalent factorizations it happens to sit in. Finding Low-rank Solutions to Matrix Problems, Efficiently and Provably (BFGD) handles this imbalance with a penalty $\tfrac{\lambda}{4}\|U^\top U - V^\top V\|_F^2$ inside plain gradient descent. Second, starting at the target rank puts the problem in the regime where spurious local minima exist. Unifying Nuclear Norm and Bilinear Factorization Approaches for Low-Rank Matrix Decomposition proves that factorizations are benign when the rank is at least the optimal rank $k^*$. It escapes the harder regime by starting high and reducing rank.
We claim that exact rebalancing, momentum transport, and rank continuation, applied as a wrapper around AdamW or Muon, close most of the gap to dense training at matched final parameter count. We further claim that this happens without the full-size memory that TRP, LRPET and post-hoc SVD require.
A specific, testable sub-claim concerns how the wrapper interacts with the base optimizer. Adam normalizes each coordinate separately, so it is not invariant to rotating the factor coordinates. Muon orthogonalizes whole-matrix updates and is invariant to such rotations. We therefore predict that rebalancing is nearly free under Muon but needs careful second-moment transport under Adam.
Research Program
Canonicalization. The central operation, run every $k$ steps (for example every 100), is a canonicalization that avoids any large decomposition. Take thin QR decompositions $U = Q_U R_U$ and $V = Q_V R_V$. Compute a small $r\times r$ SVD $R_U R_V^\top = A S B^\top$. Then set
$$U \leftarrow Q_U A S^{1/2}, \qquad V \leftarrow Q_V B S^{1/2},$$
which leaves $W = UV^\top$ unchanged, makes $U^\top U = V^\top V = S$ exactly, and reads off the singular values $S$ of $W$ at no extra cost. We are honest that an $r\times r$ SVD appears here. The sense in which the method is "SVD-free" is that no $m\times n$ matrix is ever formed, and that sense is what matters for memory.
Link to weight decay. In balanced form, ordinary weight decay on the factors equals the variational nuclear norm from Cabral et al., $\|W\|_* = \min_{UV^\top=W}\tfrac12(\|U\|_F^2+\|V\|_F^2)$, the sum of the singular values of $W$. Keeping the factors balanced therefore turns plain decoupled weight decay into a genuine low-rank-promoting penalty. Nothing extra is added to the loss.
Momentum transport. Any canonicalization is a change of factor coordinates $U \leftarrow UT$, $V \leftarrow VT^{-\top}$. Gradients with respect to the factors transform as $G_U \leftarrow G_U T^{-\top}$ and $G_V \leftarrow G_V T$. The wrapper applies this map exactly to first-moment buffers. This addresses a limitation named in the modular-wrapper literature: gradient-level wrappers do not track reparameterized weights, so momentum and factor scale fall out of step.
Adam's elementwise second moment has no exact transport. We will compare three options. The first resets it. The second applies the elementwise-squared transform as a heuristic. The third keeps second moments only in the canonical basis, which is re-established at every canonicalization.
Rank continuation. Each layer starts at rank $r_0$, for example $4\times$ the target. At canonicalization the wrapper drops the columns with the smallest entries of $S$. It then rescales the remaining ones to preserve the layer's Frobenius energy. This is the energy transfer of Compact Model Training by Low-Rank Projection With Energy Transfer, now computable without any large SVD because $S$ comes from the small decomposition. The corresponding momentum columns are simply truncated.
Layer-wise ranks come from one global energy budget pooled across the $S$ spectra of all layers. This replaces per-layer rank tuning, which the low-rank literature flags as a major source of sensitivity.
Scope. The wrapper touches only the factor tensors and the optimizer's state dictionary, so it drops in around unmodified AdamW or Muon. We deliberately do not pursue a Lookahead-style slow/fast wrapper on the factors. It is a plausible alternative route out of bad minima, but rank continuation has a theoretical rationale from Cabral et al. that Lookahead lacks in this setting. Lookahead may be revisited only if continuation fails.
Baselines
Dense reference. A full-rank model trained with AdamW, and separately with Muon, under the same token or epoch budget and a learning-rate sweep of equal size. It sets the accuracy ceiling and the memory and compute cost we aim to undercut.
On C4 pre-training of Llama-1B, Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation reports a validation perplexity of 15.56 for full-rank Adam, and 13.53 for its own LoRA-Pre. Both runs have dense weights. This is not a like-for-like comparison with compact models. It shows how much optimizer choice alone can move perplexity, and so how carefully the dense baseline must be tuned.
Naive factored training. Every layer is $UV^\top$ at the target rank from initialization, trained with AdamW plus weight decay. A second variant adds the BFGD balancing penalty. This is the do-nothing version of our idea and the skeptic's first suggestion. Adriana Fernandez-Lopez et al. report that this setup converges poorly, and we will measure by how much. It gets the same tuning budget, including a sweep over the penalty weight $\lambda$.
Strongest in-loop competitors. These are TRP (Trained Rank Pruning for Efficient Deep Neural Networks) and LRPET. Both train dense weights and project to low rank with SVD every few iterations (TRP) or once per epoch (LRPET), and they are the strongest published methods producing compact models from an in-loop procedure. We will run them with the same final parameter count and the same data. We will also give them randomized SVD so that their projection cost is not a straw man.
Reported numbers exist for the convolutional setting. On CIFAR-10 ResNet-56, TRP reports a 4.51x speedup at about a 1.5% Top-1 drop. On ImageNet, TRP reports 72.69% Top-1 for ResNet-50 at a 2.30x speedup. Also on CIFAR-10 ResNet-56, the from-scratch Tucker-factored ELRT (Towards Efficient Low-Rank Training for Compact Neural Networks) reports a 0.75% accuracy gain over the dense baseline at a 2.52x FLOPs reduction. The file gives no exact reported numbers for LRPET.
For Transformer language models on C4, no reported result found on Lacuna for from-scratch low-rank-weight training. There we will run TRP and LRPET ourselves.
Dense-then-compress. Train dense with GaLore (GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection) or with Trion/DCT-AdamW (SVD-Free Low-Rank Adaptive Gradient Optimization for Large Language Models). Then apply one post-hoc SVD and fine-tune with a fine-tuning budget equal to 10% of training. Those papers already save optimizer memory, and this pipeline is the most natural way to reuse them for a compact model.
Ablations. Four ablations isolate the key ingredients. The first removes momentum transport. The second removes rank continuation, training at fixed target rank with canonicalization only. The third removes energy transfer. The fourth swaps the base optimizer between AdamW and Muon.
What counts as a win. Beating a baseline means lower validation perplexity, or higher Top-1, at the same final parameter count and training compute. Against TRP and LRPET the win condition is an additional one: matching their accuracy while holding peak training memory near the compressed size rather than the dense size.
Why not just…?
Why not just train $UV^\top$ with AdamW and weight decay, maybe with the BFGD balancing penalty? This is the most serious threat, because in balanced form weight decay already acts as a nuclear-norm penalty. What argues against it is evidence rather than theory. Fernandez-Lopez et al. report poor convergence from scratch. BFGD's guarantees assume spectral initialization and plain gradient descent, not adaptive optimizers with momentum. A penalty only pushes toward balance, whereas our canonicalization makes it exact and moves the momentum along with it. The naive and penalized baselines, run at equal tuning budget, settle this directly. If either comes within noise of the full wrapper, the paper's conclusion becomes "balancing suffices," and we will report that.
Why not just use TRP or LRPET with a fast randomized SVD? For Transformer-sized matrices, a once-per-epoch SVD is probably cheap, so SVD cost alone is a weak argument against them. The stronger argument is memory. Both methods keep dense weights and dense Adam states for the whole run, so they save nothing at training time. The comparison therefore reports accuracy alongside peak training memory. If they win on accuracy by a clear margin, our method's value reduces to its memory savings, and we will state that trade-off explicitly.
Why not just train dense with GaLore or LoRA-Pre and compress once at the end? Prior work argues that post-hoc SVD minimizes reconstruction error rather than task loss (Andrea Vedaldi et al., Jeffrey T. H. Wong et al.). LRPET also argues that the best low-rank solution lies far from the dense one. Neither argument has been tested against modern memory-efficient dense optimizers followed by fine-tuning. The dense-then-compress baseline tests exactly that, at several compression ratios. We expect it to be competitive at mild compression and to fall behind at aggressive compression.
Why not just fix the rank and skip continuation? The fixed-rank ablation answers this. Cabral et al. predict the gap should grow as the target rank falls further below what the task needs. We will check that the benefit of continuation increases with the compression ratio.
Proposed Experiments
Language modeling. The main testbed is Llama-family pre-training on C4 at 60M, 350M and 1B parameters, following the setups used by LoRA-Pre and Trion. We will train compact models at 25% and 50% of dense parameter count and report validation perplexity, peak GPU memory, and wall-clock time.
Vision. CIFAR-10 with ResNet-20 and ResNet-56 and ImageNet with ResNet-50 allow direct comparison with TRP's and ELRT's published numbers. Convolution kernels will be reshaped into matrices, as TRP does. These results will be reported at FLOPs reductions matched to the published ones.
Decisive check 1. Under AdamW, removing momentum transport should produce visible loss spikes at canonicalization steps and worse final perplexity. Under Muon, the same removal should matter much less. Confirming this asymmetry would support the mechanism, not just the method.
Decisive check 2. Rank continuation should beat fixed-rank training by a margin that widens at 25% parameters relative to 50%.
Decisive check 3. At 1B scale, the full wrapper should match or beat TRP and LRPET on perplexity at equal final size, with peak memory well below dense AdamW. If check 1 fails, the momentum-transport contribution is not supported. If check 3 fails on accuracy, the contribution narrows to memory.
References
- Finding Low-rank Solutions to Matrix Problems, Efficiently and Provably
- Unifying Nuclear Norm and Bilinear Factorization Approaches for Low-Rank Matrix Decomposition
- Compact Model Training by Low-Rank Projection With Energy Transfer
- Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation
- Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
- Trained Rank Pruning for Efficient Deep Neural Networks
- ELRT: Towards Efficient Low-Rank Training for Compact Neural Networks
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
- SVD-Free Low-Rank Adaptive Gradient Optimization for Large Language Models
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- A3: an Analytical Low-Rank Approximation Framework for Attention