Modern deep learning often follows a "train then compress" pipeline: researchers train a massive, over-parameterized model and then use techniques like Singular Value Decomposition (SVD) to prune redundant weights. However, this post-hoc approach often leads to a "performance cliff" where aggressive compression causes catastrophic accuracy loss.
A growing body of research suggests a more integrated path: Low-Rank Training. Instead of treating compression as an afterthought, these methods embed low-rank constraints directly into the training process. By forcing the model to learn within a constrained mathematical manifold from the start, researchers can achieve massive parameter reductions—sometimes exceeding 30,000x—while maintaining or even improving performance.
The Problem with Post-Training Compression
In the era of "Red AI," characterized by Eva Memmel and Clara Menzen as a paradigm where accuracy is chased regardless of compute cost, models are inherently "fat." When we apply SVD to a pre-trained weight matrix, we are essentially trying to find a low-rank approximation of a high-rank structure that was never optimized for it.
As Samridhi Choudhary notes in the context of Factorization-Aware Training (FAT), traditional post-training methods fail because the weights haven't adapted to the lower-rank structure. The model's intelligence is scattered across the full-rank space; cutting it down after the fact is like trying to fit a square peg in a round hole.
Direct Optimization of Tensor Cores
One solution is to bypass the "fat" model entirely. Instead of a standard weight matrix $W$, we represent the layer as a product of smaller "tensor cores." This is the core intuition behind Direct Factorized Training.
In TT-PINN, Ziyue Liu and Zheng Zhang demonstrate that Physics-Informed Neural Networks can be trained from scratch using Tensor-Train decomposition. By optimizing the parameters of these compressed cores directly, the memory footprint remains low throughout the entire learning process, enabling training on resource-constrained edge devices.
Similarly, Yu Pan introduced TR-LSTM, which uses Tensor Ring Decomposition to replace standard linear layers with a circular chain of tensor cores. Because the model is defined in this compressed format from initialization, it can be optimized end-to-end. This approach achieved a staggering 34,000x compression ratio on video tasks, proving that models don't need to be "big" to be "smart."
Integrating Rank Constraints into the SGD Loop
If training from scratch in a compressed format is too restrictive, a hybrid approach involves training a full-rank model while periodically "nudging" it toward a low-rank state.
Trained Rank Pruning (TRP), developed by Yuxi Li and Wei Wen, integrates low-rank approximation directly into the Stochastic Gradient Descent (SGD) loop. The process alternates between:
- Standard Gradient Updates: Allowing the model to utilize its full representational power to learn complex features.
- Low-Rank Projection: Periodically applying SVD to the weights to enforce a rank constraint.
This "soft" constraint allows the network to gradually adapt its weight distribution. By maintaining a full-rank parameter space between compression steps, Wei Wen argues that we avoid the vanishing gradient problems that plague "thin" architectures while ensuring the final model is natively optimized for efficient hardware.
Encouraging Compressibility via Regularization
Another way to build low-rank models is to make the weights "want" to be compressed. This is often achieved through specialized regularization terms that penalize high-rank structures.
- Force Regularization: Wei Wen and Yiran Chen proposed using "attractive forces" to coordinate filters during training. By making filters more correlated, the weight space naturally becomes lower-rank, making subsequent SVD much more effective.
- Smoothness Constraints: Christina Runkel and Michael Moeller showed that applying derivative-based smoothness regularization during training allows for 70% parameter reduction via SVD without any post-hoc fine-tuning. The smoothness ensures that the "energy" of the weight matrix is concentrated in a few principal components.
Progressive and Iterative Squeezing
For massive models like LLMs, jumping straight to a low-rank representation can be too disruptive. Habib Hajimolahoseini and Yang Liu introduced Progressive Low Rank Decomposition (PLRD), which spawns smaller "child" models from a "parent" model through incremental steps. Each reduction is followed by a brief fine-tuning phase to stabilize the model.
This geometric perspective is echoed by Vasileios Tsiaras, who frames compression as a projection problem. He argues that "softer" rank-reduction schedules are the secret to extreme compression because they preserve the model's "trainability," allowing it to navigate the optimization landscape toward a low-rank manifold without losing its learned intelligence.
Why It Works: The Plasticity of Training
The recurring theme across these 55 papers is that plasticity matters. A model that is still learning can compensate for the "damage" of rank reduction. As Mostafa Elhoushi found, decomposing a model during training yields superior accuracy compared to doing it after training.
By integrating low-rank structures—whether through Kronecker products (Shuchang Zhou), basis filters (Yani Ioannou), or low-rank activations (Ziyue Liu in CoLA) — we move toward a "Green AI" future where efficiency is a first-class citizen of the training process.