Deep neural networks are notoriously overparameterized. To deploy them on edge devices or reduce training overhead, researchers frequently turn to low-rank matrix or tensor decompositions. Traditionally, this compression is treated as a post-processing step: a massive, full-rank model is trained to completion, and Singular Value Decomposition (SVD) is applied afterward to split large weight matrices into compact, low-rank products.
However, this post-hoc factorization is fundamentally suboptimal. As noted early on by Andrea Vedaldi et al., compressing a pre-trained model minimizes the geometric reconstruction error of the weight matrices rather than the task-specific loss function. This mismatch creates a performance ceiling; aggressive post-training dimension reduction inevitably leads to a catastrophic drop in accuracy, a limitation highlighted by Jeffrey T. H. Wong et al. and Samridhi Choudhary et al..
Why not simply train a low-rank architecture from day one? If we parameterize the weights as low-rank products ($W = U \times V$) from the very first gradient step, we encounter a different bottleneck. As Adriana Fernandez-Lopez et al. observe, training thin, pre-decomposed architectures from scratch typically results in poor convergence and severe accuracy degradation. The restricted parameter space limits the model's capacity to navigate the complex optimization landscape during early training phases.
To resolve this tension, a powerful paradigm has emerged: in-loop low-rank training. Instead of choosing between full-rank exploration and low-rank constraints, these methods embed the low-rank constraint directly into the optimization loop.
Iterative Low-Rank Projection
One prominent mechanism to balance exploration and constraint is iterative low-rank projection. In frameworks like Trained Rank Pruning (TRP), developed by Yuxi Li et al. and extended by Wei Wen et al., the model maintains a full-rank parameter space during the forward and backward passes. However, at regular intervals during stochastic gradient descent, a low-rank projection operator is applied to the weights.
Standard Training Step:
[Full-Rank Weights W] ---> Gradient Update ---> [Updated Full-Rank W']
In-Loop Projection Step (Periodic):
[Updated Full-Rank W'] ---> SVD / Projection ---> [Rank-Constrained W_low]
This alternating strategy provides a crucial intuitive advantage: the optimizer utilizes the high learning capacity and fluid gradient flow of a full-rank space to escape local minima, but the periodic projections gradually force the weights to adapt to a low-rank structure. By the end of training, the network learns features that are inherently compatible with a compressed representation, avoiding compounding approximation errors.
Rather than hard projections, this can also be framed as a continuous optimization problem. Miao Yin introduced an optimization framework using the Alternating Direction Method of Multipliers (ADMM) to gradually impose low-rank tensor constraints during training. By relaxing the constraint into a soft, progressive penalty, the network retains its representational capacity early on and smoothly transitions into a compressed state.
Miao Yin later extended this to high-order tensor decompositions (HODEC), showing that optimizing contractions dynamically minimizes information loss. Similarly, Huy Phan et al. utilized gradual regularization to preserve adversarial robustness alongside compactness, proving that structured low-rankness can be baked into the model without sacrificing stability.
Learning Low-Rank Factors from Scratch
An alternative to projecting full-rank matrices is to parameterize the network using low-rank components from the start, but with specialized architectural stabilizers or regularization terms that bypass typical optimization bottlenecks.
For instance, Cheng Tai et al. demonstrated that training low-rank tensor architectures from scratch becomes highly effective when integrated with Batch Normalization. Because decomposing a layer increases the structural depth of the network (turning one operation into two sequential ones), batch normalization acts as a vital stabilizer to prevent vanishing or exploding gradients. Along similar lines, Yani Ioannou et al. designed convolutional networks using low-rank basis filters from the start, optimizing the basis and mixing weights simultaneously.
To make this work without a full-rank fallback, researchers often use explicit structural regularizers. Huanrui Yang et al. proposed "SVD Training," where the components of the decomposition ($U, s, V$) are trained directly. They applied orthogonality regularization to $U$ and $V$ alongside sparsity-inducing penalties on the singular values $s$, allowing the network to dynamically prune its own rank during training.
This direct optimization ensures that the latent core tensors are inherently adapted to the task from day one. Jean Kossaifi et al. pushed this concept to its logical conclusion with T-Net, parameterizing an entire fully convolutional network as a single high-order tensor trained end-to-end. This captures global architectural correlations that layer-by-layer compression misses.
This capacity to learn optimal low-rank structures from scratch is further supported by Jun Qi et al. in speech enhancement and Chao Li et al., who proved that "low-rankness" is an inherent property of neural network learning dynamics rather than a mere reflection of spatial data correlations.
Subspace Restrictions and Architectural Bottlenecks
A third facet of this research direction involves explicitly restricting weight updates to a low-rank subspace throughout the training lifecycle.
In federated learning, where communication bandwidth is the primary bottleneck, Ngoc-Hieu Nguyen et al. introduced a framework that keeps a large projection matrix fixed and only optimizes a smaller, lightweight matrix during local training. Because the update itself is structurally confined to a low-rank subspace, communication costs drop dramatically without requiring post-hoc compression.
Federated Subspace Update:
Weight Update ΔW = W_fixed × M_lightweight
(Only M_lightweight is optimized and communicated)
In recommendation systems, Alex Shtoff et al. integrated a Diagonal Plus Low-Rank (DPLR) structure directly into the architecture of Factorization Machines, allowing the model to learn the most informative low-rank representations directly through end-to-end task optimization. Finally, Junyu Leng et al. introduced adaptive low-rank structures (AdaRL) that dynamically adjust bottleneck linear layers during a bi-level optimization process, matching the model's rank to the intrinsic dimension of the task.
Why It Works
Whether applied to mobile computer vision systems like DeepFont by Zhangyang Wang et al. or large-scale transformers, the core intuition of in-loop low-rank training remains identical. By incorporating the low-rank constraint directly into the training loop—whether through iterative projection, stabilized direct factorization, or subspace-restricted updates—the network can dynamically redistribute information across its remaining parameters during the optimization process itself. The resulting models achieve massive parameter reductions and inference speedups because their weights were born and raised in a low-rank world.
Key Foundational Work
DeepFont: Identify Your Font from An Image (2015) — contributed 2 concept elements to this direction.
Speeding up Convolutional Neural Networks with Low Rank Expansions (2014) — contributed 1 concept elements to this direction.