Deep neural networks are notoriously heavy. A standard linear layer computing $y = Wx$ with a weight matrix $W \in \mathbb{R}^{m \times n}$ requires storing $m \times n$ parameters and performing $m \times n$ multiply-accumulate operations. When $m$ and $n$ scale into the thousands—as they do in modern Large Language Models (LLMs), Automatic Speech Recognition (ASR) systems, and Vision-Language Models (VLMs)—the computational and memory footprint becomes a massive bottleneck for real-time and edge deployment.
However, trained weight matrices are highly redundant. While they live in a high-dimensional ambient space, their actual information often lies on a much lower-dimensional subspace. In linear algebra terms, the matrix $W$ has a low intrinsic rank $r$, where $r \ll \min(m, n)$. Low-rank matrix factorization exploits this property to compress networks by breaking one large, expensive matrix operation into two smaller, highly efficient ones.
The Core Mechanism: Splitting the Matrix
If a matrix $W$ of size $m \times n$ has an approximate rank $r$, it can be factorized into the product of two thin matrices:
$$W \approx U \cdot V$$
where $U \in \mathbb{R}^{m \times r}$ and $V \in \mathbb{R}^{r \times n}$. Instead of computing the layer transformation as $y = Wx$, we rewrite the operation as:
$$y = U(Vx)$$
Standard Layer:
[ W ] x [x] -> [y]
(m x n)
Low-Rank Restructured Layer:
[ V ] x [x] -> [inter] -> [ U ] x [inter] -> [y]
(r x n) (m x r)
This simple algebraic restructuring changes the computational dynamics completely:
- Parameter Count: Drops from $m \cdot n$ to $r \cdot (m + n)$.
- Computational Complexity (FLOPs): Drops from $O(mn)$ to $O(r(m+n))$ per forward pass.
For example, if $m = 4096$ and $n = 4096$, the original layer contains roughly 16.7 million parameters. If we find that a rank of $r = 512$ preserves the matrix's core behavior, the factorized layers require only $512 \times (4096 + 4096) \approx 4.2$ million parameters—a 75% reduction in both memory and compute.
Post-Training Decomposition via SVD
The most direct way to find $U$ and $V$ for an already trained model is through Singular Value Decomposition (SVD). SVD factorizes any matrix into $W = P \Sigma Q^T$, where $P$ and $Q$ are orthogonal matrices and $\Sigma$ is a diagonal matrix containing the singular values sorted by magnitude. By keeping only the top $r$ singular values and discarding the rest (singular value truncation), we obtain the optimal low-rank approximation of that matrix.
As demonstrated in early foundational work on acoustic models by Jinyu Li et al. in Restructuring of deep neural network acoustic models with singular value decomposition and by Yonghong Yan in Speeding up deep neural network based speech recognition systems, this process replaces a single high-dimensional layer with two successive linear layers. This effectively introduces a narrow "bottleneck" or projection layer into the architecture, as also detailed by Ruoming Pang et al. in Compression of End-to-End Models.
Because naive truncation discards subtle but collectively important weight dynamics, a post-processing compression step is typically required to recover performance. Matan Ben Noach in Compressing Pre-trained Language Models by Matrix Decomposition shows that combining SVD with feature distillation—where the compressed model is explicitly trained to mimic the internal activations of the original full-rank network—allows for aggressive parameter reduction with minimal accuracy loss.
Data-Driven and Layer-Wise Adaptation
Not all layers in a neural network are equally redundant. Some layers can be compressed heavily without issue, while others are highly sensitive to any loss of rank.
To navigate this, frameworks like ShrinkML, introduced by Łukasz Dudziak et al. in ShrinkML: End-to-End ASR Model Compression Using Reinforcement Learning, leverage reinforcement learning to automatically discover the optimal per-layer compression ratios. Similarly, Xin Wang et al. show in SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression that assigning unique, sensitivity-aware ranks across different layers prevents the catastrophic accuracy degradation often seen in uniform truncation.
Furthermore, analyzing a weight matrix in isolation ignores how data actually flows through the network during inference. To bridge this gap, Arnav Chavan et al. develop a training-free approach called LLM-ROM in RETHINKING COMPRESSION: REDUCED ORDER MODELLING OF LATENT FEATURES IN LARGE LANGUAGE MODELS. Instead of performing SVD directly on the static weights, they pass a small calibration dataset through the model, compute the feature covariance matrices of the activations, and use eigenvalue decomposition to find the principal components of the latent space. Re-parameterizing the weights based on these active data pathways ensures that the low-rank factorization preserves the network's actual representational capacity.
Training Directly in Low-Rank Subspaces
Instead of training a massive model and compressing it post-hoc, can we design networks to be low-rank from day one?
One approach is to train "pre-factorized" networks. Hongyi Wang et al. introduce Pufferfish in Pufferfish: Communication-efficient Models At No Extra Cost, a distributed training framework where layers are structured as thin factor matrices from the start. This directly bypasses communication bottlenecks in distributed clusters, as nodes only need to transmit smaller, low-rank gradient matrices during synchronization.
Alternatively, we can encourage standard architectures to learn low-rank weights naturally during training. Ismail Alkhouri et al. propose LoRITa in Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition. By replacing a single weight layer with a composition of multiple linear layers and applying standard weight decay, the network is regularized to concentrate its information into a tiny fraction of its singular values. Post-training compression then becomes a trivial task of truncating the near-zero singular values, eliminating the need for expensive manual rank selection or extensive fine-tuning.
Why Low-Rank Factorization Wins on Hardware
When compared to other popular compression paradigms like pruning or quantization, low-rank factorization offers a distinct structural advantage.
Pruning introduces sparsity by setting individual weights to zero. While this reduces theoretical FLOPs, it results in irregular memory access patterns. Without specialized hardware or sparse linear algebra libraries, standard hardware spends cycles processing zeros, meaning theoretical compute savings rarely translate to real-world speedups. Quantization reduces numerical precision (e.g., from FP16 to INT4), which can degrade model capabilities if pushed too low and requires hardware support for low-bit arithmetic.
In contrast, low-rank factorization preserves dense matrix operations. The resulting matrices $U$ and $V$ are standard, dense linear layers that run out-of-the-box on any standard hardware accelerator (GPUs, CPUs, or mobile chips). As surveyed by Gaurav Shinde et al. in A Survey on Efficient Vision-Language Models, this structural optimization often yields superior latency reductions on edge devices because it directly minimizes the total number of floating-point operations required by the underlying architecture, rather than just altering the precision or density of the existing ones.
Key Foundational Work
Restructuring of deep neural network acoustic models with singular value decomposition (2013) — contributed 1 concept elements to this direction.
Introduction to the Special Section on Large-Scale Optimization for Audio, Speech, and Language Processing (2013) — contributed 1 concept elements to this direction.
Speeding up deep neural network based speech recognition systems (2014) — contributed 1 concept elements to this direction.