Factorizing neural network layers—representing a large weight matrix as the product of two or more smaller matrices—is a popular strategy for model compression and efficient training. However, these factorized layers are notoriously difficult to train from scratch. Standard deep learning tricks, such as He initialization or standard weight decay, do not translate seamlessly to the individual matrix factors.
This paper introduces two simple but powerful techniques—Spectral Initialization (SI) and Frobenius Decay (FD)—to bridge this gap. By aligning the optimization of decomposed layers with their well-tuned, non-factorized counterparts, these methods achieve state-of-the-art results in model compression, "teacher-free" knowledge distillation, and Transformer performance.
The Challenge of Factorized Layers
In deep learning, replacing a large weight matrix $W$ with a factorization like $W = UV^T$ serves two main purposes:
- Compression: If $U$ and $V$ are thin (low-rank approximation), memory and computational requirements drop significantly.
- Over-parameterization: Adding "redundant" factors can sometimes smooth the optimization landscape, improving generalization in a manner similar to knowledge distillation.
Despite these benefits, standard regularization on $U$ and $V$ doesn't penalize the complexity of the resulting operation $W$ in the same way it would for a single matrix. This mismatch often leads to suboptimal training dynamics.
Proposed Methods: SI and FD
To resolve the training difficulties of factorized layers, the authors propose two primary modifications to the training pipeline:
- Spectral Initialization (SI): Instead of initializing $U$ and $V$ with independent random noise, SI starts with a standard initialization for the full matrix $W$. It then uses Singular Value Decomposition (SVD) to find the $U$ and $V$ that best approximate $W$. This ensures the factorized layer starts in a "well-behaved" state, mimicking a standard layer.
- Frobenius Decay (FD): Standard weight decay penalizes the sum of squared weights of the individual factors ($\|U\|_F^2 + \|V\|_F^2$). The authors demonstrate that this mathematically equates to penalizing the nuclear norm of the product $UV^T$, which is often overly restrictive. FD instead penalizes the squared Frobenius norm of the product itself ($\|UV^T\|_F^2$), perfectly matching the regularization behavior of non-factorized layers.
The Nuclear Norm Trap and Key Insights
The most surprising theoretical insight from the paper is the "Nuclear Norm Trap." Naive weight decay on factorized layers implicitly regularizes the nuclear norm (the sum of singular values). This aggressively encourages the matrix to be extremely low-rank, which can stifle learning if the task requires higher complexity.
By switching to Frobenius Decay, the network avoids this trap. Furthermore, in networks utilizing Batch Normalization, the magnitude of weights affects gradient updates rather than the output. FD helps maintain a healthy "effective step-size" during training, preventing the weights from growing in a way that would cause the effective learning rate to vanish.
Finally, the authors highlight Teacher-Free Distillation. By using "overcomplete" factorizations (where $U$ and $V$ are larger than strictly necessary) combined with FD, models can achieve the accuracy benefits typically associated with knowledge distillation, but without needing a pre-trained teacher model.
Key Results and Performance
The researchers evaluated SI and FD across several domains, including Computer Vision (CIFAR-10/100, Tiny-ImageNet), Machine Translation (IWSLT-14), and Language Modeling (BERT on SQuAD).
- Beating Sparsity: On ResNet architectures, low-rank factorization equipped with SI and FD outperformed popular "unstructured sparsity" methods (like the Lottery Ticket Hypothesis) for model compression.
- Efficiency Gains: An "overcomplete" ResNet56 trained with FD outperformed a standard ResNet110 baseline. Remarkably, it was 1.5x faster to train and required 2x fewer parameters at inference time.
- Improving Transformers: Applying FD to the Multi-Head Attention (MHA) layers in Transformers yielded consistent gains in BLEU scores for translation and improved F1/Exact Match scores for BERT on SQuAD.
- FLAMBé Optimizer: The authors introduced FLAMBé (Frobenius-regularized LAMB), a variant of the LAMB optimizer that improved BERT pre-training performance for both full-rank and low-rank models.
Limitations and Future Directions
While highly effective, the methods exhibit some architecture sensitivity. The gains on VGG architectures were less pronounced than on ResNets, suggesting the benefits might rely partially on residual connections or specific normalization layers. Additionally, SI had a minimal effect on Multi-Head Attention layers, likely because standard initializations for MHA are already highly specialized.
Future work may explore applying SI and FD to other structured layers, such as ACDC or K-matrices, and integrating them with adaptive rank-selection schemes to further automate efficient network design.
Summary updated