Initialization and Regularization of Factorized Neural Layers

Mikhail Khodak, Neil A. Tenenholtz, Lester Mackey, Nicolò Fusi

Factorizing neural network layers—representing a large weight matrix as the product of two or more smaller matrices—is a popular strategy for model compression and efficient training. However, these factorized layers are notoriously difficult to train from scratch. Standard deep learning tricks, such as He initialization or standard weight decay, do not translate seamlessly to the individual matrix factors.

This paper introduces two simple but powerful techniques—Spectral Initialization (SI) and Frobenius Decay (FD)—to bridge this gap. By aligning the optimization of decomposed layers with their well-tuned, non-factorized counterparts, these methods achieve state-of-the-art results in model compression, "teacher-free" knowledge distillation, and Transformer performance.

The Challenge of Factorized Layers

In deep learning, replacing a large weight matrix $W$ with a factorization like $W = UV^T$ serves two main purposes:

  1. Compression: If $U$ and $V$ are thin (low-rank approximation), memory and computational requirements drop significantly.
  2. Over-parameterization: Adding "redundant" factors can sometimes smooth the optimization landscape, improving generalization in a manner similar to knowledge distillation.

Despite these benefits, standard regularization on $U$ and $V$ doesn't penalize the complexity of the resulting operation $W$ in the same way it would for a single matrix. This mismatch often leads to suboptimal training dynamics.

Proposed Methods: SI and FD

To resolve the training difficulties of factorized layers, the authors propose two primary modifications to the training pipeline:

  • Spectral Initialization (SI): Instead of initializing $U$ and $V$ with independent random noise, SI starts with a standard initialization for the full matrix $W$. It then uses Singular Value Decomposition (SVD) to find the $U$ and $V$ that best approximate $W$. This ensures the factorized layer starts in a "well-behaved" state, mimicking a standard layer.
  • Frobenius Decay (FD): Standard weight decay penalizes the sum of squared weights of the individual factors ($\|U\|_F^2 + \|V\|_F^2$). The authors demonstrate that this mathematically equates to penalizing the nuclear norm of the product $UV^T$, which is often overly restrictive. FD instead penalizes the squared Frobenius norm of the product itself ($\|UV^T\|_F^2$), perfectly matching the regularization behavior of non-factorized layers.

The Nuclear Norm Trap and Key Insights

The most surprising theoretical insight from the paper is the "Nuclear Norm Trap." Naive weight decay on factorized layers implicitly regularizes the nuclear norm (the sum of singular values). This aggressively encourages the matrix to be extremely low-rank, which can stifle learning if the task requires higher complexity.

Nuclear norm evolution during training

By switching to Frobenius Decay, the network avoids this trap. Furthermore, in networks utilizing Batch Normalization, the magnitude of weights affects gradient updates rather than the output. FD helps maintain a healthy "effective step-size" during training, preventing the weights from growing in a way that would cause the effective learning rate to vanish.

Finally, the authors highlight Teacher-Free Distillation. By using "overcomplete" factorizations (where $U$ and $V$ are larger than strictly necessary) combined with FD, models can achieve the accuracy benefits typically associated with knowledge distillation, but without needing a pre-trained teacher model.

Key Results and Performance

The researchers evaluated SI and FD across several domains, including Computer Vision (CIFAR-10/100, Tiny-ImageNet), Machine Translation (IWSLT-14), and Language Modeling (BERT on SQuAD).

Validation accuracy comparing Frobenius decay and standard weight decay
  • Beating Sparsity: On ResNet architectures, low-rank factorization equipped with SI and FD outperformed popular "unstructured sparsity" methods (like the Lottery Ticket Hypothesis) for model compression.
  • Efficiency Gains: An "overcomplete" ResNet56 trained with FD outperformed a standard ResNet110 baseline. Remarkably, it was 1.5x faster to train and required 2x fewer parameters at inference time.
  • Improving Transformers: Applying FD to the Multi-Head Attention (MHA) layers in Transformers yielded consistent gains in BLEU scores for translation and improved F1/Exact Match scores for BERT on SQuAD.
  • FLAMBé Optimizer: The authors introduced FLAMBé (Frobenius-regularized LAMB), a variant of the LAMB optimizer that improved BERT pre-training performance for both full-rank and low-rank models.

Limitations and Future Directions

While highly effective, the methods exhibit some architecture sensitivity. The gains on VGG architectures were less pronounced than on ResNets, suggesting the benefits might rely partially on residual connections or specific normalization layers. Additionally, SI had a minimal effect on Multi-Head Attention layers, likely because standard initializations for MHA are already highly specialized.

Future work may explore applying SI and FD to other structured layers, such as ACDC or K-matrices, and integrating them with adaptive rank-selection schemes to further automate efficient network design.

Summary updated

Abstract

Factorized layers--operations parameterized by products of two or more matrices--occur in a variety of deep learning contexts, including compressed model training, certain types of knowledge distillation, and multi-head self-attention architectures. We study how to initialize and regularize deep nets containing such layers, examining two simple, understudied schemes, spectral initialization and Frobenius decay, for improving their performance. The guiding insight is to design optimization routines for these networks that are as close as possible to that of their well-tuned, non-decomposed counterparts; we back this intuition with an analysis of how the initialization and regularization schemes impact training with gradient descent, drawing on modern attempts to understand the interplay of weight-decay and batch-normalization. Empirically, we highlight the benefits of spectral initialization and Frobenius decay across a variety of settings. In model compression, we show that they enable low-rank methods to significantly outperform both unstructured sparsity and tensor methods on the task of training low-memory residual networks; analogs of the schemes also improve the performance of tensor decomposition techniques. For knowledge distillation, Frobenius decay enables a simple, overcomplete baseline that yields a compact model from over-parameterized training without requiring retraining with or pruning a teacher network. Finally, we show how both schemes applied to multi-head attention lead to improved performance on both translation and unsupervised pre-training.
BibTeX
@inproceedings{khodak2021initialization,
  author = {Mikhail Khodak and Neil A. Tenenholtz and Lester Mackey and Nicolò Fusi},
  title = {{Initialization and Regularization of Factorized Neural Layers}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
}
Figure