Abstract
Low-rank matrix factorization is a promising technique for compressing Deep Neural Networks (DNNs), yet it often suffers from the computational complexity of rank selection and significant accuracy loss. We introduce the BSR (Beam-search and Stable Rank) algorithm, a three-phase framework that automates rank selection using a modified beam-search (mBS) and preserves model performance through a novel modified stable rank (mSR) regularization. By concentrating signal in the most significant dimensions and penalizing the 'tail' singular values, BSR enables aggressive compression while maintaining or even improving original model accuracy. Our evaluations on MNIST, CIFAR, and ImageNet demonstrate that BSR outperforms existing low-rank and pruning methods, offering a streamlined pipeline with minimal hyperparameter tuning.
Introduction
Deep Neural Networks (DNNs) are essential for modern AI, but their high memory and computational demands limit deployment on edge devices. Low-rank compression approximates large weight matrices as products of smaller ones via Singular Value Decomposition (SVD), but it faces two major hurdles: the combinatorial 'Rank Selection' problem and the performance gap between uncompressed and factorized models. Traditional methods often rely on suboptimal heuristics or complex, multi-parameter training schedules that are difficult to tune.
The BSR Methodology
The BSR algorithm addresses these challenges through a structured three-phase process designed to make low-rank compression both automated and high-performing.
- Phase 1: Rank Selection via Modified Beam-Search (mBS): Instead of manual guessing, BSR uses a search algorithm to find the optimal rank configuration across all layers. A 'level step size' ($s$) is introduced to skip search levels, making it computationally feasible to find high-performing rank vectors that meet a specific global compression target.
- Phase 2: Rank Regularized Training via Modified Stable Rank (mSR): To prepare the network for factorization, a regularization term is added to the loss function. Unlike standard stable rank, mSR specifically penalizes energy in the 'tail' dimensions (those to be discarded) while encouraging signal concentration in the 'top' dimensions (those to be kept).
- Phase 3: Decomposition & Fine-tuning: Weights are factorized via SVD using the Phase 1 ranks, followed by standard fine-tuning to recover any residual accuracy loss.
Optimizing Rank Selection and Signal Concentration
A key insight of this work is that calculating optimal ranks before compression-friendly training is more effective than dynamic updates. The mSR regularization is particularly powerful because it forces the network to learn features that are inherently low-rank, making the subsequent SVD step much less destructive. To ensure training efficiency, the authors utilize randomized matrix decomposition, keeping training time nearly identical to standard uncompressed training.
Experimental Results
BSR was evaluated across standard benchmarks including MNIST, CIFAR-10/100, and ImageNet. On ResNet56 (CIFAR-100), BSR's rank selection significantly outperformed existing methods like CA and LC. In several instances, BSR actually improved base model accuracy; for example, a ResNet56 on CIFAR-10 saw accuracy rise from 92.73% to 93.53% even after a 44% reduction in FLOPs. Furthermore, BSR proved competitive with state-of-the-art pruning methods, achieving higher accuracy gains at similar compression levels. When combined with 8-bit quantization, BSR achieved up to a 17x reduction in memory usage with only a 1% drop in accuracy.
Conclusion
The BSR algorithm provides a robust, automated solution for DNN compression that bridges the gap between low-rank factorization and pruning. By leveraging efficient search and targeted regularization, it simplifies the compression pipeline while delivering state-of-the-art results. Future work will explore more advanced rank surrogates and the potential for combining BSR with other compression techniques sequentially.
Summary updated