A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank

Moonjung Eo, Suhyun Kang, Wonjong RheearXiv 2021 · arXiv

Abstract

Low-rank matrix factorization is a promising technique for compressing Deep Neural Networks (DNNs), yet it often suffers from the computational complexity of rank selection and significant accuracy loss. We introduce the BSR (Beam-search and Stable Rank) algorithm, a three-phase framework that automates rank selection using a modified beam-search (mBS) and preserves model performance through a novel modified stable rank (mSR) regularization. By concentrating signal in the most significant dimensions and penalizing the 'tail' singular values, BSR enables aggressive compression while maintaining or even improving original model accuracy. Our evaluations on MNIST, CIFAR, and ImageNet demonstrate that BSR outperforms existing low-rank and pruning methods, offering a streamlined pipeline with minimal hyperparameter tuning.

Introduction

Deep Neural Networks (DNNs) are essential for modern AI, but their high memory and computational demands limit deployment on edge devices. Low-rank compression approximates large weight matrices as products of smaller ones via Singular Value Decomposition (SVD), but it faces two major hurdles: the combinatorial 'Rank Selection' problem and the performance gap between uncompressed and factorized models. Traditional methods often rely on suboptimal heuristics or complex, multi-parameter training schedules that are difficult to tune.

The BSR Methodology

The BSR algorithm addresses these challenges through a structured three-phase process designed to make low-rank compression both automated and high-performing.

  1. Phase 1: Rank Selection via Modified Beam-Search (mBS): Instead of manual guessing, BSR uses a search algorithm to find the optimal rank configuration across all layers. A 'level step size' ($s$) is introduced to skip search levels, making it computationally feasible to find high-performing rank vectors that meet a specific global compression target.
  2. Phase 2: Rank Regularized Training via Modified Stable Rank (mSR): To prepare the network for factorization, a regularization term is added to the loss function. Unlike standard stable rank, mSR specifically penalizes energy in the 'tail' dimensions (those to be discarded) while encouraging signal concentration in the 'top' dimensions (those to be kept).
  3. Phase 3: Decomposition & Fine-tuning: Weights are factorized via SVD using the Phase 1 ranks, followed by standard fine-tuning to recover any residual accuracy loss.
The three-phase BSR framework: (1) Rank selection via mBS, (2) Rank regularized training via mSR, and (3) Decomposition and fine-tuning.

Optimizing Rank Selection and Signal Concentration

A key insight of this work is that calculating optimal ranks before compression-friendly training is more effective than dynamic updates. The mSR regularization is particularly powerful because it forces the network to learn features that are inherently low-rank, making the subsequent SVD step much less destructive. To ensure training efficiency, the authors utilize randomized matrix decomposition, keeping training time nearly identical to standard uncompressed training.

Comparison of rank selection methods (left) and the resulting singular value distribution (right). BSR maintains higher accuracy at various compression ratios and effectively concentrates signal in the top singular values.

Experimental Results

BSR was evaluated across standard benchmarks including MNIST, CIFAR-10/100, and ImageNet. On ResNet56 (CIFAR-100), BSR's rank selection significantly outperformed existing methods like CA and LC. In several instances, BSR actually improved base model accuracy; for example, a ResNet56 on CIFAR-10 saw accuracy rise from 92.73% to 93.53% even after a 44% reduction in FLOPs. Furthermore, BSR proved competitive with state-of-the-art pruning methods, achieving higher accuracy gains at similar compression levels. When combined with 8-bit quantization, BSR achieved up to a 17x reduction in memory usage with only a 1% drop in accuracy.

Conclusion

The BSR algorithm provides a robust, automated solution for DNN compression that bridges the gap between low-rank factorization and pruning. By leveraging efficient search and targeted regularization, it simplifies the compression pipeline while delivering state-of-the-art results. Future work will explore more advanced rank surrogates and the potential for combining BSR with other compression techniques sequentially.

Summary updated

Abstract

Compression has emerged as one of the essential deep learning research topics, especially for the edge devices that have limited computation power and storage capacity. Among the main compression techniques, low-rank compression via matrix factorization has been known to have two problems. First, an extensive tuning is required. Second, the resulting compression performance is typically not impressive. In this work, we propose a low-rank compression method that utilizes a modified beam-search for an automatic rank selection and a modified stable rank for a compression-friendly training. The resulting BSR (Beam-search and Stable Rank) algorithm requires only a single hyperparameter to be tuned for the desired compression ratio. The performance of BSR in terms of accuracy and compression ratio trade-off curve turns out to be superior to the previously known low-rank compression methods. Furthermore, BSR can perform on par with or better than the state-of-the-art structured pruning methods. As with pruning, BSR can be easily combined with quantization for an additional compression.
BibTeX
@misc{eo2021highly,
  author = {Moonjung Eo and Suhyun Kang and Wonjong Rhee},
  title = {{A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank}},
  year = {2021},
  eprint = {2111.15179},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.15179},
}
Figure