Abstract
The low-rank tensor approximation is very promising for the compression of deep neural networks. We propose a new simple and efficient iterative approach, which alternates low-rank factorization with a smart rank selection and fine-tuning. We demonstrate the efficiency of our method comparing to non-iterative ones. Our approach improves the compression rate while maintaining the accuracy for a variety of tasks.
The Challenge of Model Size
Deep learning models have grown massive, often containing tens of millions of parameters. This makes them difficult to deploy on "edge" devices like smartphones or embedded systems, which have strict memory and computational limits.
Existing "low-rank" compression methods typically follow a two-step process: compress the weights once, then fine-tune the model to recover lost accuracy. However, if you try to compress too much in that single step, the model's quality often drops off a cliff, and fine-tuning cannot fully recover it. The authors identify that a "one-shot" approach creates a massive jump away from the local optimum on the loss surface, making it difficult for the optimizer to find its way back to a high-performing state.
The MUSCO Framework: Iterative Refinement
The core of MUSCO (Multi-Stage COmpression) is an iterative loop that gradually reduces model complexity. By taking many small steps instead of one giant leap, MUSCO keeps the model closer to a high-quality solution throughout the process.
The algorithm follows these key steps:
- Smart Rank Selection: The algorithm automatically determines how much to compress each layer. It uses a Bayesian approach (Global Analytic Solution of Empirical Variational Bayesian Matrix Factorization, or EVBMF) to find the "extreme rank" where redundancy is eliminated, or a constant compression rate strategy to hit specific hardware targets.
- Low-Rank Factorization: For convolutional layers, MUSCO primarily uses Tucker-2 decomposition. This replaces one heavy convolutional layer with three lighter, consecutive ones by factorizing the input and output channel dimensions.
- Fine-Tuning: After a modest compression step, the model is fine-tuned on training data to recover its performance.
- Repeat: The process repeats, gradually squeezing the "rank" (and thus the parameter count) of the layers until the desired compression target is reached.
Empirical Evaluation
The authors evaluated MUSCO on several standard computer vision benchmarks, including Object Detection (Faster R-CNN on PASCAL VOC 2007 and COCO 2014), Classification (AlexNet, VGG-16), and Real-time Detection (YOLOv2).
Key results include:
- Faster R-CNN (VGG-16): Achieved a 10.49x reduction in FLOPs (floating-point operations) with only a 1.9 point drop in mAP (mean Average Precision). This significantly outperformed non-iterative state-of-the-art methods like Channel Pruning.
- Faster R-CNN (ResNet-50): Achieved a 1.39x FLOPs reduction while actually increasing accuracy by 2.0 mAP, suggesting the compression process acts as a form of regularization.
- CPU Speedups: Real-world speedups on standard Intel CPUs reached up to 4.73x for AlexNet and 3.11x for VGG-16.
Conclusion
MUSCO demonstrates that gradual, multi-stage compression is more effective than aggressive one-shot methods. While the iterative nature increases total compression time due to multiple fine-tuning rounds, the resulting models are significantly more efficient for inference. Future work may explore combining MUSCO with other techniques, such as weight pruning or quantization, to achieve even more extreme compression ratios.
Summary updated