Compressing by Learning in a Low-Rank and Sparse Decomposition Form

Kailing Guo, Xiaona Xie, Xiangmin Xu, Xiaofen XingIEEE Access 2019 · DOI

Abstract

Modern convolutional neural networks (CNNs) achieve state-of-the-art performance at the cost of massive parameter counts, hindering deployment on resource-constrained devices. We present the Low-Rank and Sparse Decomposition (LRSD) framework, a novel compression strategy that combines low-rank factorization and weight pruning. Unlike traditional methods that approximate pre-trained models, LRSD trains compressed architectures from scratch. Our approach achieves up to 97% parameter reduction on VGG-19 with negligible accuracy loss, demonstrating that learning compressed structures directly is more effective than post-hoc pruning.

Introduction

As CNNs grow deeper and wider, their memory footprint and computational demands increase significantly. For instance, VGG-19 contains approximately 143 million parameters, making it unsuitable for mobile or embedded deployment. Existing compression techniques typically focus on either low-rank approximation (exploiting global structure) or pruning/sparsity (removing local unimportant connections).

However, these two properties are rarely unified in a single learning framework. Furthermore, most methods attempt to "fix" a pre-trained model, which may not possess the optimal weights for a reduced-capacity architecture. LRSD addresses these gaps by defining a compressed structure before training begins, allowing the network to optimize its limited capacity from the start.

The LRSD Framework

The LRSD framework operates through a three-stage pipeline designed to capture both global and local weight structures.

1. Structure Designing

Every weight matrix $W$ in the network is decomposed into the sum of a low-rank component ($L$) and a sparse component ($S$), such that $W = L + S$. To minimize memory usage, the low-rank component is further factorized into two smaller matrices, $U$ and $V$ ($L = UV$).

Visualization of the LRSD framework where a weight matrix W is decomposed into a low-rank component (UV) and a sparse component (S).

In practice, this replaces a standard convolution with a sequence of two smaller convolutions (the low-rank part) running in parallel with a sparse convolution.

2. Training from Scratch

The network is trained using standard Stochastic Gradient Descent (SGD). To ensure the $S$ component becomes truly sparse, an $\ell_1$ regularization term is added to the loss function. This penalty pushes unimportant weights in $S$ toward zero during the learning process, allowing the model to "learn" which connections are vital for performance.

3. Post-processing: Energy-based Pruning

After training, a novel pruning criterion is applied. Rather than using a fixed magnitude threshold, the authors employ an "energy ratio" ($\alpha$). Weights in $S$ are sorted by magnitude, and only the largest are kept to preserve a specific percentage of the matrix's total "energy" (the sum of absolute values). This adaptive approach allows different layers to have different sparse rates based on their sensitivity.

Experimental Evaluation

The LRSD method was evaluated on three standard benchmarks: CIFAR-10, CIFAR-100, and ImageNet (ILSVRC-2012). The results consistently outperformed state-of-the-art techniques like "Slimming" and "DCP."

  • VGGNet-19 (CIFAR-10): The model was compressed to just 3.14% of its original size (a ~32x reduction) with zero accuracy drop.
  • PreActResNet-56 (CIFAR-10): Parameters were reduced to 29.78% of the original while slightly increasing accuracy.
  • ResNet-50 (ImageNet): Achieved a 62.43% parameter reduction with only a minor 0.55% loss in top-5 accuracy.

Key Insights

  • Global vs. Local: Low-rank components capture the "big picture" global structure, while the sparse component captures fine-grained local details. Combining them retains more expressive power than using either method in isolation.
  • The "Scratch" Advantage: Optimizing $U, V,$ and $S$ together from initialization avoids the performance gap often seen when a pre-trained model is suddenly pruned.
  • Efficiency in 1x1 Layers: The authors noted that $1 \times 1$ convolutions (common in ResNet bottlenecks) are inherently low-rank. For these layers, applying only sparse constraints prevents unnecessary overhead.

Conclusion and Future Work

LRSD demonstrates that a unified decomposition framework can achieve massive compression rates across various architectures. While the method requires hyperparameter tuning (rank $r$, trade-off $\lambda$, and energy ratio $\alpha$), its robustness on both large and compact models is significant. Future research will explore combining LRSD with knowledge distillation to further bridge the accuracy gap on large-scale datasets like ImageNet and optimizing hardware support for sparse masked convolutions.

Summary updated

BibTeX
@article{guo2019compressing,
  author = {Kailing Guo and Xiaona Xie and Xiangmin Xu and Xiaofen Xing},
  title = {{Compressing by Learning in a Low-Rank and Sparse Decomposition Form}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2947846},
}
Figure