Towards Better Structured Pruning Saliency by Reorganizing Convolution

Xinglong Sun, Humphrey ShiWACV 2024

TL;DR

This paper introduces SPSRC, a novel framework for structured pruning in Convolutional Neural Networks (CNNs). By reorganizing convolutional kernels into matrix multiplications and analyzing their singular values, SPSRC offers a more robust measure of channel importance than traditional kernel-norm-based methods, leading to improved accuracy and compression.

What Problem Does This Paper Address?

Deep neural networks, while powerful, are computationally expensive and require significant storage. Network pruning is a key technique for compressing these models, but structured pruning, which removes entire filters or channels, is particularly challenging. Existing methods for determining which channels to prune often rely on simple kernel-norm-based saliency scores (like L1 magnitude). However, these methods have a critical weakness: they fail to capture the "spatial saliency" of a kernel, meaning they don't account for the fact that certain positions within a kernel (e.g., the center) are more important for computation, especially in zero-padded convolutions. This oversight can lead to suboptimal pruning decisions and a loss of accuracy.

Methods

The core of SPSRC lies in its novel approach to measuring channel importance. Instead of directly analyzing the convolutional kernel weights, SPSRC first reorganizes them. Specifically, a convolutional kernel of shape $N_{in} \times N_{out} \times k \times k$ is transformed into a set of matrices. For each output channel, the weights connecting it to all input channels are reshaped and concatenated into a single large matrix, $A$. This matrix $A$ represents the entire convolution operation for that output channel.

Once this reorganization is done, SPSRC leverages matrix analysis techniques. The paper proposes three variants based on different matrix norms applied to $A$:

  • SPSRC-Frobenius: Uses the Frobenius norm of matrix $A$. This norm implicitly weights central kernel elements more due to the matrix transformation.
  • SPSRC-Spectral: Uses the spectral norm (largest singular value) of matrix $A$. This captures the maximum scaling effect of the convolution on input features.
  • SPSRC-Nuclear: Uses the nuclear norm of matrix $A$. This is related to the rank of the matrix and can be seen as a measure of its complexity.

The importance of a channel is then determined by the chosen norm of its corresponding matrix $A$. Channels with lower scores are pruned. The process is described as a four-step procedure: calculate importance, sort channels, keep the top $P$ channels, and then adjust subsequent layers (like Batch Normalization) to match the reduced channel dimensions. The paper emphasizes that this method is "property-based," meaning it operates on a pre-trained model without requiring joint optimization or learning-based approaches.

Datasets

The paper evaluates SPSRC on the following datasets:

  • CIFAR-10
  • CIFAR-100
  • ImageNet

Key Insights and Ideas

  • Spatial Saliency Matters: The paper convincingly argues that traditional kernel-norm-based pruning methods overlook the spatial importance of kernel elements, particularly the central ones in zero-padded convolutions.
  • Matrix Reorganization Unlocks Insights: Transforming convolutional kernels into matrices ($A$) allows for the application of powerful matrix analysis tools (like SVD) that reveal richer information about channel importance.
  • Implicit Weighting of Central Elements: The Frobenius norm of the reorganized matrix $A$ naturally assigns higher importance to central kernel elements, directly addressing the identified spatial saliency issue.
  • Multiple Norms Offer Flexibility: The exploration of Frobenius, spectral, and nuclear norms provides different perspectives on channel importance, allowing for tailored pruning strategies.
  • Simplicity and Effectiveness: SPSRC is presented as a simple, yet highly effective, framework that can be applied to pre-trained models without complex retraining schemes.

Key Results

SPSRC demonstrates significant improvements across various network architectures (VGGNet, ResNet, DenseNet) and datasets (CIFAR-10, CIFAR-100, ImageNet).

  • VGG16-BN on CIFAR-10: With a 41.86% FLOPs reduction and 81.33% parameter reduction, SPSRC (using spectral norm) achieved a recovered accuracy of 93.9%, which is higher than the baseline accuracy by nearly 0.4%. This outperformed other methods with similar or even fewer compression ratios.
  • ResNet-56 on CIFAR-10: SPSRC consistently achieved higher accuracy than other methods at similar compression ratios, with its superiority becoming more pronounced as compression increased.
  • DenseNet-40 on CIFAR-10: SPSRC achieved higher accuracy than competitive methods with similar or smaller compression ratios.
  • ImageNet (ResNet-34): SPSRC achieved a Top-1 accuracy of 91.55% with 46.62 GFLOPs and 0.3M parameters, outperforming other methods.
  • Hardware-Aware Pruning: When combined with the HALP framework, SPSRC further improved accuracy and FPS.

Limitations and Future Work

  • Limitations: The paper acknowledges that pruning is generally followed by a decrease in accuracy, which is then compensated by retraining or finetuning. While SPSRC aims to minimize this loss, it's not entirely eliminated. The paper also notes that some methods might require specialized hardware or libraries for unstructured pruning, which SPSRC aims to avoid by focusing on structured pruning.
  • Future Work: The authors suggest analyzing reorganized matrices more deeply and exploring other types of compression, such as tensor decomposition. They also plan to conduct more thorough theoretical analyses. The paper also hints at future work in analyzing the reorganized matrices from convolution kernels more deeply and conducting other types of compression.

Summary updated

BibTeX
@inproceedings{sun2024towards,
  author = {Xinglong Sun and Humphrey Shi},
  title = {{Towards Better Structured Pruning Saliency by Reorganizing Convolution}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure