CR-SFP: Learning Consistent Representation for Soft Filter Pruning

Jingyang Xiang, Zhuangzhi Chen, Jianbiao Mei, Siqi Li, Jun Chen, Yong LiuarXiv 2023 · DOI · arXiv

Introduction

Deep Neural Networks (DNNs) have achieved remarkable success in computer vision but remain computationally expensive for deployment on resource-constrained edge devices. Filter pruning has emerged as a primary technique to reduce model complexity. While Hard Filter Pruning (HFP) permanently removes filters, Soft Filter Pruning (SFP) offers more flexibility by zeroing out unimportant filters during training while allowing them to be updated and potentially "regrow."

However, SFP suffers from a critical issue: representation inconsistency. During training, the model operates as a full, reconstructed network (R-NN), but at inference, it functions as a pruned network (P-NN). This discrepancy creates a performance gap, as the model is optimized for a state it does not inhabit during deployment. This "identity crisis" leads to suboptimal feature extraction in the pruned state compared to the full-capacity training phase.

Methodology: The CR-SFP Framework

To address this gap, we propose Consistent Representation for Soft Filter Pruning (CR-SFP). This approach treats pruning as a mutual learning task between the R-NN and P-NN, ensuring that the pruned network's feature space remains aligned with the full network's capacity.

The CR-SFP framework utilizes a dual-branch training strategy with bidirectional KL-divergence to align the representations of the reconstructed (R-NN) and pruned (P-NN) networks.

The framework consists of three core components:

  1. Dual-Branch Training: The R-NN and P-NN are treated as two branches sharing the same backbone parameters but utilizing separate classifier heads. This allows the model to learn features that are effective for both the full and pruned configurations simultaneously.
  2. Data Distortion: Each branch receives a differently augmented version of the same input image. This data-distortion operation encourages the model to learn more robust and invariant features, further bridging the representation gap.
  3. Bidirectional KL-Divergence Loss: A mutual learning objective forces the output probability distributions of both branches to be consistent. By minimizing the KL-divergence between the R-NN and P-NN, the pruned model learns to mimic the richer representations of the full model. A "stopgrad" operation is applied to stabilize training, alternating the target distribution between the two branches to prevent trivial solutions.

Post-training, the auxiliary classifier is discarded, leaving a compact P-NN for inference. This ensures zero additional computational cost at runtime compared to standard SFP.

Experimental Results

We evaluated CR-SFP on the ImageNet dataset using the ResNet family of architectures (ResNet18, ResNet34, and ResNet50). Our method consistently outperforms baseline SFP and other state-of-the-art pruning techniques.

Comparison of CR-SFP with state-of-the-art pruning methods on ImageNet. CR-SFP consistently achieves higher Top-1 accuracy across ResNet architectures.

Key findings include:

  • ResNet18: CR-SFP achieved a 69.2% top-1 accuracy with a 41.8% reduction in FLOPs, representing a 2.1% improvement over standard SFP.
  • ResNet50: The model reached 76.6% top-1 accuracy, surpassing several methods that require pre-trained weights (PT: Y), despite CR-SFP training from scratch (PT: N).
  • Efficiency and Robustness: While CR-SFP requires two forward passes during training, the overhead is minimal (approx. 4ms per step). Furthermore, the performance gains are independent of the specific pruning criteria used (e.g., $\ell_2$-norm or Geometric Median), suggesting that maintaining representation consistency is a fundamental requirement for effective soft pruning.

Conclusion

CR-SFP bridges the gap between training and inference in soft filter pruning by ensuring consistent representations through mutual learning. By forcing the pruned network to maintain the representational power of the full network, we achieve significant accuracy improvements without increasing inference-time complexity. Future research may extend this consistency-based philosophy to other compression techniques such as network quantization and binarization.

Summary updated

Abstract

Soft filter pruning~(SFP) has emerged as an effective pruning technique for allowing pruned filters to update and the opportunity for them to regrow to the network. However, this pruning strategy applies training and pruning in an alternative manner, which inevitably causes inconsistent representations between the reconstructed network~(R-NN) at the training and the pruned network~(P-NN) at the inference, resulting in performance degradation. In this paper, we propose to mitigate this gap by learning consistent representation for soft filter pruning, dubbed as CR-SFP. Specifically, for each training step, CR-SFP optimizes the R-NN and P-NN simultaneously with different distorted versions of the same training data, while forcing them to be consistent by minimizing their posterior distribution via the bidirectional KL-divergence loss. Meanwhile, the R-NN and P-NN share backbone parameters thus only additional classifier parameters are introduced. After training, we can export the P-NN for inference. CR-SFP is a simple yet effective training framework to improve the accuracy of P-NN without introducing any additional inference cost. It can also be combined with a variety of pruning criteria and loss functions. Extensive experiments demonstrate our CR-SFP achieves consistent improvements across various CNN architectures. Notably, on ImageNet, our CR-SFP reduces more than 41.8\% FLOPs on ResNet18 with 69.2\% top-1 accuracy, improving SFP by 2.1\% under the same training settings. The code will be publicly available on GitHub.
BibTeX
@misc{xiang2023cr,
  author = {Jingyang Xiang and Zhuangzhi Chen and Jianbiao Mei and Siqi Li and Jun Chen and Yong Liu},
  title = {{CR-SFP: Learning Consistent Representation for Soft Filter Pruning}},
  year = {2023},
  eprint = {2312.11555},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.11555},
}
Figure