Boosting Pruned Networks with Linear Over-Parameterization

Yu Qian, Xiaoshuang Li, Jian Cao, Jie Zhang, Hufei Li, Jue ChenICASSP 2024 · DOI

TL;DR

This paper tackles the accuracy drop often seen after pruning neural networks, especially at high pruning ratios. They propose a method that temporarily expands pruned layers using linear over-parameterization, fine-tunes the expanded network with similarity-preserving knowledge distillation, and then contracts it back to its original compact size. This approach significantly improves accuracy restoration compared to standard fine-tuning.

What Problem Does This Paper Address?

Deep learning models are computationally expensive, making network pruning a crucial technique for reducing their size and speeding up inference. Structured pruning, which removes entire channels, is particularly effective for hardware acceleration. However, pruning often leads to a significant decrease in model accuracy. While fine-tuning can restore some of this accuracy, it becomes increasingly difficult as the pruning ratio (the percentage of removed weights) increases. Existing fine-tuning methods either struggle to recover accuracy or bypass fine-tuning altogether. This paper aims to bridge this gap by developing a method that effectively restores accuracy even for heavily pruned networks.

Methods

The core of the proposed method lies in linear over-parameterization. Instead of directly fine-tuning the pruned, compact network, the authors temporarily expand the pruned layers. This expansion is achieved by decomposing a single layer into multiple consecutive linear layers, effectively increasing the number of learnable parameters without adding non-linearities.

Specifically:

  • Convolutional Layers: Expanded using a sequence of 1x1, kxk, and 1x1 convolutions. The expansion rate r controls the intermediate channel count.
  • Fully-Connected Layers: Expanded into a series of linear layers.

To preserve the network's feature extraction capabilities during this expansion and fine-tuning phase, they employ similarity-preserving knowledge distillation. This technique uses the activation maps from the original, unpruned network (the "teacher") to guide the training of the over-parameterized, pruned network (the "student"). This ensures that the expanded layers learn to mimic the behavior of the original layers.

After fine-tuning the over-parameterized network, it is contracted back to its original compact structure, retaining the improved accuracy. This entire process can be seamlessly integrated with existing pruning techniques.

Datasets

The paper evaluates its method on two standard image classification datasets:

  • CIFAR-10: Used for experiments with MobileNetV2.
  • ImageNet: Used for experiments with ResNet-50.

Key Insights and Ideas

  • Linear Over-parameterization as a Fine-tuning Aid: The paper demonstrates that temporarily increasing the parameter count of pruned layers through linear decomposition can significantly aid in accuracy recovery during fine-tuning, especially when standard fine-tuning struggles.
  • Knowledge Distillation is Crucial: Simply over-parameterizing isn't enough. Similarity-preserving knowledge distillation is essential to guide the expanded layers to learn meaningful features, preventing the loss of representational power.
  • High Pruning Ratios are Where it Shines: The method shows its most significant benefits when dealing with very high pruning ratios (e.g., 95.52%), where traditional fine-tuning falters.
  • Compatibility with Existing Pruning Methods: The approach is designed to be a plug-and-play module that can enhance the fine-tuning stage of various pruning strategies.

Key Results

  • On CIFAR-10 with MobileNetV2, the proposed method achieved 2.04% higher accuracy restoration than vanilla fine-tuning at a 95.52% pruning ratio.
  • Across various pruning strategies (AMC, HRank, ThiNet, EagleEye) on CIFAR-10 and ImageNet, the proposed method consistently outperformed vanilla fine-tuning, particularly at higher pruning levels.
  • Ablation studies confirmed that both linear over-parameterization and similarity-preserving knowledge distillation are critical components for achieving the observed performance gains.

Limitations and Future Work

  • Expansion Rate Sensitivity: The paper notes that while an expansion rate of 3 is beneficial, larger rates (e.g., 5) can overly complicate the network and hinder convergence. This suggests a need for careful tuning of the expansion rate.
  • Computational Overhead during Fine-tuning: The over-parameterization step temporarily increases the model size and computational cost during the fine-tuning phase. While the final model is compact, this intermediate step adds overhead.
  • Future Work: The authors suggest that their method can be concurrently applied with other techniques in pruning optimization, weight importance evaluation, and adaptive pruning ratios to further enhance overall pruning effectiveness.

Summary updated

BibTeX
@inproceedings{qian2024boosting,
  author = {Yu Qian and Xiaoshuang Li and Jian Cao and Jie Zhang and Hufei Li and Jue Chen},
  title = {{Boosting Pruned Networks with Linear Over-Parameterization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10448499},
}
Figure