Entropy Induced Pruning Framework for Convolutional Neural Networks

Yiheng Lu, Ziyu Guan, Yaming Yang, Wei Zhao, Maoguo Gong, Cai XuAAAI 2024 · DOI

Introduction

Deep learning models are notoriously over-parameterized, leading to high computational costs and memory requirements. Structured pruning—removing entire filters or channels—is a popular solution to compress these models. However, most existing pruning methods suffer from a "chicken-and-egg" problem: they require a fully trained model to accurately identify which filters are unimportant. Training a large model to convergence just to throw half of it away is computationally expensive and inefficient. The Average Filter Information Entropy (AFIE) framework addresses this by identifying redundancy early in the training process.

Methodology: Average Filter Information Entropy (AFIE)

The authors propose a three-step framework to evaluate filter importance without needing a fully optimized model. This approach moves away from weight-oriented importance to a more stable structure-oriented metric:

  1. Low-Rank Space Projection: The weight matrix of a convolutional layer is decomposed using Singular Value Decomposition (SVD) to yield eigenvalues representing the "strength" of different dimensions.
  2. Normalization: Eigenvalues are normalized using "0-1" scaling and a Softmax function to create a probability-like distribution.
  3. Entropy Calculation: The information entropy of this distribution is calculated. Low entropy indicates high redundancy (a few eigenvalues dominate), while high entropy suggests unique information across filters.
The AFIE pruning pipeline, from SVD decomposition to filter removal.

The final AFIE score is the layer's entropy divided by its number of filters. Layers with lower AFIE scores are assigned higher pruning ratios because they contain more redundant information.

Experimental Results

The framework was evaluated on standard benchmarks: MNIST (AlexNet), CIFAR-10 (VGG-16), and ImageNet (ResNet-50). A key finding is the stability of the AFIE score; it remains nearly identical whether the model is trained for one epoch or 150 epochs. This suggests that structural redundancy is determined very early by architecture and initialization, allowing for a "one-shot" pruning strategy.

Pruning ratios across convolutional layers for AlexNet, VGG-16, and ResNet-50.

Performance Highlights:

  • AlexNet (MNIST): Achieved 99.02% accuracy with 70% of parameters pruned, starting from a model trained for only 1 epoch.
  • VGG-16 (CIFAR-10): Maintained 93.12% accuracy (original 93.35%) with a 65% pruning ratio.
  • ResNet-50 (ImageNet): Pruned 30% of parameters while maintaining 74.5% accuracy (original 75.8%) using a 1-epoch model.

Conclusion

The AFIE framework demonstrates that structural redundancy can be quantified through information entropy early in the training cycle. By leveraging this stability, the method avoids the need for full-model pre-training, offering a highly efficient path to model compression. Future work may explore more sophisticated intra-layer selection methods beyond random removal and application to non-standard convolutional architectures like depthwise-separable layers.

Summary updated

BibTeX
@inproceedings{lu2024entropy,
  author = {Yiheng Lu and Ziyu Guan and Yaming Yang and Wei Zhao and Maoguo Gong and Cai Xu},
  title = {{Entropy Induced Pruning Framework for Convolutional Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28184},
}
Figure