On the Channel Pruning using Graph Convolution Network for Convolutional Neural Network Acceleration

Di Jiang, Yuan Cao, Qiang YangIJCAI 2022 · DOI

TL;DR

This paper introduces a novel channel pruning method for Convolutional Neural Networks (CNNs) that leverages Graph Convolutional Networks (GCNs) to intelligently determine which channels to remove. By modeling channel relationships as a graph and using Reinforcement Learning (RL) for training, the method achieves significant model compression with minimal accuracy loss, outperforming existing magnitude and similarity-based pruning techniques.

What Problem Does This Paper Address?

Convolutional Neural Networks (CNNs) are powerful tools for tasks like computer vision, but their large size and computational demands make them challenging to deploy on resource-constrained devices (e.g., mobile phones, embedded systems). Model compression techniques, particularly network pruning, aim to reduce model size and speed up inference without sacrificing accuracy. While existing methods often rely on simple criteria like channel weight magnitude or feature similarity, they tend to be suboptimal when used independently. This paper addresses the need for a more sophisticated pruning strategy that can effectively combine these criteria and adapt to complex network architectures, leading to better compression performance.

Methods

The core of this paper's contribution lies in its novel pruning approach that combines Graph Convolutional Networks (GCNs) with Reinforcement Learning (RL).

  • Graph Construction: For each layer in a CNN, the authors construct a graph where each channel is represented as a node. The features for these nodes are derived from the Global Average Pooling (GAP) distribution of channel activations across a batch of samples. The similarity between channels (and thus the edges in the graph) is determined using KL-divergence between their feature distributions.
  • GCN for Pruning Scheme: A GCN is then employed to process this graph. The GCN aggregates information from neighboring channels, allowing it to learn a comprehensive understanding of channel importance by considering both their individual features (magnitude-like) and their relationships with other channels (similarity-like). The output of the GCN is a probability distribution over channels, indicating their likelihood of being pruned.
  • Reinforcement Learning for Training: To obtain the final binary pruning decision (keep or prune a channel), Reinforcement Learning (RL) is used. An RL agent, incorporating the GCN, is trained to minimize a loss function that balances performance degradation (accuracy loss) and the proportion of pruned parameters. This "try-and-learn" process allows the agent to adaptively learn the optimal pruning strategy for each layer.
  • Handling Residual Connections: For networks like ResNet that have residual connections, the authors propose a grouping strategy. Layers within residual blocks are grouped to handle the inter-layer dependencies, allowing for more effective pruning in these complex architectures.

Datasets

The paper evaluates its method on the following datasets:

  • CIFAR-10 and CIFAR-100: Standard image classification datasets with 10 and 100 classes, respectively.
  • Tiny-ImageNet: A subset of ImageNet with 200 classes and higher resolution images (64x64).

Key Insights and Ideas

  • Synergy of Magnitude and Similarity: The paper demonstrates that by modeling channel relationships as a graph and using GCNs, it's possible to effectively combine the strengths of magnitude-based and similarity-based pruning criteria, leading to superior results.
  • Adaptive Pruning with RL: Using RL to train the GCN-based pruning agent allows the method to automatically learn optimal pruning strategies for different layers and network architectures without requiring manual hyperparameter tuning for each layer.
  • Graph Representation of Channels: Representing channels as nodes in a graph, with features derived from GAP distributions and edges based on KL-divergence, provides a powerful way to capture channel interdependencies.
  • Handling Complex Architectures: The proposed grouping strategy for residual connections is crucial for extending the method's effectiveness to modern, complex CNN architectures like ResNet.

Key Results

The proposed GCN-based pruning method achieves state-of-the-art performance:

  • Significant Compression: On the CIFAR-10 dataset, the method can prune VGG16 by removing 93% of its parameters without any accuracy reduction. For ResNet56 on CIFAR-10, it achieves 45.95% FLOPs reduction and 42.07% parameter reduction while maintaining the baseline accuracy.
  • Outperforms SOTA: The method consistently outperforms pure magnitude-based, similarity-based, and other state-of-the-art pruning methods (e.g., HRank, SCP) across various datasets and networks.
  • Accuracy Preservation: The approach is particularly effective at preserving accuracy, often achieving comparable or even slightly improved accuracy compared to the baseline models, especially at moderate compression rates.

Limitations and Future Work

  • Threshold Selection: The paper acknowledges that the selection of the KL-divergence threshold ($\theta$) for graph edge construction is an important factor. Inappropriate selection can lead to performance degradation, suggesting that an adaptive or learned threshold could be a future improvement.
  • Computational Cost: While the goal is acceleration, the pruning process itself, especially the GCN and RL training, can be computationally intensive.
  • Future Work: The authors suggest further validating the method by deploying it in different application domains, implying that its effectiveness might vary across diverse tasks and hardware platforms. They also hint at exploring multiple rounds of pruning with varying balance ratios ($\gamma$) for achieving very high compression rates.

Summary updated

BibTeX
@inproceedings{jiang2022channel,
  author = {Di Jiang and Yuan Cao and Qiang Yang},
  title = {{On the Channel Pruning using Graph Convolution Network for Convolutional Neural Network Acceleration}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/431},
}
Figure