Cross-Channel Intragroup Sparsity Neural Network

Zhilin Yu, Chao Wang, Qing Wu, Yong Zhao, Xundong WuarXiv 2019 · arXiv

Introduction

Deep neural network (DNN) compression is essential for deploying state-of-the-art models on resource-constrained edge devices. Pruning is a primary technique for this, yet it traditionally faces a stark tradeoff. Fine-grained (unstructured) pruning maintains high accuracy but results in irregular memory access patterns that hinder hardware acceleration. Conversely, structured pruning (removing entire filters) is hardware-friendly but often leads to significant accuracy degradation due to its rigid constraints. This paper proposes Cross-Channel Intragroup (CCI) Sparsity, a novel pruning structure designed to bridge this gap by offering dense-like hardware efficiency with the accuracy of fine-grained methods.

Cross-Channel Intragroup (CCI) Sparsity

The core innovation is the CCI-Sparsity pattern. Unlike unstructured pruning, CCI-Sparsity partitions weights into small, fixed-size groups across output channels. For a given group size $G$, exactly $s$ weights remain non-zero. For instance, with $G=4$ and $s=1$, one weight is selected from the same spatial position across four consecutive output channels.

The CCI-Sparsity pattern (b) partitions weights into regular groups across output channels, contrasting with the dense matrix (a).

This structure is mathematically optimized for memory I/O. By grouping across output channels rather than input channels, the hardware can maintain output registers in a stationary state while reading input and weight matrices in continuous blocks. This eliminates the overhead of irregular dataflow and partial sum management typical of other sparse formats.

Optimized Training: Ramping Targeted Dropout

Training networks under the strict constraints of CCI-Sparsity is challenging. Traditional iterative pruning can be suboptimal as it permanently removes weights that might have become important later. The authors employ Targeted Dropout (TD), where low-magnitude weights are stochastically dropped during training.

A key contribution of this work is an improved TD algorithm. The authors identified that standard TD (with a fixed 50% dropout rate) allows "pruned" weights to hover near zero without reaching it, which interferes with Batch Normalization (BN) statistics and degrades accuracy. To solve this "silent killer," they introduce a ramping dropout rate that increases from 50% to 100% over the training duration. This forces pruned weights to exactly zero, ensuring BN layers reflect the true sparse architecture.

Ramping dropout forces pruned weights to zero (a), leading to better accuracy compared to standard iterative pruning (b).

Experimental Results

The proposed method was evaluated on CIFAR-10 and ImageNet across various architectures including ResNet, VGG, and MobileNetV2.

  • ImageNet Performance: On MobileNetV2, CCI-Sparsity ($G=8, s=2$, equating to 75% sparsity) achieved 69.8% Top-1 accuracy with ramping dropout, a 5% improvement over fixed-rate dropout.
  • Comparison to Structured Pruning: On ResNet-18, the method achieved 70.31% accuracy while pruning roughly 70% of FLOPs. This significantly outperforms traditional structured pruning methods, which typically see accuracy drops of over 3% for much lower pruning ratios.
  • Pareto Frontier: CCI-Sparsity models consistently outperformed specialized lightweight architectures like ShuffleNetV2 and CondenseNet in Accuracy vs. FLOPs comparisons.

Conclusion

CCI-Sparsity provides a compelling middle ground in the pruning landscape. By leveraging output-channel grouping for hardware efficiency and a ramping Targeted Dropout strategy for training stability, it enables high-performance sparse models that are truly practical for edge deployment. Future work includes combining CCI-Sparsity with coarse-grained pruning and exploring advanced regularization techniques like Exclusive Lasso to further enhance convergence.

Summary updated

BibTeX
@misc{yu2019cross,
  author = {Zhilin Yu and Chao Wang and Qing Wu and Yong Zhao and Xundong Wu},
  title = {{Cross-Channel Intragroup Sparsity Neural Network}},
  year = {2019},
  eprint = {1910.11971},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.11971},
}
Figure