Abstract
Modern neural network architectures like ResNets utilize skip connections that create structurally coupled channels (SCCs), making traditional pruning methods difficult to apply without manual intervention or performance loss. This paper introduces ITPCC (Information Theoretic Pruning of structurally Coupled Channels), a novel framework that leverages Information Bottleneck (IB) theory to model channel importance probabilistically. By treating coupled channels as single representation entities, ITPCC achieves state-of-the-art compression-accuracy trade-offs and significant practical speedups on edge hardware.
The Challenge of Coupled Channels
Pruning is a standard technique for reducing the computational footprint of deep neural networks by removing redundant parameters. While straightforward for sequential architectures like VGG, it becomes complex in models with skip connections. In a ResNet, if a channel in one layer is pruned, its corresponding channel in a connected layer must also be pruned to maintain tensor shape compatibility.
Most existing pruning methods either ignore these Structurally Coupled Channels (SCCs), limiting potential compression, or use deterministic, rule-based heuristics that fail to capture the nuanced information flow within the network. The authors argue that a more principled, information-theoretic approach is required to identify which functional paths are truly redundant.
Methodology: ITPCC
ITPCC redefines channel importance by moving from fixed saliency scores to a probabilistic framework. The core innovation lies in treating a set of coupled channels as a single unit and determining its importance through the lens of Information Bottleneck (IB) theory.
Probabilistic Saliency and IB Theory
Instead of assigning a static weight to a channel, ITPCC assigns a Gaussian distribution with trainable parameters (mean $\mu$ and variance $\sigma$). This allows the model to learn not just the importance of a channel, but the uncertainty of that importance. To enable training via standard backpropagation, the authors employ the reparameterization trick.
Breaking the Markov Assumption
Previous IB-based pruning methods assumed a strictly sequential (Markovian) information flow. ITPCC derives a new mathematical upper bound for mutual information that accounts for the non-consecutive, coupled layers found in modern architectures. This theoretical advancement allows the optimization process to naturally align the indices of coupled channels, ensuring architectural integrity without manual rules.
Experimental Results
ITPCC was evaluated on standard benchmarks including CIFAR-10, CIFAR-100, and ImageNet using ResNet and MobileNetV2 architectures.
State-of-the-Art Performance
- ResNet-56 (CIFAR-10): Achieved 94.14% accuracy with a 34.38% reduction in FLOPs, outperforming previous variational methods by nearly 2% in accuracy at similar compression levels.
- ResNet-110 (CIFAR-10): Reduced FLOPs by 64.33% while maintaining a high accuracy of 94.23%.
- MobileNetV2: Successfully pruned 48.90% of FLOPs with a slight increase in accuracy over the unpruned baseline, demonstrating the method's robustness across different architecture types.
Real-World Hardware Efficiency
A key highlight of ITPCC is its performance on edge devices. By effectively pruning coupled channels, the method allows hardware to release memory from feature maps earlier, leading to better actual latency improvements compared to methods that only prune independent channels.
On a Raspberry Pi 4, ITPCC-compressed ResNet-50 models achieved practical speedups of up to 2.68x while maintaining a competitive 73.68% accuracy on ImageNet. Similar trends were observed on the NVIDIA Jetson Nano, where speedups reached up to 3.07x.
Conclusion and Future Work
ITPCC provides a theoretically grounded and practically effective solution for pruning complex, modern neural networks. By modeling coupled channels as probabilistic entities within an Information Bottleneck framework, it bridges the gap between theoretical compression and real-world hardware acceleration. Future research directions include integrating Neural Architecture Search (NAS) to optimize the baseline models before pruning and extending the SCC approach to more complex vision tasks like semantic segmentation.
Summary updated