Information Theoretic Pruning of Coupled Channels in Deep Neural Networks

Peyman Rostami, Nilotpal Sinha, Nidhal Eddine Chenni, Anis Kacem, Abd El Rahman Shabayek, Carl Shneider, Djamila AouadaWACV 2025 · DOI

Abstract

Modern neural network architectures like ResNets utilize skip connections that create structurally coupled channels (SCCs), making traditional pruning methods difficult to apply without manual intervention or performance loss. This paper introduces ITPCC (Information Theoretic Pruning of structurally Coupled Channels), a novel framework that leverages Information Bottleneck (IB) theory to model channel importance probabilistically. By treating coupled channels as single representation entities, ITPCC achieves state-of-the-art compression-accuracy trade-offs and significant practical speedups on edge hardware.

The Challenge of Coupled Channels

Pruning is a standard technique for reducing the computational footprint of deep neural networks by removing redundant parameters. While straightforward for sequential architectures like VGG, it becomes complex in models with skip connections. In a ResNet, if a channel in one layer is pruned, its corresponding channel in a connected layer must also be pruned to maintain tensor shape compatibility.

Most existing pruning methods either ignore these Structurally Coupled Channels (SCCs), limiting potential compression, or use deterministic, rule-based heuristics that fail to capture the nuanced information flow within the network. The authors argue that a more principled, information-theoretic approach is required to identify which functional paths are truly redundant.

Methodology: ITPCC

ITPCC redefines channel importance by moving from fixed saliency scores to a probabilistic framework. The core innovation lies in treating a set of coupled channels as a single unit and determining its importance through the lens of Information Bottleneck (IB) theory.

Probabilistic Saliency and IB Theory

Instead of assigning a static weight to a channel, ITPCC assigns a Gaussian distribution with trainable parameters (mean $\mu$ and variance $\sigma$). This allows the model to learn not just the importance of a channel, but the uncertainty of that importance. To enable training via standard backpropagation, the authors employ the reparameterization trick.

The ITPCC framework models the importance of independent and coupled channels using a probabilistic distribution derived from Information Bottleneck theory.

Breaking the Markov Assumption

Previous IB-based pruning methods assumed a strictly sequential (Markovian) information flow. ITPCC derives a new mathematical upper bound for mutual information that accounts for the non-consecutive, coupled layers found in modern architectures. This theoretical advancement allows the optimization process to naturally align the indices of coupled channels, ensuring architectural integrity without manual rules.

Experimental Results

ITPCC was evaluated on standard benchmarks including CIFAR-10, CIFAR-100, and ImageNet using ResNet and MobileNetV2 architectures.

State-of-the-Art Performance

  • ResNet-56 (CIFAR-10): Achieved 94.14% accuracy with a 34.38% reduction in FLOPs, outperforming previous variational methods by nearly 2% in accuracy at similar compression levels.
  • ResNet-110 (CIFAR-10): Reduced FLOPs by 64.33% while maintaining a high accuracy of 94.23%.
  • MobileNetV2: Successfully pruned 48.90% of FLOPs with a slight increase in accuracy over the unpruned baseline, demonstrating the method's robustness across different architecture types.

Real-World Hardware Efficiency

A key highlight of ITPCC is its performance on edge devices. By effectively pruning coupled channels, the method allows hardware to release memory from feature maps earlier, leading to better actual latency improvements compared to methods that only prune independent channels.

ITPCC achieves significant speedups on edge devices like Jetson Nano and Raspberry Pi while maintaining high accuracy across different compression levels.

On a Raspberry Pi 4, ITPCC-compressed ResNet-50 models achieved practical speedups of up to 2.68x while maintaining a competitive 73.68% accuracy on ImageNet. Similar trends were observed on the NVIDIA Jetson Nano, where speedups reached up to 3.07x.

Conclusion and Future Work

ITPCC provides a theoretically grounded and practically effective solution for pruning complex, modern neural networks. By modeling coupled channels as probabilistic entities within an Information Bottleneck framework, it bridges the gap between theoretical compression and real-world hardware acceleration. Future research directions include integrating Neural Architecture Search (NAS) to optimize the baseline models before pruning and extending the SCC approach to more complex vision tasks like semantic segmentation.

Summary updated

BibTeX
@inproceedings{rostami2025information,
  author = {Peyman Rostami and Nilotpal Sinha and Nidhal Eddine Chenni and Anis Kacem and Abd El Rahman Shabayek and Carl Shneider and Djamila Aouada},
  title = {{Information Theoretic Pruning of Coupled Channels in Deep Neural Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00755},
}
Figure