Channel Regeneration: Improving Channel Utilization for Compact DNNs

Ankit Kumar Sharma, Hassan ForooshAAAI 2023 · DOI

Introduction

Deep Neural Networks (DNNs) have revolutionized computer vision, yet even optimized "compact" architectures like MobileNet and ShuffleNet suffer from significant overparameterization. A primary cause of this inefficiency is the emergence of "inhibited channels"—neurons that cease learning early in the training process. This paper, presented at AAAI-2023, identifies that these channels become redundant because their gradients drop to zero and remain there, wasting the limited capacity of edge-optimized models.

To address this, the authors propose Channel Regeneration (RG), a plug-and-play training intervention that revitalizes dormant channels by periodically re-initializing their Batch Normalization (BN) scaling factors. This method requires no architectural changes and incurs zero additional inference cost.

The Channel Regeneration (RG) Algorithm

The RG algorithm is a simple, iterative process designed to ensure that all channels in a network contribute to the final learned representation. The method follows three key steps:

  1. Identification: During training, the algorithm monitors the BN scaling factor ($\gamma$) for each channel. Channels are flagged as "inhibited" if $|\gamma|$ falls below a specific threshold (typically $10^{-2}$).
  2. Regeneration: At predefined "Regeneration Epochs" ($E_{RG}$), the $\gamma$ values of inhibited channels are reset. Rather than using random noise, the authors re-initialize these values to the median $\gamma$ of all active channels in that specific layer, ensuring the re-awakened channel starts at a competitive scale.
  3. Iterative Application: The process is repeated at regular intervals (e.g., every 5 to 10 epochs) to prevent channels from relapsing into a dormant state.
The Channel Regeneration (RG) workflow: identifying inhibited channels based on low BN scaling factors (gamma) and resetting them to the median value of active channels.

The motivation for RG is grounded in the mathematical relationship between BN layers and gradient flow. The authors derive that the gradient update for a convolutional filter is directly proportional to the $\gamma$ of the subsequent BN layer. When $\gamma$ approaches zero, the gradient flow to the preceding filter is effectively choked off, rendering the channel useless.

Impact of Channel Regeneration on gradient flow. Without RG, inhibited channels lose gradient updates permanently; RG successfully revitalizes these channels, as seen at the 20th epoch.

As shown in the figure above, standard training leads to a permanent loss of gradient updates for a subset of channels. Channel Regeneration successfully "wakes up" these dead channels, allowing them to resume learning and contribute to the model's feature extraction capabilities.

Key Results and Insights

The quantitative benefits of Channel Regeneration are most pronounced in compact models where every parameter counts:

  • Accuracy Boosts: On CIFAR-100, MobileNetv2 accuracy improved by 2.8% (from 69.10% to 71.90%). On Tiny-ImageNet, the gain was even more significant at 4.53%.
  • Scalability: The method proved effective on full-scale ImageNet-1K and across diverse tasks like semantic segmentation on ADE20K, where mIoU improved by 1% to 2%.
  • Robustness: Ablation studies revealed that RG-trained models distribute knowledge more democratically across channels. When the most active channels were manually removed, RG models showed a much more gradual decline in performance compared to standard models.
  • Synergy: RG is compatible with other optimization techniques, enhancing the performance of models that are already being pruned or quantized.

Conclusion

Channel Regeneration offers a rare "free lunch" in deep learning. By addressing the root cause of inhibited channels through a simple training-time reset of BN parameters, it extracts higher performance from existing hardware-friendly architectures without any deployment-side penalties. While future work may investigate the underlying reasons why channels die, RG provides an immediate and effective solution for maximizing channel utilization in compact DNNs.

Summary updated

BibTeX
@inproceedings{sharma2023channel,
  author = {Ankit Kumar Sharma and Hassan Foroosh},
  title = {{Channel Regeneration: Improving Channel Utilization for Compact DNNs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25316},
}
Figure